X-Git-Url: https://git.mxchange.org/?p=quix0rs-apt-p2p.git;a=blobdiff_plain;f=TODO;h=bd8283c904d6ed372c9a84136ce204422bb6e183;hp=192670b54e3de79b2d0775c61e470c13ea13da79;hb=HEAD;hpb=b36fa9c7f5ad4827c8de990fbfca1a379bb60e39 diff --git a/TODO b/TODO index 192670b..bd8283c 100644 --- a/TODO +++ b/TODO @@ -1,64 +1,121 @@ -Evaluate and fix some errors in the ktable khashmir module. +Rotate DNS entries for mirrors more reliably. + +Currently the mirrors are accessed by DNS name, which can cause some +issues when there are mirror differences and the DNS gets rotated. +Instead, the HTTP Downloader should handle DNS lookups itself, store +the resulting addresses, and send requests to IP addresses. If there +is an error from the mirror (hash check or 404 response), the next IP +address in the rotation should be used. + + +Use GPG signatures as a hash for files. + +A detached GPG signature, such as is found in Release.gpg, can be used +as a hash for the file. This hash can be used to verify the file when +it is downloaded, and a shortened version can be added to the DHT to +look up peers for the file. To get the hash into a binary form from +the ascii-armored detached file, use the command +'gpg --no-options --no-default-keyring --output - --dearmor -'. The +hash should be stored as the reverse of the resulting binary string, +as the bytes at the beginning are headers that are the same for most +signatures. That way the shortened hash stored in the DHT will have a +better chance of being unique and being stored on different peers. To +verify a file, first the binary hash must be re-reversed, armored, and +written to a temporary file with the command +'gpg --no-options --no-default-keyring --output $tempfile --enarmor -'. +Then the incoming file can be verified with the command +'gpg --no-options --no-default-keyring --keyring /etc/apt/trusted.gpg +--verify $tempfile -'. + +All communication with the command-line gpg should be done using pipes +and the python module python-gnupginterface. There needs to be a new +module for GPG verification and hashing, which will make this easier. +In particular, it would need to support hashlib-like functionality +such as new(), update(), and digest(). Note that the verification +would not involve signing the file again and comparing the signatures, +as this is not possible. Instead, the verify() function would have to +behave differently for GPG hashes, and check that the verification +resulted in a VALIDSIG. CAUTION: the detached signature can have a +variable length, though it seems to be usually 65 bytes, 64 bytes has +also been observed. + + +Consider what happens when multiple requests for a file are received. + +When another request comes in for a file already being downloaded, +the new request should wait for the old one to finish. This should +also be done for multiple requests for peer downloads of files with +the same hash. -The KTable implementation has some possible errors in it. insertNode -does not remove the original and use the new node when updating a node -already in the table, as claimed by the comments. justSeenNode doesn't -verify that the found node is the node that was being looked for, nor -does it move the node to the end of the list of nodes (since they are -supposed to be sorted by their lastSeen time) or update the bucket's -last touched time.nodeFailed also doesn't verify the found node is the -right node. +Packages.diff files need to be considered. -Consider what happens when we are the closest node. +The Packages.diff/Index files contain hashes of Packages.diff/rred.gz +files, which themselves contain diffs to the Packages files previously +downloaded. Apt will request these files for the testing/unstable +distributions. They need to be dealt with properly by +adding them to the tracking done by the AptPackages module. -In some of the actions it is unclear what happens when we are one of the -closest nodes to the target key. Do we store values that we publish -ourself? +Improve the estimation of the total number of nodes -Add all cache files to the database. +The current total nodes estimation is based on the number of buckets. +A better way is to look at the average inter-node spacing for the K +closest nodes after a find_node/value completes. Be sure to measure +the inter-node spacing in log2 space to dampen any ill effects. This +can be used in the formula: + nodes = 2^160 / 2^(average of log2 spacing) +The average should also be saved using an exponentially weighted +moving average (of the log2 distance) over separate find_node/value +actions to get a better calculation over time. -All files in the cache should be added to the database, so that they can -be checked to make sure nothing has happened to them. The database would -then need a flag to indicate files that are hashed and available, but -that shouldn't be added to the DHT. +Improve the downloaded and uploaded data measurements. -Packages.diff files need to be considered. +There are 2 places that this data is measured: for statistics, and for +limiting the upload bandwidth. They both have deficiencies as they +sometimes miss the headers or the requests sent out. The upload +bandwidth calculation only considers the stream in the upload and not +the headers sent, and it also doesn't consider the upload bandwidth +from requesting downloads from peers (though that may be a good thing). +The statistics calculations for downloads include the headers of +downloaded files, but not the requests received from peers for upload +files. The statistics for uploaded data only includes the files sent +and not the headers, and also misses the requests for downloads sent to +other peers. + + +Rehash changed files instead of removing them. + +When the modification time of a file changes but the size does not, +the file could be rehased to verify it is the same instead of +automatically removing it. The DB would have to be modified to return +deferred's for a lot of its functions. + + +Consider storing deltas of packages. + +Instead of downloading full package files when a previous version of +the same package is available, peers could request a delta of the +package to the previous version. This would only be done if the delta +is significantly (>50%) smaller than the full package, and is not too +large (absolutely). A peer that has a new package and an old one would +add a list of deltas for the package to the value stored in the DHT. +The delta information would specify the old version (by hash), the +size of the delta, and the hash of the delta. A peer that has the same +old package could then download the delta from the peer by requesting +the hash of the delta. Alternatively, very small deltas could be +stored directly in the DHT. -The Packages.diff/Index files contain hashes of Packages.diff/rred.gz -files, which themselves contain diffs to the Packages files previously -downloaded. Apt will request these files for the testing/unstable -distributions. They need to either be ignored, or dealt with properly by -adding them to the tracking done by the AptPackages module. +Consider tracking security issues with packages. -PeerManager needs to download large files from multiple peers. - -The PeerManager currently chooses a peer at random from the list of -possible peers, and downloads the entire file from there. This needs to -change if both a) the file is large (more than 512 KB), and b) there are -multiple peers with the file. The PeerManager should then break up the -large file into multiple pieces of size < 512 KB, and then send requests -to multiple peers for these pieces. - -This can cause a problem with hash checking the returned data, as hashes -for the pieces are not known. Any file that fails a hash check should be -downloaded again, with each piece being downloaded from different peers -than it was previously. The peers are shifted by 1, so that if a peers -previously downloaded piece i, it now downloads piece i+1, and the first -piece is downloaded by the previous downloader of the last piece, or -preferably a previously unused peer. As each piece is downloaded the -running hash of the file should be checked to determine the place at -which the file differs from the previous download. - -If the hash check then passes, then the peer who originally provided the -bad piece can be assessed blame for the error. Otherwise, the peer who -originally provided the piece is probably at fault, since he is now -providing a later piece. This doesn't work if the differing piece is the -first piece, in which case it is downloaded from a 3rd peer, with -consensus revealing the misbehaving peer. +Since sharing information with others about what packages you have +downloaded (and probably installed) is a possible security +vulnerability, it would be advantageous to not share that information +for packages that have known security vulnerabilities. This would +require some way of obtaining a list of which packages (and versions) +are vulnerable, which is not currently available. Consider adding peer characteristics to the DHT.