X-Git-Url: https://git.mxchange.org/?p=quix0rs-apt-p2p.git;a=blobdiff_plain;f=TODO;h=cfb65611ff30ea8bf7675a11051e86dce904be22;hp=2be63a6c76e56cfd02ac2ccb64d6c4d9c01a1331;hb=aa0f99580b06354d1affbef134d5763b534026b6;hpb=957c001a1e1a0476f9885ff851de9ba1bc78fcef diff --git a/TODO b/TODO index 2be63a6..cfb6561 100644 --- a/TODO +++ b/TODO @@ -1,9 +1,22 @@ -Files for which a hash cannot be found should not be added to the DHT. +Some last few things to do before release. -If the hash can't found, it stands to reason that other peers will not -be able to find the hash either. So adding those files to the DHT will -just clutter it with useless information. Examples include Release.gpg, -Release, Translation-de.bz2, and Contents.gz. +- Handle/investigate the HTTP client pipeline errors +- DB should not always restat files (especially for expired hashes) +- remove missing files at startup (in DB's removeUntracked) +- when files modtime but not size changes, rehash them to be sure +- lengthen the expiry time for DHT entries +- remove files from the peer's download cache +- update the modtime of files downloaded from peers + - also set the Last-Modified header for the return to Apt +- make the DHT timeouts configuration parameters +- refresh expired DHT hashes concurrently instead of sequentially + +Consider what happens when multiple requests for a file are received. + +When another request comes in for a file already being downloaded, +the new request should wait for the old one to finish. This should +also be done for multiple requests for peer downloads of files with +the same hash. Packages.diff files need to be considered. @@ -11,105 +24,62 @@ Packages.diff files need to be considered. The Packages.diff/Index files contain hashes of Packages.diff/rred.gz files, which themselves contain diffs to the Packages files previously downloaded. Apt will request these files for the testing/unstable -distributions. They need to either be ignored, or dealt with properly by +distributions. They need to be dealt with properly by adding them to the tracking done by the AptPackages module. -Hashes need to be sent with requests for some files. - -Some files can change without changing the file name, since the file was -added to the DHT by the peer. Examples are Release, Packages.gz, and -Sources.bz2. For files like this (and only for files like this), the -request to download from the peer should include the downloader's -expected hash for the file as a new HTTP header. If the file is found, -the cached hash for the file will be used to determine whether the -request is for the same file as is currently available, and a special -HTTP response can be sent if it is not (i.e. not a 404). - -Alternatively, consider sharing the files by hash instead of by -directory. Then the request would be for -http://127.3.45.9:9977/, and it would always work. This -would require a database lookup for every request. - - -PeerManager needs to download large files from multiple peers. - -The PeerManager currently chooses a peer at random from the list of -possible peers, and downloads the entire file from there. This needs to -change if both a) the file is large (more than 512 KB), and b) there are -multiple peers with the file. The PeerManager should then break up the -large file into multiple pieces of size < 512 KB, and then send requests -to multiple peers for these pieces. - -This can cause a problem with hash checking the returned data, as hashes -for the pieces are not known. Any file that fails a hash check should be -downloaded again, with each piece being downloaded from different peers -than it was previously. The peers are shifted by 1, so that if a peers -previously downloaded piece i, it now downloads piece i+1, and the first -piece is downloaded by the previous downloader of the last piece, or -preferably a previously unused peer. As each piece is downloaded the -running hash of the file should be checked to determine the place at -which the file differs from the previous download. - -If the hash check then passes, then the peer who originally provided the -bad piece can be assessed blame for the error. Otherwise, the peer who -originally provided the piece is probably at fault, since he is now -providing a later piece. This doesn't work if the differing piece is the -first piece, in which case it is downloaded from a 3rd peer, with -consensus revealing the misbehaving peer. - - -Consider storing torrent-like strings in the DHT. - -Instead of only storing the file download location (which would still be -used for small files), a bencoded dictionary containing the peer's -hashes of the individual pieces could be stored for the larger files -(20% of all the files are larger than 512 KB ). This dictionary would -have the download location, a list of the piece sizes, and a list of the -piece hashes (bittorrent uses a single string of length 20*#pieces, but -for general non-sha1 case a list is needed). - -These piece hashes could be compared ahead of time to determine which -peers have the same piece hashes (they all should), and then used during -the download to verify the downloaded pieces. - -Alternatively, the peers could store the torrent-like string for large -files separately, and only contain a reference to it in their stored -value for the hash of the file. The reference would be a hash of the -bencoded dictionary, and a lookup of that hash in the DHT would give the -torrent-like string. (A 100 MB file would result in 200 hashes, which -would create a bencoded dictionary larger than 6000 bytes.) - - -PeerManager needs to track peers' properties. - -The PeerManager needs to keep track of the observed properties of seen -peers, to help determine a selection criteria for choosing peers to -download from. Each property will give a value from 0 to 1. The relevant -properties are: - - - hash errors in last day (1 = 0, 0 = 3+) - - recent download speed (1 = fastest, 0 = 0) - - lag time from request to download (1 = 0, 0 = 15s+) - - number of pending requests (1 = 0, 0 = max (10)) - - whether a connection is open (1 = yes, 0.9 = no) - -These should be combined (multiplied) to provide a sort order for peers -available to download from, which can then be used to assign new -downloads to peers. Pieces should be downloaded from the best peers -first (i.e. piece 0 from the absolute best peer). - - -Missing Kademlia implementation details are needed. - -The current implementation is missing some important features, mostly -focussed on storing values: - - values need to be republished (every hour?) - - original publishers need to republish values (every 24 hours) - - when a new node is found that is closer to some values, replicate the - values there without deleting them - - when a value lookup succeeds, store the value in the closest node - found that didn't have it - - make the expiration time of a value exponentially inversely - proportional to the number of nodes between the current node and the - node closest to the value +Improve the downloaded and uploaded data measurements. + +There are 2 places that this data is measured: for statistics, and for +limiting the upload bandwidth. They both have deficiencies as they +sometimes miss the headers or the requests sent out. The upload +bandwidth calculation only considers the stream in the upload and not +the headers sent, and it also doesn't consider the upload bandwidth +from requesting downloads from peers (though that may be a good thing). +The statistics calculations for downloads include the headers of +downloaded files, but not the requests received from peers for upload +files. The statistics for uploaded data only includes the files sent +and not the headers, and also misses the requests for downloads sent to +other peers. + + +Consider storing deltas of packages. + +Instead of downloading full package files when a previous version of +the same package is available, peers could request a delta of the +package to the previous version. This would only be done if the delta +is significantly (>50%) smaller than the full package, and is not too +large (absolutely). A peer that has a new package and an old one would +add a list of deltas for the package to the value stored in the DHT. +The delta information would specify the old version (by hash), the +size of the delta, and the hash of the delta. A peer that has the same +old package could then download the delta from the peer by requesting +the hash of the delta. Alternatively, very small deltas could be +stored directly in the DHT. + + +Consider tracking security issues with packages. + +Since sharing information with others about what packages you have +downloaded (and probably installed) is a possible security +vulnerability, it would be advantageous to not share that information +for packages that have known security vulnerabilities. This would +require some way of obtaining a list of which packages (and versions) +are vulnerable, which is not currently available. + + +Consider adding peer characteristics to the DHT. + +Bad peers could be indicated in the DHT by adding a new value that is +the NOT of their ID (so they are guaranteed not to store it) indicating +information about the peer. This could be bad votes on the peer, as +otherwise a peer could add good info about itself. + + +Consider adding pieces to the DHT instead of files. + +Instead of adding file hashes to the DHT, only piece hashes could be +added. This would allow a peer to upload to other peers while it is +still downloading the rest of the file. It is not clear that this is +needed, since peer's will not be uploading and downloading ery much of +the time.