X-Git-Url: https://git.mxchange.org/?p=quix0rs-apt-p2p.git;a=blobdiff_plain;f=TODO;h=ce33a7763381df0317bbc3f5e5ed78db22f6a579;hp=eca4e1325c26a7fb6cf758e470c4a8fb8ee32b0c;hb=c2be1eee6c0157ddcb1dc188c96711eaa21c7897;hpb=831e4f2adebf21b0427edde718668356a8df4837 diff --git a/TODO b/TODO index eca4e13..ce33a77 100644 --- a/TODO +++ b/TODO @@ -1,30 +1,95 @@ -Hashes of files need to be stored permanently. +Packages.diff files need to be considered. -A new database of files and their hashes is needed. It should store the -location and hash of the file as well as the modtime and other details -so we can check if a file needs to be rehashed on startup. The DB can -also be used to store info needed to manage the values stored in the DHT. +The Packages.diff/Index files contain hashes of Packages.diff/rred.gz +files, which themselves contain diffs to the Packages files previously +downloaded. Apt will request these files for the testing/unstable +distributions. They need to either be ignored, or dealt with properly by +adding them to the tracking done by the AptPackages module. -Change all print statements to log.msg() calls. +PeerManager needs to download large files from multiple peers. +The PeerManager currently chooses a peer at random from the list of +possible peers, and downloads the entire file from there. This needs to +change if both a) the file is large (more than 512 KB), and b) there are +multiple peers with the file. The PeerManager should then break up the +large file into multiple pieces of size < 512 KB, and then send requests +to multiple peers for these pieces. -Add ability to search and hash and DHT-store other directories. +This can cause a problem with hash checking the returned data, as hashes +for the pieces are not known. Any file that fails a hash check should be +downloaded again, with each piece being downloaded from different peers +than it was previously. The peers are shifted by 1, so that if a peers +previously downloaded piece i, it now downloads piece i+1, and the first +piece is downloaded by the previous downloader of the last piece, or +preferably a previously unused peer. As each piece is downloaded the +running hash of the file should be checked to determine the place at +which the file differs from the previous download. -The user should be able to specify a list of directories that will be -searched for files to hash and add to the DHT. +If the hash check then passes, then the peer who originally provided the +bad piece can be assessed blame for the error. Otherwise, the peer who +originally provided the piece is probably at fault, since he is now +providing a later piece. This doesn't work if the differing piece is the +first piece, in which case it is downloaded from a 3rd peer, with +consensus revealing the misbehaving peer. -Missing Kademlia implementation details are needed. +Store and share torrent-like strings for large files. -The current implementation is missing some important features, mostly -focussed on storing values: - - values need to be republished (every hour?) - - original publishers need to republish values (every 24 hours) - - when a new node is found that is closer to some values, replicate the - values there without deleting them - - when a value lookup succeeds, store the value in the closest node - found that didn't have it - - make the expiration time of a value exponentially inversely - proportional to the number of nodes between the current node and the - node closest to the value +In addition to storing the file download location (which would still be +used for small files), a bencoded dictionary containing the peer's +hashes of the individual pieces could be stored for the larger files +(20% of all the files are larger than 512 KB). This dictionary would +have the normal piece size, the hash length, and a string containing the +piece hashes of length *<#pieces>. These piece hashes could +be compared ahead of time to determine which peers have the same piece +hashes (they all should), and then used during the download to verify +the downloaded pieces. + +For very large files (5 or more pieces), the torrent strings are too +long to store in the DHT and retrieve (a single UDP packet should be +less than 1472 bytes to avoid fragmentation). Instead, the peers should +store the torrent-like string for large files separately, and only +contain a reference to it in their stored value for the hash of the +file. The reference would be a hash of the bencoded dictionary. If the +torrent-like string is short enough to store in the DHT (i.e. less than +1472 bytes, or about 70 pieces for the SHA1 hash), then a +lookup of that hash in the DHT would give the torrent-like string. +Otherwise, a request to the peer for the hash (just like files are +downloaded), should return the bencoded torrent-like string. + + +PeerManager needs to track peers' properties. + +The PeerManager needs to keep track of the observed properties of seen +peers, to help determine a selection criteria for choosing peers to +download from. Each property will give a value from 0 to 1. The relevant +properties are: + + - hash errors in last day (1 = 0, 0 = 3+) + - recent download speed (1 = fastest, 0 = 0) + - lag time from request to download (1 = 0, 0 = 15s+) + - number of pending requests (1 = 0, 0 = max (10)) + - whether a connection is open (1 = yes, 0.9 = no) + +These should be combined (multiplied) to provide a sort order for peers +available to download from, which can then be used to assign new +downloads to peers. Pieces should be downloaded from the best peers +first (i.e. piece 0 from the absolute best peer). + + +When looking up values, DHT should return nodes and values. + +When a key has multiple values in the DHT, returning a stored value may not +be sufficient, as then no more nodes can be contacted to get more stored +values. Instead, return both the stored values and the list of closest +nodes so that the peer doing the lookup can decide when to stop looking +(when it has received enough values). + +Instead of returning both, a new method could be added, "lookup_value". +This method will be like "get_value", except that every node will always +return a list of nodes, as well as the number of values it has for that +key. Once a querying node has found enough values (or all of them), then +it would send the "get_value" method to the nodes that have the most +values. The "get_value" query could also have a new parameter "number", +which is the maximum number of values to return.