X-Git-Url: https://git.mxchange.org/?p=quix0rs-apt-p2p.git;a=blobdiff_plain;f=TODO;h=192670b54e3de79b2d0775c61e470c13ea13da79;hp=5f9555b75acc1b733bf3a03383e848e20135552e;hb=928a2934ab9cfc7386269b756e1ce815059be3a9;hpb=1461780708c6ce0562ec0a27062b13b32d7dc348 diff --git a/TODO b/TODO index 5f9555b..192670b 100644 --- a/TODO +++ b/TODO @@ -1,16 +1,28 @@ -Files for which a hash cannot be found should not be added to the DHT. +Evaluate and fix some errors in the ktable khashmir module. -If the hash can't found, it stands to reason that other peers will not -be able to find the hash either. So adding those files to the DHT will -just clutter it with useless information. Examples include Release.gpg, -Release, Translation-de.bz2, and Contents.gz. +The KTable implementation has some possible errors in it. insertNode +does not remove the original and use the new node when updating a node +already in the table, as claimed by the comments. justSeenNode doesn't +verify that the found node is the node that was being looked for, nor +does it move the node to the end of the list of nodes (since they are +supposed to be sorted by their lastSeen time) or update the bucket's +last touched time.nodeFailed also doesn't verify the found node is the +right node. -Use python-debian for parsing RFC 822 files. +Consider what happens when we are the closest node. -There are already routines for parsing these files, so there is no need -to write more. In the AptPackages, change the Release file parsing to -use the python-debian routines. +In some of the actions it is unclear what happens when we are one of the +closest nodes to the target key. Do we store values that we publish +ourself? + + +Add all cache files to the database. + +All files in the cache should be added to the database, so that they can +be checked to make sure nothing has happened to them. The database would +then need a flag to indicate files that are hashed and available, but +that shouldn't be added to the DHT. Packages.diff files need to be considered. @@ -49,60 +61,18 @@ first piece, in which case it is downloaded from a 3rd peer, with consensus revealing the misbehaving peer. -Store and share torrent-like strings for large files. - -In addition to storing the file download location (which would still be -used for small files), a bencoded dictionary containing the peer's -hashes of the individual pieces could be stored for the larger files -(20% of all the files are larger than 512 KB). This dictionary would -have the normal piece size, the hash length, and a string containing the -piece hashes of length *<#pieces>. These piece hashes could -be compared ahead of time to determine which peers have the same piece -hashes (they all should), and then used during the download to verify -the downloaded pieces. - -For very large files (5 or more pieces), the torrent strings are too -long to store in the DHT and retrieve (a single UDP packet should be -less than 1472 bytes to avoid fragmentation). Instead, the peers should -store the torrent-like string for large files separately, and only -contain a reference to it in their stored value for the hash of the -file. The reference would be a hash of the bencoded dictionary. If the -torrent-like string is short enough to store in the DHT (i.e. less than -1472 bytes, or about 70 pieces for the SHA1 hash), then a -lookup of that hash in the DHT would give the torrent-like string. -Otherwise, a request to the peer for the hash (just like files are -downloaded), should return the bencoded torrent-like string. - - -PeerManager needs to track peers' properties. - -The PeerManager needs to keep track of the observed properties of seen -peers, to help determine a selection criteria for choosing peers to -download from. Each property will give a value from 0 to 1. The relevant -properties are: - - - hash errors in last day (1 = 0, 0 = 3+) - - recent download speed (1 = fastest, 0 = 0) - - lag time from request to download (1 = 0, 0 = 15s+) - - number of pending requests (1 = 0, 0 = max (10)) - - whether a connection is open (1 = yes, 0.9 = no) - -These should be combined (multiplied) to provide a sort order for peers -available to download from, which can then be used to assign new -downloads to peers. Pieces should be downloaded from the best peers -first (i.e. piece 0 from the absolute best peer). - - -Missing Kademlia implementation details are needed. - -The current implementation is missing some important features, mostly -focussed on storing values: - - values need to be republished (every hour?) - - original publishers need to republish values (every 24 hours) - - when a new node is found that is closer to some values, replicate the - values there without deleting them - - when a value lookup succeeds, store the value in the closest node - found that didn't have it - - make the expiration time of a value exponentially inversely - proportional to the number of nodes between the current node and the - node closest to the value +Consider adding peer characteristics to the DHT. + +Bad peers could be indicated in the DHT by adding a new value that is +the NOT of their ID (so they are guaranteed not to store it) indicating +information about the peer. This could be bad votes on the peer, as +otherwise a peer could add good info about itself. + + +Consider adding pieces to the DHT instead of files. + +Instead of adding file hashes to the DHT, only piece hashes could be +added. This would allow a peer to upload to other peers while it is +still downloading the rest of the file. It is not clear that this is +needed, since peer's will not be uploading and downloading ery much of +the time.