-Change all print statements to log.msg() calls.
+Comply with the newly defined protocol on the web page.
+Various things need to done to comply with the newly defined protocol:
+ - use the compact encoding of contact information
+ - remove the originated time from the value storage
+ - add the token to find_node responses
+ - use the token in store_node requests
+ - standardize the error messages (especially for a bad token)
-Add ability to search and hash and DHT-store other directories.
-The user should be able to specify a list of directories that will be
-searched for files to hash and add to the DHT.
+Packages.diff files need to be considered.
+
+The Packages.diff/Index files contain hashes of Packages.diff/rred.gz
+files, which themselves contain diffs to the Packages files previously
+downloaded. Apt will request these files for the testing/unstable
+distributions. They need to either be ignored, or dealt with properly by
+adding them to the tracking done by the AptPackages module.
+
+
+PeerManager needs to download large files from multiple peers.
+
+The PeerManager currently chooses a peer at random from the list of
+possible peers, and downloads the entire file from there. This needs to
+change if both a) the file is large (more than 512 KB), and b) there are
+multiple peers with the file. The PeerManager should then break up the
+large file into multiple pieces of size < 512 KB, and then send requests
+to multiple peers for these pieces.
+
+This can cause a problem with hash checking the returned data, as hashes
+for the pieces are not known. Any file that fails a hash check should be
+downloaded again, with each piece being downloaded from different peers
+than it was previously. The peers are shifted by 1, so that if a peers
+previously downloaded piece i, it now downloads piece i+1, and the first
+piece is downloaded by the previous downloader of the last piece, or
+preferably a previously unused peer. As each piece is downloaded the
+running hash of the file should be checked to determine the place at
+which the file differs from the previous download.
+
+If the hash check then passes, then the peer who originally provided the
+bad piece can be assessed blame for the error. Otherwise, the peer who
+originally provided the piece is probably at fault, since he is now
+providing a later piece. This doesn't work if the differing piece is the
+first piece, in which case it is downloaded from a 3rd peer, with
+consensus revealing the misbehaving peer.
+
+
+Store and share torrent-like strings for large files.
+
+In addition to storing the file download location (which would still be
+used for small files), a bencoded dictionary containing the peer's
+hashes of the individual pieces could be stored for the larger files
+(20% of all the files are larger than 512 KB). This dictionary would
+have the normal piece size, the hash length, and a string containing the
+piece hashes of length <hash length>*<#pieces>. These piece hashes could
+be compared ahead of time to determine which peers have the same piece
+hashes (they all should), and then used during the download to verify
+the downloaded pieces.
+
+For very large files (5 or more pieces), the torrent strings are too
+long to store in the DHT and retrieve (a single UDP packet should be
+less than 1472 bytes to avoid fragmentation). Instead, the peers should
+store the torrent-like string for large files separately, and only
+contain a reference to it in their stored value for the hash of the
+file. The reference would be a hash of the bencoded dictionary. If the
+torrent-like string is short enough to store in the DHT (i.e. less than
+1472 bytes, or about 70 pieces for the SHA1 hash), then a
+lookup of that hash in the DHT would give the torrent-like string.
+Otherwise, a request to the peer for the hash (just like files are
+downloaded), should return the bencoded torrent-like string.
+
+
+PeerManager needs to track peers' properties.
+
+The PeerManager needs to keep track of the observed properties of seen
+peers, to help determine a selection criteria for choosing peers to
+download from. Each property will give a value from 0 to 1. The relevant
+properties are:
+
+ - hash errors in last day (1 = 0, 0 = 3+)
+ - recent download speed (1 = fastest, 0 = 0)
+ - lag time from request to download (1 = 0, 0 = 15s+)
+ - number of pending requests (1 = 0, 0 = max (10))
+ - whether a connection is open (1 = yes, 0.9 = no)
+
+These should be combined (multiplied) to provide a sort order for peers
+available to download from, which can then be used to assign new
+downloads to peers. Pieces should be downloaded from the best peers
+first (i.e. piece 0 from the absolute best peer).
+
+
+When looking up values, DHT should return nodes and values.
+
+When a key has multiple values in the DHT, returning a stored value may not
+be sufficient, as then no more nodes can be contacted to get more stored
+values. Instead, return both the stored values and the list of closest
+nodes so that the peer doing the lookup can decide when to stop looking
+(when it has received enough values).
+
+Instead of returning both, a new method could be added, "lookup_value".
+This method will be like "get_value", except that every node will always
+return a list of nodes, as well as the number of values it has for that
+key. Once a querying node has found enough values (or all of them), then
+it would send the "get_value" method to the nodes that have the most
+values. The "get_value" query could also have a new parameter "number",
+which is the maximum number of values to return.
Missing Kademlia implementation details are needed.
The current implementation is missing some important features, mostly
focussed on storing values:
- values need to be republished (every hour?)
- - original publishers need to republish values (every 24 hours)
- - when a new node is found that is closer to some values, replicate the
- values there without deleting them
- - when a value lookup succeeds, store the value in the closest node
- found that didn't have it
- - make the expiration time of a value exponentially inversely
- proportional to the number of nodes between the current node and the
- node closest to the value
projects / quix0rs-apt-p2p.git / blobdiff