-Files for which a hash cannot be found should not be added to the DHT.
+Comply with the newly defined protocol on the web page.
-If the hash can't found, it stands to reason that other peers will not
-be able to find the hash either. So adding those files to the DHT will
-just clutter it with useless information. Examples include Release.gpg,
-Release, Translation-de.bz2, and Contents.gz.
+Various things need to done to comply with the newly defined protocol:
+ - use the compact encoding of contact information
+ - remove the originated time from the value storage
+ - add the token to find_node responses
+ - use the token in store_node requests
+ - standardize the error messages (especially for a bad token)
Packages.diff files need to be considered.
adding them to the tracking done by the AptPackages module.
-Hashes need to be sent with requests for some files.
-
-Some files can change without changing the file name, since the file was
-added to the DHT by the peer. Examples are Release, Packages.gz, and
-Sources.bz2. For files like this (and only for files like this), the
-request to download from the peer should include the downloader's
-expected hash for the file as a new HTTP header. If the file is found,
-the cached hash for the file will be used to determine whether the
-request is for the same file as is currently available, and a special
-HTTP response can be sent if it is not (i.e. not a 404).
-
-Alternatively, consider sharing the files by hash instead of by
-directory. Then the request would be for
-http://127.3.45.9:9977/<urlencodedHash>, and it would always work. This
-would require a database lookup for every request.
-
-
PeerManager needs to download large files from multiple peers.
The PeerManager currently chooses a peer at random from the list of
consensus revealing the misbehaving peer.
-Consider storing torrent-like strings in the DHT.
+Store and share torrent-like strings for large files.
-Instead of only storing the file download location (which would still be
+In addition to storing the file download location (which would still be
used for small files), a bencoded dictionary containing the peer's
hashes of the individual pieces could be stored for the larger files
-(20% of all the files are larger than 512 KB ). This dictionary would
-have the download location, a list of the piece sizes, and a list of the
-piece hashes (bittorrent uses a single string of length 20*#pieces, but
-for general non-sha1 case a list is needed).
-
-These piece hashes could be compared ahead of time to determine which
-peers have the same piece hashes (they all should), and then used during
-the download to verify the downloaded pieces.
-
-Alternatively, the peers could store the torrent-like string for large
-files separately, and only contain a reference to it in their stored
-value for the hash of the file. The reference would be a hash of the
-bencoded dictionary, and a lookup of that hash in the DHT would give the
-torrent-like string. (A 100 MB file would result in 200 hashes, which
-would create a bencoded dictionary larger than 6000 bytes.)
+(20% of all the files are larger than 512 KB). This dictionary would
+have the normal piece size, the hash length, and a string containing the
+piece hashes of length <hash length>*<#pieces>. These piece hashes could
+be compared ahead of time to determine which peers have the same piece
+hashes (they all should), and then used during the download to verify
+the downloaded pieces.
+
+For very large files (5 or more pieces), the torrent strings are too
+long to store in the DHT and retrieve (a single UDP packet should be
+less than 1472 bytes to avoid fragmentation). Instead, the peers should
+store the torrent-like string for large files separately, and only
+contain a reference to it in their stored value for the hash of the
+file. The reference would be a hash of the bencoded dictionary. If the
+torrent-like string is short enough to store in the DHT (i.e. less than
+1472 bytes, or about 70 pieces for the SHA1 hash), then a
+lookup of that hash in the DHT would give the torrent-like string.
+Otherwise, a request to the peer for the hash (just like files are
+downloaded), should return the bencoded torrent-like string.
PeerManager needs to track peers' properties.
first (i.e. piece 0 from the absolute best peer).
+When looking up values, DHT should return nodes and values.
+
+When a key has multiple values in the DHT, returning a stored value may not
+be sufficient, as then no more nodes can be contacted to get more stored
+values. Instead, return both the stored values and the list of closest
+nodes so that the peer doing the lookup can decide when to stop looking
+(when it has received enough values).
+
+Instead of returning both, a new method could be added, "lookup_value".
+This method will be like "get_value", except that every node will always
+return a list of nodes, as well as the number of values it has for that
+key. Once a querying node has found enough values (or all of them), then
+it would send the "get_value" method to the nodes that have the most
+values. The "get_value" query could also have a new parameter "number",
+which is the maximum number of values to return.
+
+
Missing Kademlia implementation details are needed.
The current implementation is missing some important features, mostly
focussed on storing values:
- values need to be republished (every hour?)
- - original publishers need to republish values (every 24 hours)
- - when a new node is found that is closer to some values, replicate the
- values there without deleting them
- - when a value lookup succeeds, store the value in the closest node
- found that didn't have it
- - make the expiration time of a value exponentially inversely
- proportional to the number of nodes between the current node and the
- node closest to the value
projects / quix0rs-apt-p2p.git / blobdiff