1 Comply with the newly defined protocol on the web page.
3 Various things need to done to comply with the newly defined protocol:
4 - use the compact encoding of contact information
5 - add the token to find_node responses
6 - use the token in store_node requests
7 - standardize the error messages (especially for a bad token)
10 Reduce the memory footprint by clearing the AptPackages caches.
12 The memory usage is a little bit high due to keeping the AptPackages
13 caches always. Instead, they should timeout after a period of inactivity
14 (say 15 minutes), and unload themselves from meory. It only takes a few
15 seconds to reload, so this should not be an issue.
18 Packages.diff files need to be considered.
20 The Packages.diff/Index files contain hashes of Packages.diff/rred.gz
21 files, which themselves contain diffs to the Packages files previously
22 downloaded. Apt will request these files for the testing/unstable
23 distributions. They need to either be ignored, or dealt with properly by
24 adding them to the tracking done by the AptPackages module.
27 PeerManager needs to download large files from multiple peers.
29 The PeerManager currently chooses a peer at random from the list of
30 possible peers, and downloads the entire file from there. This needs to
31 change if both a) the file is large (more than 512 KB), and b) there are
32 multiple peers with the file. The PeerManager should then break up the
33 large file into multiple pieces of size < 512 KB, and then send requests
34 to multiple peers for these pieces.
36 This can cause a problem with hash checking the returned data, as hashes
37 for the pieces are not known. Any file that fails a hash check should be
38 downloaded again, with each piece being downloaded from different peers
39 than it was previously. The peers are shifted by 1, so that if a peers
40 previously downloaded piece i, it now downloads piece i+1, and the first
41 piece is downloaded by the previous downloader of the last piece, or
42 preferably a previously unused peer. As each piece is downloaded the
43 running hash of the file should be checked to determine the place at
44 which the file differs from the previous download.
46 If the hash check then passes, then the peer who originally provided the
47 bad piece can be assessed blame for the error. Otherwise, the peer who
48 originally provided the piece is probably at fault, since he is now
49 providing a later piece. This doesn't work if the differing piece is the
50 first piece, in which case it is downloaded from a 3rd peer, with
51 consensus revealing the misbehaving peer.
54 Store and share torrent-like strings for large files.
56 In addition to storing the file download location (which would still be
57 used for small files), a bencoded dictionary containing the peer's
58 hashes of the individual pieces could be stored for the larger files
59 (20% of all the files are larger than 512 KB). This dictionary would
60 have the normal piece size, the hash length, and a string containing the
61 piece hashes of length <hash length>*<#pieces>. These piece hashes could
62 be compared ahead of time to determine which peers have the same piece
63 hashes (they all should), and then used during the download to verify
64 the downloaded pieces.
66 For very large files (5 or more pieces), the torrent strings are too
67 long to store in the DHT and retrieve (a single UDP packet should be
68 less than 1472 bytes to avoid fragmentation). Instead, the peers should
69 store the torrent-like string for large files separately, and only
70 contain a reference to it in their stored value for the hash of the
71 file. The reference would be a hash of the bencoded dictionary. If the
72 torrent-like string is short enough to store in the DHT (i.e. less than
73 1472 bytes, or about 70 pieces for the SHA1 hash), then a
74 lookup of that hash in the DHT would give the torrent-like string.
75 Otherwise, a request to the peer for the hash (just like files are
76 downloaded), should return the bencoded torrent-like string.
79 PeerManager needs to track peers' properties.
81 The PeerManager needs to keep track of the observed properties of seen
82 peers, to help determine a selection criteria for choosing peers to
83 download from. Each property will give a value from 0 to 1. The relevant
86 - hash errors in last day (1 = 0, 0 = 3+)
87 - recent download speed (1 = fastest, 0 = 0)
88 - lag time from request to download (1 = 0, 0 = 15s+)
89 - number of pending requests (1 = 0, 0 = max (10))
90 - whether a connection is open (1 = yes, 0.9 = no)
92 These should be combined (multiplied) to provide a sort order for peers
93 available to download from, which can then be used to assign new
94 downloads to peers. Pieces should be downloaded from the best peers
95 first (i.e. piece 0 from the absolute best peer).
98 When looking up values, DHT should return nodes and values.
100 When a key has multiple values in the DHT, returning a stored value may not
101 be sufficient, as then no more nodes can be contacted to get more stored
102 values. Instead, return both the stored values and the list of closest
103 nodes so that the peer doing the lookup can decide when to stop looking
104 (when it has received enough values).
106 Instead of returning both, a new method could be added, "lookup_value".
107 This method will be like "get_value", except that every node will always
108 return a list of nodes, as well as the number of values it has for that
109 key. Once a querying node has found enough values (or all of them), then
110 it would send the "get_value" method to the nodes that have the most
111 values. The "get_value" query could also have a new parameter "number",
112 which is the maximum number of values to return.
115 Missing Kademlia implementation details are needed.
117 The current implementation is missing some important features, mostly
118 focussed on storing values:
119 - values need to be republished (every hour?)