1 Consider what happens when we are the closest node.
3 In some of the actions it is unclear what happens when we are one of the
4 closest nodes to the target key. Do we store values that we publish
8 Add all cache files to the database.
10 All files in the cache should be added to the database, so that they can
11 be checked to make sure nothing has happened to them. The database would
12 then need a flag to indicate files that are hashed and available, but
13 that shouldn't be added to the DHT.
16 Packages.diff files need to be considered.
18 The Packages.diff/Index files contain hashes of Packages.diff/rred.gz
19 files, which themselves contain diffs to the Packages files previously
20 downloaded. Apt will request these files for the testing/unstable
21 distributions. They need to either be ignored, or dealt with properly by
22 adding them to the tracking done by the AptPackages module.
25 PeerManager needs to download large files from multiple peers.
27 The PeerManager currently chooses a peer at random from the list of
28 possible peers, and downloads the entire file from there. This needs to
29 change if both a) the file is large (more than 512 KB), and b) there are
30 multiple peers with the file. The PeerManager should then break up the
31 large file into multiple pieces of size < 512 KB, and then send requests
32 to multiple peers for these pieces.
34 This can cause a problem with hash checking the returned data, as hashes
35 for the pieces are not known. Any file that fails a hash check should be
36 downloaded again, with each piece being downloaded from different peers
37 than it was previously. The peers are shifted by 1, so that if a peers
38 previously downloaded piece i, it now downloads piece i+1, and the first
39 piece is downloaded by the previous downloader of the last piece, or
40 preferably a previously unused peer. As each piece is downloaded the
41 running hash of the file should be checked to determine the place at
42 which the file differs from the previous download.
44 If the hash check then passes, then the peer who originally provided the
45 bad piece can be assessed blame for the error. Otherwise, the peer who
46 originally provided the piece is probably at fault, since he is now
47 providing a later piece. This doesn't work if the differing piece is the
48 first piece, in which case it is downloaded from a 3rd peer, with
49 consensus revealing the misbehaving peer.
52 Store and share torrent-like strings for large files.
54 In addition to storing the file download location (which would still be
55 used for small files), a bencoded dictionary containing the peer's
56 hashes of the individual pieces could be stored for the larger files
57 (20% of all the files are larger than 512 KB). This dictionary would
58 have the normal piece size, the hash length, and a string containing the
59 piece hashes of length <hash length>*<#pieces>. These piece hashes could
60 be compared ahead of time to determine which peers have the same piece
61 hashes (they all should), and then used during the download to verify
62 the downloaded pieces.
64 For very large files (5 or more pieces), the torrent strings are too
65 long to store in the DHT and retrieve (a single UDP packet should be
66 less than 1472 bytes to avoid fragmentation). Instead, the peers should
67 store the torrent-like string for large files separately, and only
68 contain a reference to it in their stored value for the hash of the
69 file. The reference would be a hash of the bencoded dictionary. If the
70 torrent-like string is short enough to store in the DHT (i.e. less than
71 1472 bytes, or about 70 pieces for the SHA1 hash), then a
72 lookup of that hash in the DHT would give the torrent-like string.
73 Otherwise, a request to the peer for the hash (just like files are
74 downloaded), should return the bencoded torrent-like string.
77 PeerManager needs to track peers' properties.
79 The PeerManager needs to keep track of the observed properties of seen
80 peers, to help determine a selection criteria for choosing peers to
81 download from. Each property will give a value from 0 to 1. The relevant
84 - hash errors in last day (1 = 0, 0 = 3+)
85 - recent download speed (1 = fastest, 0 = 0)
86 - lag time from request to download (1 = 0, 0 = 15s+)
87 - number of pending requests (1 = 0, 0 = max (10))
88 - whether a connection is open (1 = yes, 0.9 = no)
90 These should be combined (multiplied) to provide a sort order for peers
91 available to download from, which can then be used to assign new
92 downloads to peers. Pieces should be downloaded from the best peers
93 first (i.e. piece 0 from the absolute best peer).