Updates after more though on a couple of of the TODO items.

[quix0rs-apt-p2p.git] / TODO

diff --git a/TODO b/TODO
index 2be63a6c76e56cfd02ac2ccb64d6c4d9c01a1331..2ff301ac1065758b7d606da98e2ce582e1df02d9 100644 (file)
--- a/TODO
+++ b/TODO
@@ -15,21 +15,14 @@ distributions. They need to either be ignored, or dealt with properly by
 adding them to the tracking done by the AptPackages module.
 
 
 adding them to the tracking done by the AptPackages module.
 
 

-Hashes need to be sent with requests for some files.
+Change file identifier from path to hash.

 
 

-Some files can change without changing the file name, since the file was 
+Some files can change without changing the path, since the file was 

 added to the DHT by the peer. Examples are Release, Packages.gz, and 
 added to the DHT by the peer. Examples are Release, Packages.gz, and 

-Sources.bz2. For files like this (and only for files like this), the 
-request to download from the peer should include the downloader's 
-expected hash for the file as a new HTTP header. If the file is found, 
-the cached hash for the file will be used to determine whether the 
-request is for the same file as is currently available, and a special 
-HTTP response can be sent if it is not (i.e. not a 404).
-
-Alternatively, consider sharing the files by hash instead of by 
-directory. Then the request would be for
-http://127.3.45.9:9977/<urlencodedHash>, and it would always work. This 
-would require a database lookup for every request.
+Sources.bz2. This would cause problems when requesting these files by 
+path. Instead, share the files by hash, then the request would be for 
+http://127.3.45.9:9977/~<urlencodedHash>, and it would always work. This 
+will require a database lookup for every request.

 
 
 PeerManager needs to download large files from multiple peers.
 
 
 PeerManager needs to download large files from multiple peers.


@@ -59,26 +52,29 @@ first piece, in which case it is downloaded from a 3rd peer, with
 consensus revealing the misbehaving peer.
 
 
 consensus revealing the misbehaving peer.
 
 

-Consider storing torrent-like strings in the DHT.
+Store and share torrent-like strings for large files.

 
 

-Instead of only storing the file download location (which would still be 
+In addition to storing the file download location (which would still be 

 used for small files), a bencoded dictionary containing the peer's 
 hashes of the individual pieces could be stored for the larger files 
 used for small files), a bencoded dictionary containing the peer's 
 hashes of the individual pieces could be stored for the larger files 

-(20% of all the files are larger than 512 KB ). This dictionary would 
-have the download location, a list of the piece sizes, and a list of the 
-piece hashes (bittorrent uses a single string of length 20*#pieces, but 
-for general non-sha1 case a list is needed).
-
-These piece hashes could be compared ahead of time to determine which 
-peers have the same piece hashes (they all should), and then used during 
-the download to verify the downloaded pieces.
-
-Alternatively, the peers could store the torrent-like string for large 
-files separately, and only contain a reference to it in their stored 
-value for the hash of the file. The reference would be a hash of the 
-bencoded dictionary, and a lookup of that hash in the DHT would give the 
-torrent-like string. (A 100 MB file would result in 200 hashes, which 
-would create a bencoded dictionary larger than 6000 bytes.)
+(20% of all the files are larger than 512 KB). This dictionary would 
+have the normal piece size, the hash length, and a string containing the 
+piece hashes of length <hash length>*<#pieces>. These piece hashes could 
+be compared ahead of time to determine which peers have the same piece 
+hashes (they all should), and then used during the download to verify 
+the downloaded pieces.
+
+For very large files (5 or more pieces), the torrent strings are too 
+long to store in the DHT and retrieve (a single UDP packet should be 
+less than 1472 bytes to avoid fragmentation). Instead, the peers should 
+store the torrent-like string for large files separately, and only 
+contain a reference to it in their stored value for the hash of the 
+file. The reference would be a hash of the bencoded dictionary. If the 
+torrent-like string is short enough to store in the DHT (i.e. less than 
+1472 bytes, or about 70 pieces for the SHA1 hash), then a 
+lookup of that hash in the DHT would give the torrent-like string. 
+Otherwise, a request to the peer for the hash (just like files are 
+downloaded), should return the bencoded torrent-like string.

 
 
 PeerManager needs to track peers' properties.
 
 
 PeerManager needs to track peers' properties.