X-Git-Url: https://git.mxchange.org/?p=quix0rs-apt-p2p.git;a=blobdiff_plain;f=TODO;h=b0724b4bd35671f916b35b39b250cdcdbec93411;hp=ff65f9a12091ecfedd86a202f2082a108e6f1e57;hb=560d950422ec0610ee5ec855c1774bd73ccdd8cb;hpb=b68a1ebc0bbe79c8f9d3981ed83ef0ffcf201752 diff --git a/TODO b/TODO index ff65f9a..b0724b4 100644 --- a/TODO +++ b/TODO @@ -1,19 +1,9 @@ -Comply with the newly defined protocol on the web page. +Add all cache files to the database. -Various things need to done to comply with the newly defined protocol: - - use the compact encoding of contact information - - remove the originated time from the value storage - - add the token to find_node responses - - use the token in store_node requests - - standardize the error messages (especially for a bad token) - - -Reduce the memory footprint by clearing the AptPackages caches. - -The memory usage is a little bit high due to keeping the AptPackages -caches always. Instead, they should timeout after a period of inactivity -(say 15 minutes), and unload themselves from meory. It only takes a few -seconds to reload, so this should not be an issue. +All files in the cache should be added to the database, so that they can +be checked to make sure nothing has happened to them. The database would +then need a flag to indicate files that are hashed and available, but +that shouldn't be added to the DHT. Packages.diff files need to be considered. @@ -52,69 +42,43 @@ first piece, in which case it is downloaded from a 3rd peer, with consensus revealing the misbehaving peer. -Store and share torrent-like strings for large files. - -In addition to storing the file download location (which would still be -used for small files), a bencoded dictionary containing the peer's -hashes of the individual pieces could be stored for the larger files -(20% of all the files are larger than 512 KB). This dictionary would -have the normal piece size, the hash length, and a string containing the -piece hashes of length *<#pieces>. These piece hashes could -be compared ahead of time to determine which peers have the same piece -hashes (they all should), and then used during the download to verify -the downloaded pieces. - -For very large files (5 or more pieces), the torrent strings are too -long to store in the DHT and retrieve (a single UDP packet should be -less than 1472 bytes to avoid fragmentation). Instead, the peers should -store the torrent-like string for large files separately, and only -contain a reference to it in their stored value for the hash of the -file. The reference would be a hash of the bencoded dictionary. If the -torrent-like string is short enough to store in the DHT (i.e. less than -1472 bytes, or about 70 pieces for the SHA1 hash), then a -lookup of that hash in the DHT would give the torrent-like string. -Otherwise, a request to the peer for the hash (just like files are -downloaded), should return the bencoded torrent-like string. - - -PeerManager needs to track peers' properties. - -The PeerManager needs to keep track of the observed properties of seen -peers, to help determine a selection criteria for choosing peers to -download from. Each property will give a value from 0 to 1. The relevant -properties are: - - - hash errors in last day (1 = 0, 0 = 3+) - - recent download speed (1 = fastest, 0 = 0) - - lag time from request to download (1 = 0, 0 = 15s+) - - number of pending requests (1 = 0, 0 = max (10)) - - whether a connection is open (1 = yes, 0.9 = no) - -These should be combined (multiplied) to provide a sort order for peers -available to download from, which can then be used to assign new -downloads to peers. Pieces should be downloaded from the best peers -first (i.e. piece 0 from the absolute best peer). - - -When looking up values, DHT should return nodes and values. - -When a key has multiple values in the DHT, returning a stored value may not -be sufficient, as then no more nodes can be contacted to get more stored -values. Instead, return both the stored values and the list of closest -nodes so that the peer doing the lookup can decide when to stop looking -(when it has received enough values). - -Instead of returning both, a new method could be added, "lookup_value". -This method will be like "get_value", except that every node will always -return a list of nodes, as well as the number of values it has for that -key. Once a querying node has found enough values (or all of them), then -it would send the "get_value" method to the nodes that have the most -values. The "get_value" query could also have a new parameter "number", -which is the maximum number of values to return. - - -Missing Kademlia implementation details are needed. - -The current implementation is missing some important features, mostly -focussed on storing values: - - values need to be republished (every hour?) +Consider storing deltas of packages. + +Instead of downloading full package files when a previous version of +the same package is available, peers could request a delta of the +package to the previous version. This would only be done if the delta +is significantly (>50%) smaller than the full package, and is not too +large (absolutely). A peer that has a new package and an old one would +add a list of deltas for the package to the value stored in the DHT. +The delta information would specify the old version (by hash), the +size of the delta, and the hash of the delta. A peer that has the same +old package could then download the delta from the peer by requesting +the hash of the delta. Alternatively, very small deltas could be +stored directly in the DHT. + + +Consider tracking security issues with packages. + +Since sharing information with others about what packages you have +downloaded (and probably installed) is a possible security +vulnerability, it would be advantageous to not share that information +for packages that have known security vulnerabilities. This would +require some way of obtaining a list of which packages (and versions) +are vulnerable, which is not currently available. + + +Consider adding peer characteristics to the DHT. + +Bad peers could be indicated in the DHT by adding a new value that is +the NOT of their ID (so they are guaranteed not to store it) indicating +information about the peer. This could be bad votes on the peer, as +otherwise a peer could add good info about itself. + + +Consider adding pieces to the DHT instead of files. + +Instead of adding file hashes to the DHT, only piece hashes could be +added. This would allow a peer to upload to other peers while it is +still downloading the rest of the file. It is not clear that this is +needed, since peer's will not be uploading and downloading ery much of +the time.