]> git.mxchange.org Git - quix0rs-apt-p2p.git/log
quix0rs-apt-p2p.git
16 years agoAlways download max PIECE_SIZE from peers, even if there's only one peer.
Cameron Dale [Mon, 14 Apr 2008 01:02:39 +0000 (18:02 -0700)]
Always download max PIECE_SIZE from peers, even if there's only one peer.

Used to avoid ticket #1993 in Twisted:
http://twistedmatrix.com/trac/ticket/1993

16 years agoFix some tests and add some TODO items.
Cameron Dale [Sun, 13 Apr 2008 06:52:15 +0000 (23:52 -0700)]
Fix some tests and add some TODO items.

16 years agoActually add piece hashes to the database.
Cameron Dale [Sat, 12 Apr 2008 22:47:29 +0000 (15:47 -0700)]
Actually add piece hashes to the database.

16 years agoUpdated and added a lot of unittests.
Cameron Dale [Fri, 11 Apr 2008 23:21:18 +0000 (16:21 -0700)]
Updated and added a lot of unittests.

Still need to add tests for the Peer and Cache Managers.

16 years agoAlways use SHA1 hashes for pieces.
Cameron Dale [Fri, 11 Apr 2008 20:00:03 +0000 (13:00 -0700)]
Always use SHA1 hashes for pieces.

Only affects non-sha1 files like source files.
Also added a unit test for it.
Also added a unit test for the previous piece boundaries bug.

16 years agoMinor typo fixes in the Peer manager.
Cameron Dale [Fri, 11 Apr 2008 19:47:57 +0000 (12:47 -0700)]
Minor typo fixes in the Peer manager.

16 years agoMultiple peer downloading is mostly working now.
Cameron Dale [Fri, 11 Apr 2008 06:40:04 +0000 (23:40 -0700)]
Multiple peer downloading is mostly working now.

Many bug fixes, too numerous to mention.
Lots of new (probably temporary) logging.
New test stuff for the multi-peer downloading.
Still fails on large files when retrieving piece hashes from peers.

16 years agoFix a bug in the ThrottlingProtocol when writeSequence is called with a tuple.
Cameron Dale [Fri, 11 Apr 2008 06:34:40 +0000 (23:34 -0700)]
Fix a bug in the ThrottlingProtocol when writeSequence is called with a tuple.

16 years agoFix a bug in the Hash checker when updating exactly to a piece boundary.
Cameron Dale [Fri, 11 Apr 2008 06:33:30 +0000 (23:33 -0700)]
Fix a bug in the Hash checker when updating exactly to a piece boundary.

16 years agoFix some minor documentation typos.
Cameron Dale [Thu, 10 Apr 2008 00:32:35 +0000 (17:32 -0700)]
Fix some minor documentation typos.

16 years agoMore multiple peer downloading work, finished but still untested.
Cameron Dale [Thu, 10 Apr 2008 00:27:29 +0000 (17:27 -0700)]
More multiple peer downloading work, finished but still untested.

16 years agoMinor update to the multiple peer downloading (still not working).
Cameron Dale [Mon, 7 Apr 2008 05:51:24 +0000 (22:51 -0700)]
Minor update to the multiple peer downloading (still not working).

16 years agoMassive work on the peer downloading, not working or complete yet.
Cameron Dale [Fri, 4 Apr 2008 23:48:48 +0000 (16:48 -0700)]
Massive work on the peer downloading, not working or complete yet.

16 years agoWIP on multiple peer downloading (completely broken).
Cameron Dale [Wed, 2 Apr 2008 22:44:00 +0000 (15:44 -0700)]
WIP on multiple peer downloading (completely broken).

16 years agoFix a bug caused by twisted unquoting URLs received by the web server.
Cameron Dale [Tue, 1 Apr 2008 23:18:00 +0000 (16:18 -0700)]
Fix a bug caused by twisted unquoting URLs received by the web server.

16 years agoStore the peer rank as an attribute and recalculate it automatically.
Cameron Dale [Mon, 31 Mar 2008 23:41:01 +0000 (16:41 -0700)]
Store the peer rank as an attribute and recalculate it automatically.

16 years agoMake all the peer rankings exponentially decreasing.
Cameron Dale [Mon, 31 Mar 2008 23:21:15 +0000 (16:21 -0700)]
Make all the peer rankings exponentially decreasing.

16 years agoReturn 404s to all requests for Packages.diff and Sources.diff.
Cameron Dale [Mon, 31 Mar 2008 23:03:05 +0000 (16:03 -0700)]
Return 404s to all requests for Packages.diff and Sources.diff.

16 years agoMake the upload limit a config option.
Cameron Dale [Mon, 31 Mar 2008 22:47:15 +0000 (15:47 -0700)]
Make the upload limit a config option.

16 years agoOnly throttle uploads to peers, not to apt.
Cameron Dale [Mon, 31 Mar 2008 22:32:04 +0000 (15:32 -0700)]
Only throttle uploads to peers, not to apt.

16 years agoRetire one TODO and add another.
Cameron Dale [Mon, 31 Mar 2008 21:35:20 +0000 (14:35 -0700)]
Retire one TODO and add another.

16 years agoAllow the actions to call the local node's remote interface.
Cameron Dale [Thu, 27 Mar 2008 23:55:08 +0000 (16:55 -0700)]
Allow the actions to call the local node's remote interface.

The find_* actions now start with only the local node instead of
the nodes from the routing table.

16 years agoKRPC calls callback with the response dictionary by itself.
Cameron Dale [Thu, 27 Mar 2008 23:05:10 +0000 (16:05 -0700)]
KRPC calls callback with the response dictionary by itself.

Previously it was a dictionary, whose key 'rsp' contained the response dictionary.
The response dictionary has a key '_krpc_sender' added.

16 years agoReorganize the DHT's remote interface parameters.
Cameron Dale [Thu, 27 Mar 2008 22:00:50 +0000 (15:00 -0700)]
Reorganize the DHT's remote interface parameters.

Now the parameter ordering matches the one in the knode module.

Also makes the _krpc_sender optional so the remote interface can
be called by the local node.

16 years agoUse copies of the routing table nodes in actions.
Cameron Dale [Thu, 27 Mar 2008 00:09:04 +0000 (17:09 -0700)]
Use copies of the routing table nodes in actions.

The recursive actions need to use copies of the nodes in the routing
table, otherwise the token and num_values could be preset or set
twice by multiple actions.

16 years agoImport the log module, DOH.
Cameron Dale [Fri, 21 Mar 2008 05:17:32 +0000 (22:17 -0700)]
Import the log module, DOH.

16 years agoMerge small DHT buckets when a node is removed.
Cameron Dale [Fri, 21 Mar 2008 01:36:33 +0000 (18:36 -0700)]
Merge small DHT buckets when a node is removed.

16 years agoWorkaround old sqlite not having 'select count(distinct key)'.
Cameron Dale [Thu, 20 Mar 2008 22:47:14 +0000 (15:47 -0700)]
Workaround old sqlite not having 'select count(distinct key)'.

16 years agoRemove the duplicate error logging from the knode module.
Cameron Dale [Thu, 20 Mar 2008 22:26:41 +0000 (15:26 -0700)]
Remove the duplicate error logging from the knode module.

16 years agoRemove the unused steveholt entry from the bootstrap list.
Cameron Dale [Thu, 20 Mar 2008 22:12:29 +0000 (15:12 -0700)]
Remove the unused steveholt entry from the bootstrap list.

16 years agoFix some bugs in the pinging and ID checking.
Cameron Dale [Thu, 20 Mar 2008 22:09:29 +0000 (15:09 -0700)]
Fix some bugs in the pinging and ID checking.

Raise an exception if the node's ID has changed. This means the node's
returned results will be ignored.

The KTable replaceStaleNode now adds the new node if there's space even
if the old one isn't found. This is due to the node ID having changed
resulting in instantly removing the node from the routing table.

The handler for the ping result in insertNode was mistakenly taking the
Failure returned as a result as a node and passing it to replaceStaleNode.

16 years agoMove the HTML stats page generation for the DHT into the stats module.
Cameron Dale [Mon, 17 Mar 2008 23:43:30 +0000 (16:43 -0700)]
Move the HTML stats page generation for the DHT into the stats module.

16 years agoRename the knode functions to match the krpc method names.
Cameron Dale [Mon, 17 Mar 2008 22:27:56 +0000 (15:27 -0700)]
Rename the knode functions to match the krpc method names.

16 years agoAdded the number of times each action was started to the DHT stats.
Cameron Dale [Mon, 17 Mar 2008 22:21:26 +0000 (15:21 -0700)]
Added the number of times each action was started to the DHT stats.

16 years agoAdd bytes/sec to the DHT stats.
Cameron Dale [Mon, 17 Mar 2008 21:07:59 +0000 (14:07 -0700)]
Add bytes/sec to the DHT stats.

16 years agoDisplay DHT statistics to the HTTP user.
Cameron Dale [Fri, 14 Mar 2008 23:40:05 +0000 (16:40 -0700)]
Display DHT statistics to the HTTP user.

Adds 2 new interfaces to the DHT implementation:
 * DHTStats: DHT that supports gathering statistics
 * DHTStatsFactory: DHT that supports creating it's own HTTP server
apt_dht_Khashmir supports IDHTStats and formats stats for HTML.
apt_dht_Khashmir also supports IDHTStatsFactory if twisted.web2 is found.
Main script passes the DHT class to the main program.
Main script (DHT only) starts the stats factory if the DHT supports it.
Main program retrieves statistics from the DHT when asked by the HTTPServer.

16 years agoStart using the new DHT stats module (untested).
Cameron Dale [Wed, 12 Mar 2008 06:40:09 +0000 (23:40 -0700)]
Start using the new DHT stats module (untested).

16 years agoAdd the new stats module to the DHT.
Cameron Dale [Wed, 12 Mar 2008 06:39:29 +0000 (23:39 -0700)]
Add the new stats module to the DHT.

16 years agoFix minor bugs in the use of sha module and misspelled b2a_hex.
Cameron Dale [Tue, 11 Mar 2008 18:40:43 +0000 (11:40 -0700)]
Fix minor bugs in the use of sha module and misspelled b2a_hex.

16 years agoFixed justSeenNode in KTable to update the bucket properly.
Cameron Dale [Sat, 8 Mar 2008 03:56:17 +0000 (19:56 -0800)]
Fixed justSeenNode in KTable to update the bucket properly.

It now moves the seen node to the end and marks the bucket as
having been updated.

16 years agoAnother TODO item.
Cameron Dale [Sat, 8 Mar 2008 00:38:33 +0000 (16:38 -0800)]
Another TODO item.

16 years agoDon't use the tables findNodes when you can just bisect for it.
Cameron Dale [Sat, 8 Mar 2008 00:37:10 +0000 (16:37 -0800)]
Don't use the tables findNodes when you can just bisect for it.

16 years agoAdd new node to table when refreshing old node.
Cameron Dale [Sat, 8 Mar 2008 00:06:57 +0000 (16:06 -0800)]
Add new node to table when refreshing old node.

Fixes a bug introduced by commit:

    6ce38471: fix peer update problem connection refused in store value

16 years agoRename all apt-dht files to apt-p2p.
Cameron Dale [Thu, 6 Mar 2008 00:17:03 +0000 (16:17 -0800)]
Rename all apt-dht files to apt-p2p.

16 years agoRename project to apt-p2p.
Cameron Dale [Thu, 6 Mar 2008 00:10:07 +0000 (16:10 -0800)]
Rename project to apt-p2p.

16 years agoMerge branch 'documentation' into apt-dht
Cameron Dale [Wed, 5 Mar 2008 23:06:44 +0000 (15:06 -0800)]
Merge branch 'documentation' into apt-dht

16 years agoVarious documentation fixes and additions.
Cameron Dale [Wed, 5 Mar 2008 23:04:50 +0000 (15:04 -0800)]
Various documentation fixes and additions.

16 years agoDocument the DHT package.
Cameron Dale [Wed, 5 Mar 2008 23:03:33 +0000 (15:03 -0800)]
Document the DHT package.

16 years agoDocument the DHT's main khashmir module.
Cameron Dale [Wed, 5 Mar 2008 23:03:11 +0000 (15:03 -0800)]
Document the DHT's main khashmir module.

16 years agoDocument the DHT's krpc module.
Cameron Dale [Wed, 5 Mar 2008 23:02:44 +0000 (15:02 -0800)]
Document the DHT's krpc module.

16 years agoMerge branch 'apt-dht' of ssh://git.camrdale.org/git/apt-dht into apt-dht
Cameron Dale [Wed, 5 Mar 2008 00:05:22 +0000 (16:05 -0800)]
Merge branch 'apt-dht' of ssh://git.camrdale.org/git/apt-dht into apt-dht

16 years agoMore todo items.
Cameron Dale [Wed, 5 Mar 2008 00:04:40 +0000 (16:04 -0800)]
More todo items.

16 years agoDocument the DHT's util module.
Cameron Dale [Wed, 5 Mar 2008 00:04:12 +0000 (16:04 -0800)]
Document the DHT's util module.

16 years agoDocument the DHT's node module.
Cameron Dale [Wed, 5 Mar 2008 00:03:58 +0000 (16:03 -0800)]
Document the DHT's node module.

16 years agoDocument the DHT's ktable module.
Cameron Dale [Wed, 5 Mar 2008 00:03:45 +0000 (16:03 -0800)]
Document the DHT's ktable module.

16 years agoDocument the DHT's knode module.
Cameron Dale [Wed, 5 Mar 2008 00:03:28 +0000 (16:03 -0800)]
Document the DHT's knode module.

16 years agoDocument the DHT's khash module.
Cameron Dale [Wed, 5 Mar 2008 00:03:16 +0000 (16:03 -0800)]
Document the DHT's khash module.

16 years agoDocument the DHT's db module.
Cameron Dale [Wed, 5 Mar 2008 00:02:58 +0000 (16:02 -0800)]
Document the DHT's db module.

16 years agoDocument the DHT's actions module.
Cameron Dale [Wed, 5 Mar 2008 00:02:43 +0000 (16:02 -0800)]
Document the DHT's actions module.

16 years agoDocument the DHT's DHT module.
Cameron Dale [Wed, 5 Mar 2008 00:02:27 +0000 (16:02 -0800)]
Document the DHT's DHT module.

16 years agoAdd cleanup to the MirrorManager for the AptPackages.
Cameron Dale [Sun, 2 Mar 2008 20:40:17 +0000 (12:40 -0800)]
Add cleanup to the MirrorManager for the AptPackages.

16 years agoFix a minor error in the DB's storeFile function.
Cameron Dale [Sun, 2 Mar 2008 20:26:16 +0000 (12:26 -0800)]
Fix a minor error in the DB's storeFile function.

16 years agoA few more documentation fixes.
Cameron Dale [Sat, 1 Mar 2008 02:46:06 +0000 (18:46 -0800)]
A few more documentation fixes.

16 years agoDocumented the util module.
Cameron Dale [Sat, 1 Mar 2008 02:44:56 +0000 (18:44 -0800)]
Documented the util module.

16 years agoDocumented the db module.
Cameron Dale [Sat, 1 Mar 2008 02:44:40 +0000 (18:44 -0800)]
Documented the db module.

16 years agoDocumented the PeerManager module.
Cameron Dale [Sat, 1 Mar 2008 02:44:23 +0000 (18:44 -0800)]
Documented the PeerManager module.

16 years agoDocumented the Hash module.
Cameron Dale [Sat, 1 Mar 2008 02:43:58 +0000 (18:43 -0800)]
Documented the Hash module.

16 years agoDocumented the HTTPServer module.
Cameron Dale [Sat, 1 Mar 2008 02:43:41 +0000 (18:43 -0800)]
Documented the HTTPServer module.

16 years agoDocumented the HTTPDownloader module.
Cameron Dale [Sat, 1 Mar 2008 02:43:12 +0000 (18:43 -0800)]
Documented the HTTPDownloader module.

16 years agoDocument the MirrorManager module.
Cameron Dale [Fri, 29 Feb 2008 23:40:29 +0000 (15:40 -0800)]
Document the MirrorManager module.

16 years agoDocument the AptPackages module.
Cameron Dale [Fri, 29 Feb 2008 23:40:11 +0000 (15:40 -0800)]
Document the AptPackages module.

16 years agoDocument the apt_dht module.
Cameron Dale [Fri, 29 Feb 2008 21:42:44 +0000 (13:42 -0800)]
Document the apt_dht module.

16 years agoDocument the apt_dht_conf module.
Cameron Dale [Fri, 29 Feb 2008 21:42:28 +0000 (13:42 -0800)]
Document the apt_dht_conf module.

16 years agoDocument the apt_dht module's __init__ file.
Cameron Dale [Fri, 29 Feb 2008 21:41:50 +0000 (13:41 -0800)]
Document the apt_dht module's __init__ file.

16 years agoMove the normalization of key lengths from the HashObject to the DHT.
Cameron Dale [Fri, 29 Feb 2008 21:10:46 +0000 (13:10 -0800)]
Move the normalization of key lengths from the HashObject to the DHT.

16 years agoAdd property tracking to downloads from peers.
Cameron Dale [Fri, 29 Feb 2008 04:54:45 +0000 (20:54 -0800)]
Add property tracking to downloads from peers.

16 years agoHTTPServer responds correctly to requests for torrent strings.
Cameron Dale [Fri, 29 Feb 2008 00:50:10 +0000 (16:50 -0800)]
HTTPServer responds correctly to requests for torrent strings.

16 years agoUpgrade policies to SVN version and fix a small bug.
Cameron Dale [Fri, 29 Feb 2008 00:17:22 +0000 (16:17 -0800)]
Upgrade policies to SVN version and fix a small bug.

Fixed the policies usage when writeLimit was None to not do anything.

16 years agoStore piece hashes in the DB.
Cameron Dale [Wed, 27 Feb 2008 23:58:05 +0000 (15:58 -0800)]
Store piece hashes in the DB.

Also broke up single table into files and hashes.
Main code uses the new hash routine.
lookupHash now checks for torrent string hashes.
The HTTPServer still needs to interpret torrent string hashes.

16 years agoAdd appropriate piece strings to the DHT.
Cameron Dale [Wed, 27 Feb 2008 20:05:30 +0000 (12:05 -0800)]
Add appropriate piece strings to the DHT.

16 years agoAdded piece hashing to the HashObject.
Cameron Dale [Wed, 27 Feb 2008 19:42:54 +0000 (11:42 -0800)]
Added piece hashing to the HashObject.

16 years agoOnly use the ThrottlingProtocol if the producer is streaming.
Cameron Dale [Wed, 27 Feb 2008 07:05:00 +0000 (23:05 -0800)]
Only use the ThrottlingProtocol if the producer is streaming.

16 years agoFixed the ThrottlingFactory to work with web2 static streams from the web server.
Cameron Dale [Wed, 27 Feb 2008 06:42:23 +0000 (22:42 -0800)]
Fixed the ThrottlingFactory to work with web2 static streams from the web server.

16 years agoAnother attempt at throttling, still not working.
Cameron Dale [Tue, 26 Feb 2008 01:10:58 +0000 (17:10 -0800)]
Another attempt at throttling, still not working.

16 years agoA new TODO item.
Cameron Dale [Mon, 25 Feb 2008 07:04:39 +0000 (23:04 -0800)]
A new TODO item.

16 years agoAlso remove changed cache files during directory scan.
Cameron Dale [Sun, 24 Feb 2008 23:14:00 +0000 (15:14 -0800)]
Also remove changed cache files during directory scan.

16 years agoIgnore unknown cache files during directory scan.
Cameron Dale [Sun, 24 Feb 2008 23:13:15 +0000 (15:13 -0800)]
Ignore unknown cache files during directory scan.

16 years agoStandardize the number of values retrieved from the DHT.
Cameron Dale [Sun, 24 Feb 2008 22:01:13 +0000 (14:01 -0800)]
Standardize the number of values retrieved from the DHT.

Added the new RETRIEVE_VALUES configuration parameter.
The desired number of values can now be negative, indicating
that only the closest STORE_REDUNDANCY nodes are to be queried.

16 years agoRewrite of the actions to take advantage of the commonalities between them.
Cameron Dale [Sun, 24 Feb 2008 08:31:45 +0000 (00:31 -0800)]
Rewrite of the actions to take advantage of the commonalities between them.

16 years agoCheck response packet lengths before sending.
Cameron Dale [Sun, 24 Feb 2008 04:30:26 +0000 (20:30 -0800)]
Check response packet lengths before sending.

16 years agoRemove an unneeded khashmir file.
Cameron Dale [Sun, 24 Feb 2008 01:37:44 +0000 (17:37 -0800)]
Remove an unneeded khashmir file.

16 years agoBreak up the find_value into 2 parts (with get_value).
Cameron Dale [Sat, 23 Feb 2008 02:44:13 +0000 (18:44 -0800)]
Break up the find_value into 2 parts (with get_value).

To better support multiple values per key, the old find_value request is
broken up into find_value, which doesn't stop when it finds a node with
values but just records the number of values the node has, and
get_value, which queries the found nodes that have values to retrieve
them.

16 years agoRefresh DHT values just before they are due to expire.
Cameron Dale [Fri, 22 Feb 2008 23:30:57 +0000 (15:30 -0800)]
Refresh DHT values just before they are due to expire.

16 years agoMove the key expiring to the checkpoint function.
Cameron Dale [Fri, 22 Feb 2008 19:33:50 +0000 (11:33 -0800)]
Move the key expiring to the checkpoint function.

16 years agoUnload the AptPackages caches after a period of inactivity.
Cameron Dale [Fri, 22 Feb 2008 19:22:28 +0000 (11:22 -0800)]
Unload the AptPackages caches after a period of inactivity.

16 years agoUpdated the copyright definitions.
Cameron Dale [Fri, 22 Feb 2008 18:55:16 +0000 (10:55 -0800)]
Updated the copyright definitions.

16 years agoMore and better error messages in the DHT.
Cameron Dale [Fri, 22 Feb 2008 02:40:56 +0000 (18:40 -0800)]
More and better error messages in the DHT.

Also added message verification for the incoming message
to prevent abuse.

16 years agoReturn a token in find_node responses, use it in store_value requests.
Cameron Dale [Thu, 21 Feb 2008 23:18:44 +0000 (15:18 -0800)]
Return a token in find_node responses, use it in store_value requests.

Also changed the checkpointing frequency to 5 minutes so that storing 3
old tokens will give an age of 10-15 minutes (as per protocol).

16 years agoUse compact encoding of peer info for downloads.
Cameron Dale [Thu, 21 Feb 2008 22:27:21 +0000 (14:27 -0800)]
Use compact encoding of peer info for downloads.

16 years agoRemove some extra logging that shouldn't have been in the last commit.
Cameron Dale [Thu, 21 Feb 2008 21:43:03 +0000 (13:43 -0800)]
Remove some extra logging that shouldn't have been in the last commit.