12 years agoBetter handling and logging for intermittent HTTP client submission errors.
Cameron Dale [Thu, 24 Apr 2008 16:54:31 +0000 (09:54 -0700)]
Better handling and logging for intermittent HTTP client submission errors.

12 years agoDon't add attributes to HTTP requests, store them separately.
Cameron Dale [Thu, 24 Apr 2008 16:33:34 +0000 (09:33 -0700)]
Don't add attributes to HTTP requests, store them separately.

12 years agoHTTP client no longer keeps a response queue of requests.
Cameron Dale [Thu, 24 Apr 2008 05:02:43 +0000 (22:02 -0700)]
HTTP client no longer keeps a response queue of requests.

Instead the requests are added as variables to the call/errbacks.
There is a new variable keeping track of how many responses are outstanding.
clientGone no longer remove/errback responses, as the errbacks will be
called by the client protocol instead.

12 years agoFix a typo in commit 80399176ed66d.
Cameron Dale [Thu, 24 Apr 2008 04:41:38 +0000 (21:41 -0700)]
Fix a typo in commit 80399176ed66d.

12 years agoFixed an error in downloading source packages from peers.
Cameron Dale [Thu, 24 Apr 2008 03:52:11 +0000 (20:52 -0700)]
Fixed an error in downloading source packages from peers.

Source packages with no pieces were hash checked with sha1, which would fail.
Also added a test for source packages.
Also removed some unneeded logging in the peer downloader.

12 years agoFix a typo in commit e82e704e27.
Cameron Dale [Thu, 24 Apr 2008 03:50:06 +0000 (20:50 -0700)]
Fix a typo in commit e82e704e27.

12 years agoSet the Last-Modified header when downloading from peers.
Cameron Dale [Wed, 23 Apr 2008 23:12:29 +0000 (16:12 -0700)]
Set the Last-Modified header when downloading from peers.

12 years agoRemove temporary files from the peer's download cache.
Cameron Dale [Wed, 23 Apr 2008 22:50:14 +0000 (15:50 -0700)]
Remove temporary files from the peer's download cache.

12 years agoRefresh expired DHT hashes concurrently instead of sequentially.
Cameron Dale [Wed, 23 Apr 2008 22:30:39 +0000 (15:30 -0700)]
Refresh expired DHT hashes concurrently instead of sequentially.

12 years agoDon't check file when listing expired hashes, instead check files at startup.
Cameron Dale [Wed, 23 Apr 2008 21:31:03 +0000 (14:31 -0700)]
Don't check file when listing expired hashes, instead check files at startup.

12 years agoTrap KRPC errors in writing to the transport and wait for timeout instead.
Cameron Dale [Wed, 23 Apr 2008 03:16:21 +0000 (20:16 -0700)]
Trap KRPC errors in writing to the transport and wait for timeout instead.

12 years agoDisconnect the insertNode calls from the callers so errors don't affect them.
Cameron Dale [Wed, 23 Apr 2008 02:10:21 +0000 (19:10 -0700)]
Disconnect the insertNode calls from the callers so errors don't affect them.

12 years agoRemove some unnecessary calls to log.err and just print the error message instead.
Cameron Dale [Mon, 21 Apr 2008 21:15:14 +0000 (14:15 -0700)]
Remove some unnecessary calls to log.err and just print the error message instead.

12 years agoFix a bug in loading the new KRPC timeout config params.
Cameron Dale [Mon, 21 Apr 2008 20:08:13 +0000 (13:08 -0700)]
Fix a bug in loading the new KRPC timeout config params.

12 years agoMake the DHT timeouts configuration parameters.
Cameron Dale [Mon, 21 Apr 2008 19:41:24 +0000 (12:41 -0700)]
Make the DHT timeouts configuration parameters.

12 years agoConf time parser works with floats, and lengthened DHT hash expiry to 3 hours.
Cameron Dale [Mon, 21 Apr 2008 19:05:56 +0000 (12:05 -0700)]
Conf time parser works with floats, and lengthened DHT hash expiry to 3 hours.

12 years agoFix an error in the actions that allowed for the result to be sent twice.
Cameron Dale [Mon, 21 Apr 2008 18:54:05 +0000 (11:54 -0700)]
Fix an error in the actions that allowed for the result to be sent twice.

12 years agoSome more TODO items.
Cameron Dale [Mon, 21 Apr 2008 05:55:40 +0000 (22:55 -0700)]
Some more TODO items.

12 years agoApt packages cache now saves an unload request received during loading.
Cameron Dale [Mon, 21 Apr 2008 05:09:34 +0000 (22:09 -0700)]
Apt packages cache now saves an unload request received during loading.

12 years agoSync the AptPackages db after modification, and close when done.
Cameron Dale [Mon, 21 Apr 2008 02:47:31 +0000 (19:47 -0700)]
Sync the AptPackages db after modification, and close when done.

12 years agoAdd all files to the DB with their hashes.
Cameron Dale [Mon, 21 Apr 2008 02:42:31 +0000 (19:42 -0700)]
Add all files to the DB with their hashes.

Non-DHT files are marked as such.
HTTP server looks up the cache path to decide whether to return a file.

12 years agoFix an HTTP download bug that caused HEAD requests to not pipeline.
Cameron Dale [Mon, 21 Apr 2008 02:39:33 +0000 (19:39 -0700)]
Fix an HTTP download bug that caused HEAD requests to not pipeline.

Also improve the testing for this.

12 years agoReorder the main application to find cached hashes before checking freshness.
Cameron Dale [Sat, 19 Apr 2008 18:39:24 +0000 (11:39 -0700)]
Reorder the main application to find cached hashes before checking freshness.

Eliminates some unnecessary HEAD requests to the server.
HEAD requests are only done if there is no hash for the file.

12 years agoAdd a new test and improve the cleaning of apt files.
Cameron Dale [Sat, 19 Apr 2008 17:48:47 +0000 (10:48 -0700)]
Add a new test and improve the cleaning of apt files.

12 years agoDHT stats now include completed actions and various elapsed times for actions.
Cameron Dale [Fri, 18 Apr 2008 21:03:26 +0000 (14:03 -0700)]
DHT stats now include completed actions and various elapsed times for actions.

12 years agoReturn 404's for hashes that aren't found.
Cameron Dale [Fri, 18 Apr 2008 05:13:17 +0000 (22:13 -0700)]
Return 404's for hashes that aren't found.

12 years agoRemove some unnecessary log messages and use better Exceptions.
Cameron Dale [Fri, 18 Apr 2008 02:14:05 +0000 (19:14 -0700)]
Remove some unnecessary log messages and use better Exceptions.

12 years agoCache returns errors if apt closes the connection and deletes the incomplete files.
Cameron Dale [Fri, 18 Apr 2008 02:05:45 +0000 (19:05 -0700)]
Cache returns errors if apt closes the connection and deletes the incomplete files.

12 years agoRemove some headers Apt sets before returning full requests from the server.
Cameron Dale [Fri, 18 Apr 2008 01:15:35 +0000 (18:15 -0700)]
Remove some headers Apt sets before returning full requests from the server.

Otherwise the streaming will be interrupted or changed by Twisted's
modification of responses before sending (e.g. sending 304's based
on 'If-Modified-Since' and 'Last-Modified' headers). The streaming
is currently needed though to load the file in the cache.

12 years agoBetter identification of peer uploads by the HTTP server.
Cameron Dale [Fri, 18 Apr 2008 01:12:54 +0000 (18:12 -0700)]
Better identification of peer uploads by the HTTP server.

12 years agoAlways try and find the mirror site, even if no updated files have been saved.
Cameron Dale [Thu, 17 Apr 2008 19:58:59 +0000 (12:58 -0700)]
Always try and find the mirror site, even if no updated files have been saved.

12 years agoImprove the peer ranking and fix a minor bug.
Cameron Dale [Thu, 17 Apr 2008 19:58:00 +0000 (12:58 -0700)]
Improve the peer ranking and fix a minor bug.

12 years agoDelay the creation of files until after the application has finished loading.
Cameron Dale [Thu, 17 Apr 2008 19:57:18 +0000 (12:57 -0700)]
Delay the creation of files until after the application has finished loading.

12 years agoDon't create a thread to load the packages cache if it's already loaded.
Cameron Dale [Thu, 17 Apr 2008 18:56:15 +0000 (11:56 -0700)]
Don't create a thread to load the packages cache if it's already loaded.

12 years agoSwitch the user name on startup.
Cameron Dale [Wed, 16 Apr 2008 19:50:14 +0000 (12:50 -0700)]
Switch the user name on startup.

12 years agoBlock favicon.ico and allow remote stats requests (configurable).
Cameron Dale [Wed, 16 Apr 2008 18:05:31 +0000 (11:05 -0700)]
Block favicon.ico and allow remote stats requests (configurable).

12 years agoClean up the DHT config, making K and HASH_LENGTH constants instead.
Cameron Dale [Wed, 16 Apr 2008 17:40:56 +0000 (10:40 -0700)]
Clean up the DHT config, making K and HASH_LENGTH constants instead.

12 years agoSave the files downloaded from peers in a peers subdirectory.
Cameron Dale [Wed, 16 Apr 2008 17:10:33 +0000 (10:10 -0700)]
Save the files downloaded from peers in a peers subdirectory.

12 years agoMake the downloader statistics work.
Cameron Dale [Wed, 16 Apr 2008 03:36:39 +0000 (20:36 -0700)]
Make the downloader statistics work.

The previous (untested) commit was not working at all, this one now does,
though the TODO mentions some future work.

12 years agoMake better use of defer.Fail for returning deferred errors.
Cameron Dale [Tue, 15 Apr 2008 23:07:05 +0000 (16:07 -0700)]
Make better use of defer.Fail for returning deferred errors.

12 years agoRetransmit DHT requests before timeout occurs is complete.
Cameron Dale [Tue, 15 Apr 2008 22:57:03 +0000 (15:57 -0700)]
Retransmit DHT requests before timeout occurs is complete.

12 years agoKrpc now works as before but with a new request infrastructure.
Cameron Dale [Tue, 15 Apr 2008 21:59:38 +0000 (14:59 -0700)]
Krpc now works as before but with a new request infrastructure.

12 years agoWIP on sending multiple KRPC requests before timeout.
Cameron Dale [Tue, 15 Apr 2008 00:28:13 +0000 (17:28 -0700)]
WIP on sending multiple KRPC requests before timeout.

12 years agoAdd statistics reporting to the main program (untested).
Cameron Dale [Mon, 14 Apr 2008 23:35:12 +0000 (16:35 -0700)]
Add statistics reporting to the main program (untested).

12 years agoMore strict use of errbacks when using deferreds.
Cameron Dale [Mon, 14 Apr 2008 21:35:12 +0000 (14:35 -0700)]
More strict use of errbacks when using deferreds.

Now almost all calls to addCallback include adding an errback as well.
HTTPDownloader will remove the next request on a connection error.
HTTPServer will download the whole file after a rendering error.
A failed check of the freshness of a file will cause the whole file to be downloaded.
A resolving error in a DHT join behaves as if the bootstrap node was unreachable.
Removed the errback args to khashmir's find routines as they were never used.

12 years agoTry to rejoin DHT periodically after failures using exponential backoff.
Cameron Dale [Mon, 14 Apr 2008 19:23:33 +0000 (12:23 -0700)]
Try to rejoin DHT periodically after failures using exponential backoff.

Also make the get and store functions call errbacks if not joined.
Also modify the callers of the get and store functions to respond well
to errbacks.
Also add a unittest for the rejoining functionality.

12 years agoUse the apt_p2p_conf config import rather than passing parameters around.
Cameron Dale [Mon, 14 Apr 2008 05:22:25 +0000 (22:22 -0700)]
Use the apt_p2p_conf config import rather than passing parameters around.

12 years agoUse the mirror as a peer when there are few peers for a file.
Cameron Dale [Mon, 14 Apr 2008 03:18:17 +0000 (20:18 -0700)]
Use the mirror as a peer when there are few peers for a file.

12 years agoUnquote urls before trying to look them up in index files.
Cameron Dale [Mon, 14 Apr 2008 02:46:56 +0000 (19:46 -0700)]
Unquote urls before trying to look them up in index files.

12 years agoAlways download max PIECE_SIZE from peers, even if there's only one peer.
Cameron Dale [Mon, 14 Apr 2008 01:02:39 +0000 (18:02 -0700)]
Always download max PIECE_SIZE from peers, even if there's only one peer.

Used to avoid ticket #1993 in Twisted:

12 years agoFix some tests and add some TODO items.
Cameron Dale [Sun, 13 Apr 2008 06:52:15 +0000 (23:52 -0700)]
Fix some tests and add some TODO items.

12 years agoActually add piece hashes to the database.
Cameron Dale [Sat, 12 Apr 2008 22:47:29 +0000 (15:47 -0700)]
Actually add piece hashes to the database.

12 years agoUpdated and added a lot of unittests.
Cameron Dale [Fri, 11 Apr 2008 23:21:18 +0000 (16:21 -0700)]
Updated and added a lot of unittests.

Still need to add tests for the Peer and Cache Managers.

12 years agoAlways use SHA1 hashes for pieces.
Cameron Dale [Fri, 11 Apr 2008 20:00:03 +0000 (13:00 -0700)]
Always use SHA1 hashes for pieces.

Only affects non-sha1 files like source files.
Also added a unit test for it.
Also added a unit test for the previous piece boundaries bug.

12 years agoMinor typo fixes in the Peer manager.
Cameron Dale [Fri, 11 Apr 2008 19:47:57 +0000 (12:47 -0700)]
Minor typo fixes in the Peer manager.

12 years agoMultiple peer downloading is mostly working now.
Cameron Dale [Fri, 11 Apr 2008 06:40:04 +0000 (23:40 -0700)]
Multiple peer downloading is mostly working now.

Many bug fixes, too numerous to mention.
Lots of new (probably temporary) logging.
New test stuff for the multi-peer downloading.
Still fails on large files when retrieving piece hashes from peers.

12 years agoFix a bug in the ThrottlingProtocol when writeSequence is called with a tuple.
Cameron Dale [Fri, 11 Apr 2008 06:34:40 +0000 (23:34 -0700)]
Fix a bug in the ThrottlingProtocol when writeSequence is called with a tuple.

12 years agoFix a bug in the Hash checker when updating exactly to a piece boundary.
Cameron Dale [Fri, 11 Apr 2008 06:33:30 +0000 (23:33 -0700)]
Fix a bug in the Hash checker when updating exactly to a piece boundary.

12 years agoFix some minor documentation typos.
Cameron Dale [Thu, 10 Apr 2008 00:32:35 +0000 (17:32 -0700)]
Fix some minor documentation typos.

12 years agoMore multiple peer downloading work, finished but still untested.
Cameron Dale [Thu, 10 Apr 2008 00:27:29 +0000 (17:27 -0700)]
More multiple peer downloading work, finished but still untested.

12 years agoMinor update to the multiple peer downloading (still not working).
Cameron Dale [Mon, 7 Apr 2008 05:51:24 +0000 (22:51 -0700)]
Minor update to the multiple peer downloading (still not working).

12 years agoMassive work on the peer downloading, not working or complete yet.
Cameron Dale [Fri, 4 Apr 2008 23:48:48 +0000 (16:48 -0700)]
Massive work on the peer downloading, not working or complete yet.

12 years agoWIP on multiple peer downloading (completely broken).
Cameron Dale [Wed, 2 Apr 2008 22:44:00 +0000 (15:44 -0700)]
WIP on multiple peer downloading (completely broken).

12 years agoFix a bug caused by twisted unquoting URLs received by the web server.
Cameron Dale [Tue, 1 Apr 2008 23:18:00 +0000 (16:18 -0700)]
Fix a bug caused by twisted unquoting URLs received by the web server.

12 years agoStore the peer rank as an attribute and recalculate it automatically.
Cameron Dale [Mon, 31 Mar 2008 23:41:01 +0000 (16:41 -0700)]
Store the peer rank as an attribute and recalculate it automatically.

12 years agoMake all the peer rankings exponentially decreasing.
Cameron Dale [Mon, 31 Mar 2008 23:21:15 +0000 (16:21 -0700)]
Make all the peer rankings exponentially decreasing.

12 years agoReturn 404s to all requests for Packages.diff and Sources.diff.
Cameron Dale [Mon, 31 Mar 2008 23:03:05 +0000 (16:03 -0700)]
Return 404s to all requests for Packages.diff and Sources.diff.

12 years agoMake the upload limit a config option.
Cameron Dale [Mon, 31 Mar 2008 22:47:15 +0000 (15:47 -0700)]
Make the upload limit a config option.

12 years agoOnly throttle uploads to peers, not to apt.
Cameron Dale [Mon, 31 Mar 2008 22:32:04 +0000 (15:32 -0700)]
Only throttle uploads to peers, not to apt.

12 years agoRetire one TODO and add another.
Cameron Dale [Mon, 31 Mar 2008 21:35:20 +0000 (14:35 -0700)]
Retire one TODO and add another.

12 years agoAllow the actions to call the local node's remote interface.
Cameron Dale [Thu, 27 Mar 2008 23:55:08 +0000 (16:55 -0700)]
Allow the actions to call the local node's remote interface.

The find_* actions now start with only the local node instead of
the nodes from the routing table.

12 years agoKRPC calls callback with the response dictionary by itself.
Cameron Dale [Thu, 27 Mar 2008 23:05:10 +0000 (16:05 -0700)]
KRPC calls callback with the response dictionary by itself.

Previously it was a dictionary, whose key 'rsp' contained the response dictionary.
The response dictionary has a key '_krpc_sender' added.

12 years agoReorganize the DHT's remote interface parameters.
Cameron Dale [Thu, 27 Mar 2008 22:00:50 +0000 (15:00 -0700)]
Reorganize the DHT's remote interface parameters.

Now the parameter ordering matches the one in the knode module.

Also makes the _krpc_sender optional so the remote interface can
be called by the local node.

12 years agoUse copies of the routing table nodes in actions.
Cameron Dale [Thu, 27 Mar 2008 00:09:04 +0000 (17:09 -0700)]
Use copies of the routing table nodes in actions.

The recursive actions need to use copies of the nodes in the routing
table, otherwise the token and num_values could be preset or set
twice by multiple actions.

12 years agoImport the log module, DOH.
Cameron Dale [Fri, 21 Mar 2008 05:17:32 +0000 (22:17 -0700)]
Import the log module, DOH.

12 years agoMerge small DHT buckets when a node is removed.
Cameron Dale [Fri, 21 Mar 2008 01:36:33 +0000 (18:36 -0700)]
Merge small DHT buckets when a node is removed.

12 years agoWorkaround old sqlite not having 'select count(distinct key)'.
Cameron Dale [Thu, 20 Mar 2008 22:47:14 +0000 (15:47 -0700)]
Workaround old sqlite not having 'select count(distinct key)'.

12 years agoRemove the duplicate error logging from the knode module.
Cameron Dale [Thu, 20 Mar 2008 22:26:41 +0000 (15:26 -0700)]
Remove the duplicate error logging from the knode module.

12 years agoRemove the unused steveholt entry from the bootstrap list.
Cameron Dale [Thu, 20 Mar 2008 22:12:29 +0000 (15:12 -0700)]
Remove the unused steveholt entry from the bootstrap list.

12 years agoFix some bugs in the pinging and ID checking.
Cameron Dale [Thu, 20 Mar 2008 22:09:29 +0000 (15:09 -0700)]
Fix some bugs in the pinging and ID checking.

Raise an exception if the node's ID has changed. This means the node's
returned results will be ignored.

The KTable replaceStaleNode now adds the new node if there's space even
if the old one isn't found. This is due to the node ID having changed
resulting in instantly removing the node from the routing table.

The handler for the ping result in insertNode was mistakenly taking the
Failure returned as a result as a node and passing it to replaceStaleNode.

12 years agoMove the HTML stats page generation for the DHT into the stats module.
Cameron Dale [Mon, 17 Mar 2008 23:43:30 +0000 (16:43 -0700)]
Move the HTML stats page generation for the DHT into the stats module.

12 years agoRename the knode functions to match the krpc method names.
Cameron Dale [Mon, 17 Mar 2008 22:27:56 +0000 (15:27 -0700)]
Rename the knode functions to match the krpc method names.

12 years agoAdded the number of times each action was started to the DHT stats.
Cameron Dale [Mon, 17 Mar 2008 22:21:26 +0000 (15:21 -0700)]
Added the number of times each action was started to the DHT stats.

12 years agoAdd bytes/sec to the DHT stats.
Cameron Dale [Mon, 17 Mar 2008 21:07:59 +0000 (14:07 -0700)]
Add bytes/sec to the DHT stats.

12 years agoDisplay DHT statistics to the HTTP user.
Cameron Dale [Fri, 14 Mar 2008 23:40:05 +0000 (16:40 -0700)]
Display DHT statistics to the HTTP user.

Adds 2 new interfaces to the DHT implementation:
 * DHTStats: DHT that supports gathering statistics
 * DHTStatsFactory: DHT that supports creating it's own HTTP server
apt_dht_Khashmir supports IDHTStats and formats stats for HTML.
apt_dht_Khashmir also supports IDHTStatsFactory if twisted.web2 is found.
Main script passes the DHT class to the main program.
Main script (DHT only) starts the stats factory if the DHT supports it.
Main program retrieves statistics from the DHT when asked by the HTTPServer.

12 years agoStart using the new DHT stats module (untested).
Cameron Dale [Wed, 12 Mar 2008 06:40:09 +0000 (23:40 -0700)]
Start using the new DHT stats module (untested).

12 years agoAdd the new stats module to the DHT.
Cameron Dale [Wed, 12 Mar 2008 06:39:29 +0000 (23:39 -0700)]
Add the new stats module to the DHT.

12 years agoFix minor bugs in the use of sha module and misspelled b2a_hex.
Cameron Dale [Tue, 11 Mar 2008 18:40:43 +0000 (11:40 -0700)]
Fix minor bugs in the use of sha module and misspelled b2a_hex.

12 years agoFixed justSeenNode in KTable to update the bucket properly.
Cameron Dale [Sat, 8 Mar 2008 03:56:17 +0000 (19:56 -0800)]
Fixed justSeenNode in KTable to update the bucket properly.

It now moves the seen node to the end and marks the bucket as
having been updated.

12 years agoAnother TODO item.
Cameron Dale [Sat, 8 Mar 2008 00:38:33 +0000 (16:38 -0800)]
Another TODO item.

12 years agoDon't use the tables findNodes when you can just bisect for it.
Cameron Dale [Sat, 8 Mar 2008 00:37:10 +0000 (16:37 -0800)]
Don't use the tables findNodes when you can just bisect for it.

12 years agoAdd new node to table when refreshing old node.
Cameron Dale [Sat, 8 Mar 2008 00:06:57 +0000 (16:06 -0800)]
Add new node to table when refreshing old node.

Fixes a bug introduced by commit:

    6ce38471: fix peer update problem connection refused in store value

12 years agoRename all apt-dht files to apt-p2p.
Cameron Dale [Thu, 6 Mar 2008 00:17:03 +0000 (16:17 -0800)]
Rename all apt-dht files to apt-p2p.

12 years agoRename project to apt-p2p.
Cameron Dale [Thu, 6 Mar 2008 00:10:07 +0000 (16:10 -0800)]
Rename project to apt-p2p.

12 years agoMerge branch 'documentation' into apt-dht
Cameron Dale [Wed, 5 Mar 2008 23:06:44 +0000 (15:06 -0800)]
Merge branch 'documentation' into apt-dht

12 years agoVarious documentation fixes and additions.
Cameron Dale [Wed, 5 Mar 2008 23:04:50 +0000 (15:04 -0800)]
Various documentation fixes and additions.

12 years agoDocument the DHT package.
Cameron Dale [Wed, 5 Mar 2008 23:03:33 +0000 (15:03 -0800)]
Document the DHT package.

12 years agoDocument the DHT's main khashmir module.
Cameron Dale [Wed, 5 Mar 2008 23:03:11 +0000 (15:03 -0800)]
Document the DHT's main khashmir module.

12 years agoDocument the DHT's krpc module.
Cameron Dale [Wed, 5 Mar 2008 23:02:44 +0000 (15:02 -0800)]
Document the DHT's krpc module.

12 years agoMerge branch 'apt-dht' of ssh://git.camrdale.org/git/apt-dht into apt-dht
Cameron Dale [Wed, 5 Mar 2008 00:05:22 +0000 (16:05 -0800)]
Merge branch 'apt-dht' of ssh://git.camrdale.org/git/apt-dht into apt-dht