]>
git.mxchange.org Git - quix0rs-apt-p2p.git/log
Cameron Dale [Fri, 9 May 2008 00:28:07 +0000 (17:28 -0700)]
PeerManager keeps a list of site names, peers are kept in a dictionary.
Cameron Dale [Thu, 8 May 2008 22:24:34 +0000 (15:24 -0700)]
Speed up the recursive DHT actions when timeouts occur.
If the action proceeds too far beyond a node, ignore it and move on.
If the closest K nodes have all responded, abort early.
Cameron Dale [Thu, 8 May 2008 21:04:57 +0000 (14:04 -0700)]
Move the bucket manipulation into the KBucket class.
Cameron Dale [Wed, 7 May 2008 07:10:38 +0000 (00:10 -0700)]
Fix a traceback generating bug in the refreshing of DHT entries.
Also change the DB's expiredHashes to return a sorted list.
Cameron Dale [Mon, 5 May 2008 06:43:18 +0000 (23:43 -0700)]
New TODO item for rotating DNS entries.
Cameron Dale [Mon, 5 May 2008 06:18:40 +0000 (23:18 -0700)]
Update versions and changelog for new release.
Cameron Dale [Sun, 4 May 2008 21:16:25 +0000 (14:16 -0700)]
Fixed a bug in the HTTP downloader that caused errors with some mirrors that always close the connections.
Cameron Dale [Sat, 3 May 2008 18:14:57 +0000 (11:14 -0700)]
Make sure full timeouts occur even if a resend timeout has not.
Cameron Dale [Sat, 3 May 2008 17:27:35 +0000 (10:27 -0700)]
Update the version numbers and changelog for new release.
Cameron Dale [Sat, 3 May 2008 17:21:38 +0000 (10:21 -0700)]
Lower the timeout to 9 sec to reduce delays.
Cameron Dale [Fri, 2 May 2008 21:15:18 +0000 (14:15 -0700)]
Final abstract for submission.
Cameron Dale [Thu, 1 May 2008 20:04:46 +0000 (13:04 -0700)]
Fix the broken local IP regexes.
Cameron Dale [Thu, 1 May 2008 02:31:53 +0000 (19:31 -0700)]
Massive update to most of the abstract.
Cameron Dale [Tue, 29 Apr 2008 17:16:50 +0000 (10:16 -0700)]
Don't log KRPC errors for receiving an invalid store_value token.
Cameron Dale [Tue, 29 Apr 2008 00:46:22 +0000 (17:46 -0700)]
Create the Kile project for the SIGCOMM poster abstract.
Cameron Dale [Tue, 29 Apr 2008 00:44:45 +0000 (17:44 -0700)]
Only add nodes to the routing table that have responded to a request.
If they haven't, ping them before adding them.
Still ping the old node first to see if it's stale.
Move the temporary functions for ping/join to be module functions.
Cameron Dale [Tue, 29 Apr 2008 00:36:30 +0000 (17:36 -0700)]
Don't try and add ourself to the routing table during an action.
Cameron Dale [Mon, 28 Apr 2008 23:28:06 +0000 (16:28 -0700)]
Don't add local IP addresses to the routing table (with config option to override).
Cameron Dale [Mon, 28 Apr 2008 22:08:35 +0000 (15:08 -0700)]
New TODO for hashing files with detached GPG signatures.
Cameron Dale [Sat, 26 Apr 2008 17:01:08 +0000 (10:01 -0700)]
Fix: no files were added to the DHT due to missing hashing results.
Cameron Dale [Sat, 26 Apr 2008 03:10:23 +0000 (20:10 -0700)]
Fix a postrm bug due to debconf generating errors if the apt-p2p user is missing.
Cameron Dale [Sat, 26 Apr 2008 02:38:39 +0000 (19:38 -0700)]
Clean up the copyrights mentioned in the code.
Cameron Dale [Sat, 26 Apr 2008 01:57:05 +0000 (18:57 -0700)]
Remove the changelog parsing from setup.py and just hardcode the version.
Cameron Dale [Sat, 26 Apr 2008 01:50:42 +0000 (18:50 -0700)]
Fix some documentation errors.
Cameron Dale [Sat, 26 Apr 2008 01:25:19 +0000 (18:25 -0700)]
Update the version number and changelog for the initial release.
Cameron Dale [Sat, 26 Apr 2008 01:24:50 +0000 (18:24 -0700)]
Clean up the language in the description and man pages.
Also tightened the dependency on Twisted to 8.0.
Cameron Dale [Sat, 26 Apr 2008 01:23:12 +0000 (18:23 -0700)]
Update the config file for the change in default key expiry length.
Cameron Dale [Fri, 25 Apr 2008 21:43:14 +0000 (14:43 -0700)]
Add an optimization to the HTTP client protocol so that pieplining begins sooner.
Specifically, it begins after the first requests headers are sent, instead of the whole response.
Cameron Dale [Fri, 25 Apr 2008 05:20:43 +0000 (22:20 -0700)]
Don't use the fallback mirror retry download.
Until we can distinuish real errors from apt-get closing the
connection, this is not a good idea. See the connectionLost()
function of the web2.http.Request object for a possible solution.
Cameron Dale [Fri, 25 Apr 2008 05:18:58 +0000 (22:18 -0700)]
More fixes for the broken twisted HTTP client.
Cameron Dale [Fri, 25 Apr 2008 05:18:29 +0000 (22:18 -0700)]
Move all streams to new Streams module and replace ProxyFileStream with GrowingFileStream.
Instead of ProxyFileStream, now use a combination of GrowingFileStream
and StreamToFile like the PeerManager does to download and upload. It's
mostly working, only some errors when aborting apt-get in test 'a'.
Also added a new test 'c' for downloading out of order, which is why
ProxyFileStream can not be used.
Cameron Dale [Thu, 24 Apr 2008 23:19:09 +0000 (16:19 -0700)]
Add a final retry of all errback mirror downloads.
Cameron Dale [Thu, 24 Apr 2008 23:17:54 +0000 (16:17 -0700)]
Fix some errors in the new twisted HTTP client's connectionLost() methods.
See http://twistedmatrix.com/trac/ticket/3207 for more info.
Cameron Dale [Thu, 24 Apr 2008 19:05:03 +0000 (12:05 -0700)]
Add debconf to inform the user of the necessity of port forwarding.
Cameron Dale [Thu, 24 Apr 2008 18:15:58 +0000 (11:15 -0700)]
Move the DHT stuff out of the main program and into the new DHTManager module.
Cameron Dale [Thu, 24 Apr 2008 16:54:31 +0000 (09:54 -0700)]
Better handling and logging for intermittent HTTP client submission errors.
Cameron Dale [Thu, 24 Apr 2008 16:33:34 +0000 (09:33 -0700)]
Don't add attributes to HTTP requests, store them separately.
Cameron Dale [Thu, 24 Apr 2008 05:02:43 +0000 (22:02 -0700)]
HTTP client no longer keeps a response queue of requests.
Instead the requests are added as variables to the call/errbacks.
There is a new variable keeping track of how many responses are outstanding.
clientGone no longer remove/errback responses, as the errbacks will be
called by the client protocol instead.
Cameron Dale [Thu, 24 Apr 2008 04:41:38 +0000 (21:41 -0700)]
Fix a typo in commit
80399176ed66d .
Cameron Dale [Thu, 24 Apr 2008 03:52:11 +0000 (20:52 -0700)]
Fixed an error in downloading source packages from peers.
Source packages with no pieces were hash checked with sha1, which would fail.
Also added a test for source packages.
Also removed some unneeded logging in the peer downloader.
Cameron Dale [Thu, 24 Apr 2008 03:50:06 +0000 (20:50 -0700)]
Fix a typo in commit
e82e704e27 .
Cameron Dale [Wed, 23 Apr 2008 23:12:29 +0000 (16:12 -0700)]
Set the Last-Modified header when downloading from peers.
Cameron Dale [Wed, 23 Apr 2008 22:50:14 +0000 (15:50 -0700)]
Remove temporary files from the peer's download cache.
Cameron Dale [Wed, 23 Apr 2008 22:30:39 +0000 (15:30 -0700)]
Refresh expired DHT hashes concurrently instead of sequentially.
Cameron Dale [Wed, 23 Apr 2008 21:31:03 +0000 (14:31 -0700)]
Don't check file when listing expired hashes, instead check files at startup.
Cameron Dale [Wed, 23 Apr 2008 03:16:21 +0000 (20:16 -0700)]
Trap KRPC errors in writing to the transport and wait for timeout instead.
Cameron Dale [Wed, 23 Apr 2008 02:10:21 +0000 (19:10 -0700)]
Disconnect the insertNode calls from the callers so errors don't affect them.
Cameron Dale [Mon, 21 Apr 2008 21:15:14 +0000 (14:15 -0700)]
Remove some unnecessary calls to log.err and just print the error message instead.
Cameron Dale [Mon, 21 Apr 2008 20:08:13 +0000 (13:08 -0700)]
Fix a bug in loading the new KRPC timeout config params.
Cameron Dale [Mon, 21 Apr 2008 19:41:24 +0000 (12:41 -0700)]
Make the DHT timeouts configuration parameters.
Cameron Dale [Mon, 21 Apr 2008 19:05:56 +0000 (12:05 -0700)]
Conf time parser works with floats, and lengthened DHT hash expiry to 3 hours.
Cameron Dale [Mon, 21 Apr 2008 18:54:05 +0000 (11:54 -0700)]
Fix an error in the actions that allowed for the result to be sent twice.
Cameron Dale [Mon, 21 Apr 2008 05:55:40 +0000 (22:55 -0700)]
Some more TODO items.
Cameron Dale [Mon, 21 Apr 2008 05:09:34 +0000 (22:09 -0700)]
Apt packages cache now saves an unload request received during loading.
Cameron Dale [Mon, 21 Apr 2008 02:47:31 +0000 (19:47 -0700)]
Sync the AptPackages db after modification, and close when done.
Cameron Dale [Mon, 21 Apr 2008 02:42:31 +0000 (19:42 -0700)]
Add all files to the DB with their hashes.
Non-DHT files are marked as such.
HTTP server looks up the cache path to decide whether to return a file.
Cameron Dale [Mon, 21 Apr 2008 02:39:33 +0000 (19:39 -0700)]
Fix an HTTP download bug that caused HEAD requests to not pipeline.
Also improve the testing for this.
Cameron Dale [Sat, 19 Apr 2008 18:39:24 +0000 (11:39 -0700)]
Reorder the main application to find cached hashes before checking freshness.
Eliminates some unnecessary HEAD requests to the server.
HEAD requests are only done if there is no hash for the file.
Cameron Dale [Sat, 19 Apr 2008 17:48:47 +0000 (10:48 -0700)]
Add a new test and improve the cleaning of apt files.
Cameron Dale [Fri, 18 Apr 2008 21:03:26 +0000 (14:03 -0700)]
DHT stats now include completed actions and various elapsed times for actions.
Cameron Dale [Fri, 18 Apr 2008 05:13:17 +0000 (22:13 -0700)]
Return 404's for hashes that aren't found.
Cameron Dale [Fri, 18 Apr 2008 02:14:05 +0000 (19:14 -0700)]
Remove some unnecessary log messages and use better Exceptions.
Cameron Dale [Fri, 18 Apr 2008 02:05:45 +0000 (19:05 -0700)]
Cache returns errors if apt closes the connection and deletes the incomplete files.
Cameron Dale [Fri, 18 Apr 2008 01:15:35 +0000 (18:15 -0700)]
Remove some headers Apt sets before returning full requests from the server.
Otherwise the streaming will be interrupted or changed by Twisted's
modification of responses before sending (e.g. sending 304's based
on 'If-Modified-Since' and 'Last-Modified' headers). The streaming
is currently needed though to load the file in the cache.
Cameron Dale [Fri, 18 Apr 2008 01:12:54 +0000 (18:12 -0700)]
Better identification of peer uploads by the HTTP server.
Cameron Dale [Thu, 17 Apr 2008 19:58:59 +0000 (12:58 -0700)]
Always try and find the mirror site, even if no updated files have been saved.
Cameron Dale [Thu, 17 Apr 2008 19:58:00 +0000 (12:58 -0700)]
Improve the peer ranking and fix a minor bug.
Cameron Dale [Thu, 17 Apr 2008 19:57:18 +0000 (12:57 -0700)]
Delay the creation of files until after the application has finished loading.
Cameron Dale [Thu, 17 Apr 2008 18:56:15 +0000 (11:56 -0700)]
Don't create a thread to load the packages cache if it's already loaded.
Cameron Dale [Wed, 16 Apr 2008 19:50:14 +0000 (12:50 -0700)]
Switch the user name on startup.
Cameron Dale [Wed, 16 Apr 2008 18:05:31 +0000 (11:05 -0700)]
Block favicon.ico and allow remote stats requests (configurable).
Cameron Dale [Wed, 16 Apr 2008 17:40:56 +0000 (10:40 -0700)]
Clean up the DHT config, making K and HASH_LENGTH constants instead.
Cameron Dale [Wed, 16 Apr 2008 17:10:33 +0000 (10:10 -0700)]
Save the files downloaded from peers in a peers subdirectory.
Cameron Dale [Wed, 16 Apr 2008 03:36:39 +0000 (20:36 -0700)]
Make the downloader statistics work.
The previous (untested) commit was not working at all, this one now does,
though the TODO mentions some future work.
Cameron Dale [Tue, 15 Apr 2008 23:07:05 +0000 (16:07 -0700)]
Make better use of defer.Fail for returning deferred errors.
Cameron Dale [Tue, 15 Apr 2008 22:57:03 +0000 (15:57 -0700)]
Retransmit DHT requests before timeout occurs is complete.
Cameron Dale [Tue, 15 Apr 2008 21:59:38 +0000 (14:59 -0700)]
Krpc now works as before but with a new request infrastructure.
Cameron Dale [Tue, 15 Apr 2008 00:28:13 +0000 (17:28 -0700)]
WIP on sending multiple KRPC requests before timeout.
Cameron Dale [Mon, 14 Apr 2008 23:35:12 +0000 (16:35 -0700)]
Add statistics reporting to the main program (untested).
Cameron Dale [Mon, 14 Apr 2008 21:35:12 +0000 (14:35 -0700)]
More strict use of errbacks when using deferreds.
Now almost all calls to addCallback include adding an errback as well.
HTTPDownloader will remove the next request on a connection error.
HTTPServer will download the whole file after a rendering error.
A failed check of the freshness of a file will cause the whole file to be downloaded.
A resolving error in a DHT join behaves as if the bootstrap node was unreachable.
Removed the errback args to khashmir's find routines as they were never used.
Cameron Dale [Mon, 14 Apr 2008 19:23:33 +0000 (12:23 -0700)]
Try to rejoin DHT periodically after failures using exponential backoff.
Also make the get and store functions call errbacks if not joined.
Also modify the callers of the get and store functions to respond well
to errbacks.
Also add a unittest for the rejoining functionality.
Cameron Dale [Mon, 14 Apr 2008 05:22:25 +0000 (22:22 -0700)]
Use the apt_p2p_conf config import rather than passing parameters around.
Cameron Dale [Mon, 14 Apr 2008 03:18:17 +0000 (20:18 -0700)]
Use the mirror as a peer when there are few peers for a file.
Cameron Dale [Mon, 14 Apr 2008 02:46:56 +0000 (19:46 -0700)]
Unquote urls before trying to look them up in index files.
Cameron Dale [Mon, 14 Apr 2008 01:02:39 +0000 (18:02 -0700)]
Always download max PIECE_SIZE from peers, even if there's only one peer.
Used to avoid ticket #1993 in Twisted:
http://twistedmatrix.com/trac/ticket/1993
Cameron Dale [Sun, 13 Apr 2008 06:52:15 +0000 (23:52 -0700)]
Fix some tests and add some TODO items.
Cameron Dale [Sat, 12 Apr 2008 22:47:29 +0000 (15:47 -0700)]
Actually add piece hashes to the database.
Cameron Dale [Fri, 11 Apr 2008 23:21:18 +0000 (16:21 -0700)]
Updated and added a lot of unittests.
Still need to add tests for the Peer and Cache Managers.
Cameron Dale [Fri, 11 Apr 2008 20:00:03 +0000 (13:00 -0700)]
Always use SHA1 hashes for pieces.
Only affects non-sha1 files like source files.
Also added a unit test for it.
Also added a unit test for the previous piece boundaries bug.
Cameron Dale [Fri, 11 Apr 2008 19:47:57 +0000 (12:47 -0700)]
Minor typo fixes in the Peer manager.
Cameron Dale [Fri, 11 Apr 2008 06:40:04 +0000 (23:40 -0700)]
Multiple peer downloading is mostly working now.
Many bug fixes, too numerous to mention.
Lots of new (probably temporary) logging.
New test stuff for the multi-peer downloading.
Still fails on large files when retrieving piece hashes from peers.
Cameron Dale [Fri, 11 Apr 2008 06:34:40 +0000 (23:34 -0700)]
Fix a bug in the ThrottlingProtocol when writeSequence is called with a tuple.
Cameron Dale [Fri, 11 Apr 2008 06:33:30 +0000 (23:33 -0700)]
Fix a bug in the Hash checker when updating exactly to a piece boundary.
Cameron Dale [Thu, 10 Apr 2008 00:32:35 +0000 (17:32 -0700)]
Fix some minor documentation typos.
Cameron Dale [Thu, 10 Apr 2008 00:27:29 +0000 (17:27 -0700)]
More multiple peer downloading work, finished but still untested.
Cameron Dale [Mon, 7 Apr 2008 05:51:24 +0000 (22:51 -0700)]
Minor update to the multiple peer downloading (still not working).
Cameron Dale [Fri, 4 Apr 2008 23:48:48 +0000 (16:48 -0700)]
Massive work on the peer downloading, not working or complete yet.
Cameron Dale [Wed, 2 Apr 2008 22:44:00 +0000 (15:44 -0700)]
WIP on multiple peer downloading (completely broken).
Cameron Dale [Tue, 1 Apr 2008 23:18:00 +0000 (16:18 -0700)]
Fix a bug caused by twisted unquoting URLs received by the web server.
Cameron Dale [Mon, 31 Mar 2008 23:41:01 +0000 (16:41 -0700)]
Store the peer rank as an attribute and recalculate it automatically.