quix0rs-apt-p2p.git
11 years agoAdd the new stats module to the DHT.
Cameron Dale [Wed, 12 Mar 2008 06:39:29 +0000 (23:39 -0700)]
Add the new stats module to the DHT.

11 years agoFix minor bugs in the use of sha module and misspelled b2a_hex.
Cameron Dale [Tue, 11 Mar 2008 18:40:43 +0000 (11:40 -0700)]
Fix minor bugs in the use of sha module and misspelled b2a_hex.

11 years agoFixed justSeenNode in KTable to update the bucket properly.
Cameron Dale [Sat, 8 Mar 2008 03:56:17 +0000 (19:56 -0800)]
Fixed justSeenNode in KTable to update the bucket properly.

It now moves the seen node to the end and marks the bucket as
having been updated.

11 years agoAnother TODO item.
Cameron Dale [Sat, 8 Mar 2008 00:38:33 +0000 (16:38 -0800)]
Another TODO item.

11 years agoDon't use the tables findNodes when you can just bisect for it.
Cameron Dale [Sat, 8 Mar 2008 00:37:10 +0000 (16:37 -0800)]
Don't use the tables findNodes when you can just bisect for it.

11 years agoAdd new node to table when refreshing old node.
Cameron Dale [Sat, 8 Mar 2008 00:06:57 +0000 (16:06 -0800)]
Add new node to table when refreshing old node.

Fixes a bug introduced by commit:

    6ce38471: fix peer update problem connection refused in store value

11 years agoRename all apt-dht files to apt-p2p.
Cameron Dale [Thu, 6 Mar 2008 00:17:03 +0000 (16:17 -0800)]
Rename all apt-dht files to apt-p2p.

11 years agoRename project to apt-p2p.
Cameron Dale [Thu, 6 Mar 2008 00:10:07 +0000 (16:10 -0800)]
Rename project to apt-p2p.

11 years agoMerge branch 'documentation' into apt-dht
Cameron Dale [Wed, 5 Mar 2008 23:06:44 +0000 (15:06 -0800)]
Merge branch 'documentation' into apt-dht

11 years agoVarious documentation fixes and additions.
Cameron Dale [Wed, 5 Mar 2008 23:04:50 +0000 (15:04 -0800)]
Various documentation fixes and additions.

11 years agoDocument the DHT package.
Cameron Dale [Wed, 5 Mar 2008 23:03:33 +0000 (15:03 -0800)]
Document the DHT package.

11 years agoDocument the DHT's main khashmir module.
Cameron Dale [Wed, 5 Mar 2008 23:03:11 +0000 (15:03 -0800)]
Document the DHT's main khashmir module.

11 years agoDocument the DHT's krpc module.
Cameron Dale [Wed, 5 Mar 2008 23:02:44 +0000 (15:02 -0800)]
Document the DHT's krpc module.

11 years agoMerge branch 'apt-dht' of ssh://git.camrdale.org/git/apt-dht into apt-dht
Cameron Dale [Wed, 5 Mar 2008 00:05:22 +0000 (16:05 -0800)]
Merge branch 'apt-dht' of ssh://git.camrdale.org/git/apt-dht into apt-dht

11 years agoMore todo items.
Cameron Dale [Wed, 5 Mar 2008 00:04:40 +0000 (16:04 -0800)]
More todo items.

11 years agoDocument the DHT's util module.
Cameron Dale [Wed, 5 Mar 2008 00:04:12 +0000 (16:04 -0800)]
Document the DHT's util module.

11 years agoDocument the DHT's node module.
Cameron Dale [Wed, 5 Mar 2008 00:03:58 +0000 (16:03 -0800)]
Document the DHT's node module.

11 years agoDocument the DHT's ktable module.
Cameron Dale [Wed, 5 Mar 2008 00:03:45 +0000 (16:03 -0800)]
Document the DHT's ktable module.

11 years agoDocument the DHT's knode module.
Cameron Dale [Wed, 5 Mar 2008 00:03:28 +0000 (16:03 -0800)]
Document the DHT's knode module.

11 years agoDocument the DHT's khash module.
Cameron Dale [Wed, 5 Mar 2008 00:03:16 +0000 (16:03 -0800)]
Document the DHT's khash module.

11 years agoDocument the DHT's db module.
Cameron Dale [Wed, 5 Mar 2008 00:02:58 +0000 (16:02 -0800)]
Document the DHT's db module.

11 years agoDocument the DHT's actions module.
Cameron Dale [Wed, 5 Mar 2008 00:02:43 +0000 (16:02 -0800)]
Document the DHT's actions module.

11 years agoDocument the DHT's DHT module.
Cameron Dale [Wed, 5 Mar 2008 00:02:27 +0000 (16:02 -0800)]
Document the DHT's DHT module.

11 years agoAdd cleanup to the MirrorManager for the AptPackages.
Cameron Dale [Sun, 2 Mar 2008 20:40:17 +0000 (12:40 -0800)]
Add cleanup to the MirrorManager for the AptPackages.

11 years agoFix a minor error in the DB's storeFile function.
Cameron Dale [Sun, 2 Mar 2008 20:26:16 +0000 (12:26 -0800)]
Fix a minor error in the DB's storeFile function.

11 years agoA few more documentation fixes.
Cameron Dale [Sat, 1 Mar 2008 02:46:06 +0000 (18:46 -0800)]
A few more documentation fixes.

11 years agoDocumented the util module.
Cameron Dale [Sat, 1 Mar 2008 02:44:56 +0000 (18:44 -0800)]
Documented the util module.

11 years agoDocumented the db module.
Cameron Dale [Sat, 1 Mar 2008 02:44:40 +0000 (18:44 -0800)]
Documented the db module.

11 years agoDocumented the PeerManager module.
Cameron Dale [Sat, 1 Mar 2008 02:44:23 +0000 (18:44 -0800)]
Documented the PeerManager module.

11 years agoDocumented the Hash module.
Cameron Dale [Sat, 1 Mar 2008 02:43:58 +0000 (18:43 -0800)]
Documented the Hash module.

11 years agoDocumented the HTTPServer module.
Cameron Dale [Sat, 1 Mar 2008 02:43:41 +0000 (18:43 -0800)]
Documented the HTTPServer module.

11 years agoDocumented the HTTPDownloader module.
Cameron Dale [Sat, 1 Mar 2008 02:43:12 +0000 (18:43 -0800)]
Documented the HTTPDownloader module.

11 years agoDocument the MirrorManager module.
Cameron Dale [Fri, 29 Feb 2008 23:40:29 +0000 (15:40 -0800)]
Document the MirrorManager module.

11 years agoDocument the AptPackages module.
Cameron Dale [Fri, 29 Feb 2008 23:40:11 +0000 (15:40 -0800)]
Document the AptPackages module.

11 years agoDocument the apt_dht module.
Cameron Dale [Fri, 29 Feb 2008 21:42:44 +0000 (13:42 -0800)]
Document the apt_dht module.

11 years agoDocument the apt_dht_conf module.
Cameron Dale [Fri, 29 Feb 2008 21:42:28 +0000 (13:42 -0800)]
Document the apt_dht_conf module.

11 years agoDocument the apt_dht module's __init__ file.
Cameron Dale [Fri, 29 Feb 2008 21:41:50 +0000 (13:41 -0800)]
Document the apt_dht module's __init__ file.

11 years agoMove the normalization of key lengths from the HashObject to the DHT.
Cameron Dale [Fri, 29 Feb 2008 21:10:46 +0000 (13:10 -0800)]
Move the normalization of key lengths from the HashObject to the DHT.

11 years agoAdd property tracking to downloads from peers.
Cameron Dale [Fri, 29 Feb 2008 04:54:45 +0000 (20:54 -0800)]
Add property tracking to downloads from peers.

11 years agoHTTPServer responds correctly to requests for torrent strings.
Cameron Dale [Fri, 29 Feb 2008 00:50:10 +0000 (16:50 -0800)]
HTTPServer responds correctly to requests for torrent strings.

11 years agoUpgrade policies to SVN version and fix a small bug.
Cameron Dale [Fri, 29 Feb 2008 00:17:22 +0000 (16:17 -0800)]
Upgrade policies to SVN version and fix a small bug.

Fixed the policies usage when writeLimit was None to not do anything.

11 years agoStore piece hashes in the DB.
Cameron Dale [Wed, 27 Feb 2008 23:58:05 +0000 (15:58 -0800)]
Store piece hashes in the DB.

Also broke up single table into files and hashes.
Main code uses the new hash routine.
lookupHash now checks for torrent string hashes.
The HTTPServer still needs to interpret torrent string hashes.

11 years agoAdd appropriate piece strings to the DHT.
Cameron Dale [Wed, 27 Feb 2008 20:05:30 +0000 (12:05 -0800)]
Add appropriate piece strings to the DHT.

11 years agoAdded piece hashing to the HashObject.
Cameron Dale [Wed, 27 Feb 2008 19:42:54 +0000 (11:42 -0800)]
Added piece hashing to the HashObject.

11 years agoOnly use the ThrottlingProtocol if the producer is streaming.
Cameron Dale [Wed, 27 Feb 2008 07:05:00 +0000 (23:05 -0800)]
Only use the ThrottlingProtocol if the producer is streaming.

11 years agoFixed the ThrottlingFactory to work with web2 static streams from the web server.
Cameron Dale [Wed, 27 Feb 2008 06:42:23 +0000 (22:42 -0800)]
Fixed the ThrottlingFactory to work with web2 static streams from the web server.

11 years agoAnother attempt at throttling, still not working.
Cameron Dale [Tue, 26 Feb 2008 01:10:58 +0000 (17:10 -0800)]
Another attempt at throttling, still not working.

11 years agoA new TODO item.
Cameron Dale [Mon, 25 Feb 2008 07:04:39 +0000 (23:04 -0800)]
A new TODO item.

11 years agoAlso remove changed cache files during directory scan.
Cameron Dale [Sun, 24 Feb 2008 23:14:00 +0000 (15:14 -0800)]
Also remove changed cache files during directory scan.

11 years agoIgnore unknown cache files during directory scan.
Cameron Dale [Sun, 24 Feb 2008 23:13:15 +0000 (15:13 -0800)]
Ignore unknown cache files during directory scan.

11 years agoStandardize the number of values retrieved from the DHT.
Cameron Dale [Sun, 24 Feb 2008 22:01:13 +0000 (14:01 -0800)]
Standardize the number of values retrieved from the DHT.

Added the new RETRIEVE_VALUES configuration parameter.
The desired number of values can now be negative, indicating
that only the closest STORE_REDUNDANCY nodes are to be queried.

11 years agoRewrite of the actions to take advantage of the commonalities between them.
Cameron Dale [Sun, 24 Feb 2008 08:31:45 +0000 (00:31 -0800)]
Rewrite of the actions to take advantage of the commonalities between them.

11 years agoCheck response packet lengths before sending.
Cameron Dale [Sun, 24 Feb 2008 04:30:26 +0000 (20:30 -0800)]
Check response packet lengths before sending.

11 years agoRemove an unneeded khashmir file.
Cameron Dale [Sun, 24 Feb 2008 01:37:44 +0000 (17:37 -0800)]
Remove an unneeded khashmir file.

11 years agoBreak up the find_value into 2 parts (with get_value).
Cameron Dale [Sat, 23 Feb 2008 02:44:13 +0000 (18:44 -0800)]
Break up the find_value into 2 parts (with get_value).

To better support multiple values per key, the old find_value request is
broken up into find_value, which doesn't stop when it finds a node with
values but just records the number of values the node has, and
get_value, which queries the found nodes that have values to retrieve
them.

11 years agoRefresh DHT values just before they are due to expire.
Cameron Dale [Fri, 22 Feb 2008 23:30:57 +0000 (15:30 -0800)]
Refresh DHT values just before they are due to expire.

11 years agoMove the key expiring to the checkpoint function.
Cameron Dale [Fri, 22 Feb 2008 19:33:50 +0000 (11:33 -0800)]
Move the key expiring to the checkpoint function.

11 years agoUnload the AptPackages caches after a period of inactivity.
Cameron Dale [Fri, 22 Feb 2008 19:22:28 +0000 (11:22 -0800)]
Unload the AptPackages caches after a period of inactivity.

11 years agoUpdated the copyright definitions.
Cameron Dale [Fri, 22 Feb 2008 18:55:16 +0000 (10:55 -0800)]
Updated the copyright definitions.

11 years agoMore and better error messages in the DHT.
Cameron Dale [Fri, 22 Feb 2008 02:40:56 +0000 (18:40 -0800)]
More and better error messages in the DHT.

Also added message verification for the incoming message
to prevent abuse.

11 years agoReturn a token in find_node responses, use it in store_value requests.
Cameron Dale [Thu, 21 Feb 2008 23:18:44 +0000 (15:18 -0800)]
Return a token in find_node responses, use it in store_value requests.

Also changed the checkpointing frequency to 5 minutes so that storing 3
old tokens will give an age of 10-15 minutes (as per protocol).

11 years agoUse compact encoding of peer info for downloads.
Cameron Dale [Thu, 21 Feb 2008 22:27:21 +0000 (14:27 -0800)]
Use compact encoding of peer info for downloads.

11 years agoRemove some extra logging that shouldn't have been in the last commit.
Cameron Dale [Thu, 21 Feb 2008 21:43:03 +0000 (13:43 -0800)]
Remove some extra logging that shouldn't have been in the last commit.

11 years agoUse compact encoding of node contact info in the DHT.
Cameron Dale [Thu, 21 Feb 2008 21:42:02 +0000 (13:42 -0800)]
Use compact encoding of node contact info in the DHT.

11 years agoAllow arbitrary strings to be stored in the DHT database.
Cameron Dale [Thu, 21 Feb 2008 20:11:29 +0000 (12:11 -0800)]
Allow arbitrary strings to be stored in the DHT database.

Also removed the refreshValues as it is no longer needed.

11 years agoRemove the originated time from the DHT value storage.
Cameron Dale [Thu, 21 Feb 2008 19:26:22 +0000 (11:26 -0800)]
Remove the originated time from the DHT value storage.

11 years agoAutomatically set the version in setup.py from the changelog.
Cameron Dale [Thu, 21 Feb 2008 19:10:38 +0000 (11:10 -0800)]
Automatically set the version in setup.py from the changelog.

11 years agoClean up some admin things.
Cameron Dale [Thu, 21 Feb 2008 18:26:46 +0000 (10:26 -0800)]
Clean up some admin things.

11 years agoAdd and clean up some of the epydoc documentation.
Cameron Dale [Thu, 21 Feb 2008 04:59:58 +0000 (20:59 -0800)]
Add and clean up some of the epydoc documentation.

11 years agoAnother new TODO item.
Cameron Dale [Thu, 21 Feb 2008 03:03:28 +0000 (19:03 -0800)]
Another new TODO item.

11 years agoMore work on the TODO.
Cameron Dale [Thu, 21 Feb 2008 02:58:29 +0000 (18:58 -0800)]
More work on the TODO.

11 years agoSnapshot build 2.
Cameron Dale [Wed, 20 Feb 2008 19:36:07 +0000 (11:36 -0800)]
Snapshot build 2.

11 years agoFixed some packaging issues with config file handling.
Cameron Dale [Wed, 20 Feb 2008 19:34:47 +0000 (11:34 -0800)]
Fixed some packaging issues with config file handling.

11 years agoSnapshot build 1.
Cameron Dale [Wed, 20 Feb 2008 18:54:48 +0000 (10:54 -0800)]
Snapshot build 1.

11 years agoAdded 2 (commented) failed attempts to throttle the web server.
Cameron Dale [Wed, 20 Feb 2008 05:52:36 +0000 (21:52 -0800)]
Added 2 (commented) failed attempts to throttle the web server.

11 years agoAnother new TODO item: better DHT lookups for multiple values per key.
Cameron Dale [Wed, 20 Feb 2008 04:45:48 +0000 (20:45 -0800)]
Another new TODO item: better DHT lookups for multiple values per key.

11 years agoFix some minor bugs in the previous commits.
Cameron Dale [Wed, 20 Feb 2008 04:44:54 +0000 (20:44 -0800)]
Fix some minor bugs in the previous commits.

Improper hash_type use in AptPackages.
Use URL quoting and unquoting for peer downloads.
More and better log messages.

11 years agoUse python-debian for parsing RFC 822 files (untested).
Cameron Dale [Tue, 19 Feb 2008 00:17:57 +0000 (16:17 -0800)]
Use python-debian for parsing RFC 822 files (untested).

11 years agoDon't add files to the DHT if a hash could not be found for them.
Cameron Dale [Tue, 19 Feb 2008 00:03:18 +0000 (16:03 -0800)]
Don't add files to the DHT if a hash could not be found for them.

11 years agoDownload from peers using the hash instead of a directory location.
Cameron Dale [Mon, 18 Feb 2008 23:50:48 +0000 (15:50 -0800)]
Download from peers using the hash instead of a directory location.

Also updated the HTTP downloading tests to not use www.camrdale.org
but instead use the less changing IETF RFCs.

11 years agoHTTPServer uses the hash to lookup the file in the DB (no more directories).
Cameron Dale [Mon, 18 Feb 2008 22:40:04 +0000 (14:40 -0800)]
HTTPServer uses the hash to lookup the file in the DB (no more directories).

11 years agoAdd the git-buildpackage config file.
Cameron Dale [Mon, 18 Feb 2008 07:24:17 +0000 (23:24 -0800)]
Add the git-buildpackage config file.

11 years agoFix up some errors in the packaging.
Cameron Dale [Mon, 18 Feb 2008 04:30:06 +0000 (20:30 -0800)]
Fix up some errors in the packaging.

11 years agoInitial debian directory and packaging related changes.
Cameron Dale [Mon, 18 Feb 2008 03:42:45 +0000 (19:42 -0800)]
Initial debian directory and packaging related changes.

11 years agoFinalized initial motivation paper.
Cameron Dale [Wed, 13 Feb 2008 18:27:16 +0000 (10:27 -0800)]
Finalized initial motivation paper.

11 years agoLots of work on the motivating paper, and a new figure.
Cameron Dale [Sat, 9 Feb 2008 01:30:14 +0000 (17:30 -0800)]
Lots of work on the motivating paper, and a new figure.

11 years agoAdded the kile project file.
Cameron Dale [Thu, 7 Feb 2008 00:30:16 +0000 (16:30 -0800)]
Added the kile project file.

11 years agoInitial creation of the motivating paper.
Cameron Dale [Fri, 25 Jan 2008 18:30:33 +0000 (10:30 -0800)]
Initial creation of the motivating paper.

11 years agoMore TODOs.
Cameron Dale [Tue, 22 Jan 2008 21:55:34 +0000 (13:55 -0800)]
More TODOs.

11 years agoUpdates after more though on a couple of of the TODO items.
Cameron Dale [Sat, 19 Jan 2008 01:45:34 +0000 (17:45 -0800)]
Updates after more though on a couple of of the TODO items.

11 years agoAdded more TODO items and a pretty diagram to the documentation.
Cameron Dale [Fri, 18 Jan 2008 01:38:50 +0000 (17:38 -0800)]
Added more TODO items and a pretty diagram to the documentation.

11 years agoChange all print statements to log.msg calls.
Cameron Dale [Wed, 16 Jan 2008 03:58:53 +0000 (19:58 -0800)]
Change all print statements to log.msg calls.

11 years agoSupply a modified factory with increased pipelining and timeouts.
Cameron Dale [Wed, 16 Jan 2008 03:26:29 +0000 (19:26 -0800)]
Supply a modified factory with increased pipelining and timeouts.

11 years agoAdded a test apparatus similar to DebTorrent's.
Cameron Dale [Wed, 16 Jan 2008 00:09:59 +0000 (16:09 -0800)]
Added a test apparatus similar to DebTorrent's.

11 years agoAdd a log-file command line option to the main program.
Cameron Dale [Tue, 15 Jan 2008 22:50:08 +0000 (14:50 -0800)]
Add a log-file command line option to the main program.

11 years agoAdd an option to not error out when only a local IP address can be found.
Cameron Dale [Tue, 15 Jan 2008 22:36:19 +0000 (14:36 -0800)]
Add an option to not error out when only a local IP address can be found.

11 years agoScanning cache directories on startup waits for DHT storeValue to return.
Cameron Dale [Tue, 15 Jan 2008 22:08:14 +0000 (14:08 -0800)]
Scanning cache directories on startup waits for DHT storeValue to return.

This slows down the scanning a lot, but is more consistent for adding
values to the DHT (prevents flooding).

11 years agoLookup the hash in the cache database.
Cameron Dale [Tue, 15 Jan 2008 07:40:21 +0000 (23:40 -0800)]
Lookup the hash in the cache database.

If it's found, return that file.

Still need to implement copying the file to the cache if it's
not in there already (compare directory to cache_dir variable).

11 years agoAdded scanning of other directories for cached packages.
Cameron Dale [Tue, 15 Jan 2008 06:28:26 +0000 (22:28 -0800)]
Added scanning of other directories for cached packages.

11 years agoCacheManager scans the cache directory during initialization.
Cameron Dale [Tue, 15 Jan 2008 05:55:22 +0000 (21:55 -0800)]
CacheManager scans the cache directory during initialization.

Actually the scanning is done after a successful join so that
files can be added to the DHT.

The HashObjects can now hash a file in a separate thread.

The donwload cache is scanned for new files too, causing the
decompressed versions of compressed Packages files to be
hashed and added to the DHT. This isn't currently consistent
since they are not added when they are first downloaded
and decompressed.