quix0rs-apt-p2p.git
12 years agoStandardize the number of values retrieved from the DHT.
Cameron Dale [Sun, 24 Feb 2008 22:01:13 +0000 (14:01 -0800)]
Standardize the number of values retrieved from the DHT.

Added the new RETRIEVE_VALUES configuration parameter.
The desired number of values can now be negative, indicating
that only the closest STORE_REDUNDANCY nodes are to be queried.

12 years agoRewrite of the actions to take advantage of the commonalities between them.
Cameron Dale [Sun, 24 Feb 2008 08:31:45 +0000 (00:31 -0800)]
Rewrite of the actions to take advantage of the commonalities between them.

12 years agoCheck response packet lengths before sending.
Cameron Dale [Sun, 24 Feb 2008 04:30:26 +0000 (20:30 -0800)]
Check response packet lengths before sending.

12 years agoRemove an unneeded khashmir file.
Cameron Dale [Sun, 24 Feb 2008 01:37:44 +0000 (17:37 -0800)]
Remove an unneeded khashmir file.

12 years agoBreak up the find_value into 2 parts (with get_value).
Cameron Dale [Sat, 23 Feb 2008 02:44:13 +0000 (18:44 -0800)]
Break up the find_value into 2 parts (with get_value).

To better support multiple values per key, the old find_value request is
broken up into find_value, which doesn't stop when it finds a node with
values but just records the number of values the node has, and
get_value, which queries the found nodes that have values to retrieve
them.

12 years agoRefresh DHT values just before they are due to expire.
Cameron Dale [Fri, 22 Feb 2008 23:30:57 +0000 (15:30 -0800)]
Refresh DHT values just before they are due to expire.

12 years agoMove the key expiring to the checkpoint function.
Cameron Dale [Fri, 22 Feb 2008 19:33:50 +0000 (11:33 -0800)]
Move the key expiring to the checkpoint function.

12 years agoUnload the AptPackages caches after a period of inactivity.
Cameron Dale [Fri, 22 Feb 2008 19:22:28 +0000 (11:22 -0800)]
Unload the AptPackages caches after a period of inactivity.

12 years agoUpdated the copyright definitions.
Cameron Dale [Fri, 22 Feb 2008 18:55:16 +0000 (10:55 -0800)]
Updated the copyright definitions.

12 years agoMore and better error messages in the DHT.
Cameron Dale [Fri, 22 Feb 2008 02:40:56 +0000 (18:40 -0800)]
More and better error messages in the DHT.

Also added message verification for the incoming message
to prevent abuse.

12 years agoReturn a token in find_node responses, use it in store_value requests.
Cameron Dale [Thu, 21 Feb 2008 23:18:44 +0000 (15:18 -0800)]
Return a token in find_node responses, use it in store_value requests.

Also changed the checkpointing frequency to 5 minutes so that storing 3
old tokens will give an age of 10-15 minutes (as per protocol).

12 years agoUse compact encoding of peer info for downloads.
Cameron Dale [Thu, 21 Feb 2008 22:27:21 +0000 (14:27 -0800)]
Use compact encoding of peer info for downloads.

12 years agoRemove some extra logging that shouldn't have been in the last commit.
Cameron Dale [Thu, 21 Feb 2008 21:43:03 +0000 (13:43 -0800)]
Remove some extra logging that shouldn't have been in the last commit.

12 years agoUse compact encoding of node contact info in the DHT.
Cameron Dale [Thu, 21 Feb 2008 21:42:02 +0000 (13:42 -0800)]
Use compact encoding of node contact info in the DHT.

12 years agoAllow arbitrary strings to be stored in the DHT database.
Cameron Dale [Thu, 21 Feb 2008 20:11:29 +0000 (12:11 -0800)]
Allow arbitrary strings to be stored in the DHT database.

Also removed the refreshValues as it is no longer needed.

12 years agoRemove the originated time from the DHT value storage.
Cameron Dale [Thu, 21 Feb 2008 19:26:22 +0000 (11:26 -0800)]
Remove the originated time from the DHT value storage.

12 years agoAutomatically set the version in setup.py from the changelog.
Cameron Dale [Thu, 21 Feb 2008 19:10:38 +0000 (11:10 -0800)]
Automatically set the version in setup.py from the changelog.

12 years agoClean up some admin things.
Cameron Dale [Thu, 21 Feb 2008 18:26:46 +0000 (10:26 -0800)]
Clean up some admin things.

12 years agoAdd and clean up some of the epydoc documentation.
Cameron Dale [Thu, 21 Feb 2008 04:59:58 +0000 (20:59 -0800)]
Add and clean up some of the epydoc documentation.

12 years agoAnother new TODO item.
Cameron Dale [Thu, 21 Feb 2008 03:03:28 +0000 (19:03 -0800)]
Another new TODO item.

12 years agoMore work on the TODO.
Cameron Dale [Thu, 21 Feb 2008 02:58:29 +0000 (18:58 -0800)]
More work on the TODO.

12 years agoSnapshot build 2.
Cameron Dale [Wed, 20 Feb 2008 19:36:07 +0000 (11:36 -0800)]
Snapshot build 2.

12 years agoFixed some packaging issues with config file handling.
Cameron Dale [Wed, 20 Feb 2008 19:34:47 +0000 (11:34 -0800)]
Fixed some packaging issues with config file handling.

12 years agoSnapshot build 1.
Cameron Dale [Wed, 20 Feb 2008 18:54:48 +0000 (10:54 -0800)]
Snapshot build 1.

12 years agoAdded 2 (commented) failed attempts to throttle the web server.
Cameron Dale [Wed, 20 Feb 2008 05:52:36 +0000 (21:52 -0800)]
Added 2 (commented) failed attempts to throttle the web server.

12 years agoAnother new TODO item: better DHT lookups for multiple values per key.
Cameron Dale [Wed, 20 Feb 2008 04:45:48 +0000 (20:45 -0800)]
Another new TODO item: better DHT lookups for multiple values per key.

12 years agoFix some minor bugs in the previous commits.
Cameron Dale [Wed, 20 Feb 2008 04:44:54 +0000 (20:44 -0800)]
Fix some minor bugs in the previous commits.

Improper hash_type use in AptPackages.
Use URL quoting and unquoting for peer downloads.
More and better log messages.

12 years agoUse python-debian for parsing RFC 822 files (untested).
Cameron Dale [Tue, 19 Feb 2008 00:17:57 +0000 (16:17 -0800)]
Use python-debian for parsing RFC 822 files (untested).

12 years agoDon't add files to the DHT if a hash could not be found for them.
Cameron Dale [Tue, 19 Feb 2008 00:03:18 +0000 (16:03 -0800)]
Don't add files to the DHT if a hash could not be found for them.

12 years agoDownload from peers using the hash instead of a directory location.
Cameron Dale [Mon, 18 Feb 2008 23:50:48 +0000 (15:50 -0800)]
Download from peers using the hash instead of a directory location.

Also updated the HTTP downloading tests to not use www.camrdale.org
but instead use the less changing IETF RFCs.

12 years agoHTTPServer uses the hash to lookup the file in the DB (no more directories).
Cameron Dale [Mon, 18 Feb 2008 22:40:04 +0000 (14:40 -0800)]
HTTPServer uses the hash to lookup the file in the DB (no more directories).

12 years agoAdd the git-buildpackage config file.
Cameron Dale [Mon, 18 Feb 2008 07:24:17 +0000 (23:24 -0800)]
Add the git-buildpackage config file.

12 years agoFix up some errors in the packaging.
Cameron Dale [Mon, 18 Feb 2008 04:30:06 +0000 (20:30 -0800)]
Fix up some errors in the packaging.

12 years agoInitial debian directory and packaging related changes.
Cameron Dale [Mon, 18 Feb 2008 03:42:45 +0000 (19:42 -0800)]
Initial debian directory and packaging related changes.

12 years agoFinalized initial motivation paper.
Cameron Dale [Wed, 13 Feb 2008 18:27:16 +0000 (10:27 -0800)]
Finalized initial motivation paper.

12 years agoLots of work on the motivating paper, and a new figure.
Cameron Dale [Sat, 9 Feb 2008 01:30:14 +0000 (17:30 -0800)]
Lots of work on the motivating paper, and a new figure.

12 years agoAdded the kile project file.
Cameron Dale [Thu, 7 Feb 2008 00:30:16 +0000 (16:30 -0800)]
Added the kile project file.

12 years agoInitial creation of the motivating paper.
Cameron Dale [Fri, 25 Jan 2008 18:30:33 +0000 (10:30 -0800)]
Initial creation of the motivating paper.

12 years agoMore TODOs.
Cameron Dale [Tue, 22 Jan 2008 21:55:34 +0000 (13:55 -0800)]
More TODOs.

12 years agoUpdates after more though on a couple of of the TODO items.
Cameron Dale [Sat, 19 Jan 2008 01:45:34 +0000 (17:45 -0800)]
Updates after more though on a couple of of the TODO items.

12 years agoAdded more TODO items and a pretty diagram to the documentation.
Cameron Dale [Fri, 18 Jan 2008 01:38:50 +0000 (17:38 -0800)]
Added more TODO items and a pretty diagram to the documentation.

12 years agoChange all print statements to log.msg calls.
Cameron Dale [Wed, 16 Jan 2008 03:58:53 +0000 (19:58 -0800)]
Change all print statements to log.msg calls.

12 years agoSupply a modified factory with increased pipelining and timeouts.
Cameron Dale [Wed, 16 Jan 2008 03:26:29 +0000 (19:26 -0800)]
Supply a modified factory with increased pipelining and timeouts.

12 years agoAdded a test apparatus similar to DebTorrent's.
Cameron Dale [Wed, 16 Jan 2008 00:09:59 +0000 (16:09 -0800)]
Added a test apparatus similar to DebTorrent's.

12 years agoAdd a log-file command line option to the main program.
Cameron Dale [Tue, 15 Jan 2008 22:50:08 +0000 (14:50 -0800)]
Add a log-file command line option to the main program.

12 years agoAdd an option to not error out when only a local IP address can be found.
Cameron Dale [Tue, 15 Jan 2008 22:36:19 +0000 (14:36 -0800)]
Add an option to not error out when only a local IP address can be found.

12 years agoScanning cache directories on startup waits for DHT storeValue to return.
Cameron Dale [Tue, 15 Jan 2008 22:08:14 +0000 (14:08 -0800)]
Scanning cache directories on startup waits for DHT storeValue to return.

This slows down the scanning a lot, but is more consistent for adding
values to the DHT (prevents flooding).

12 years agoLookup the hash in the cache database.
Cameron Dale [Tue, 15 Jan 2008 07:40:21 +0000 (23:40 -0800)]
Lookup the hash in the cache database.

If it's found, return that file.

Still need to implement copying the file to the cache if it's
not in there already (compare directory to cache_dir variable).

12 years agoAdded scanning of other directories for cached packages.
Cameron Dale [Tue, 15 Jan 2008 06:28:26 +0000 (22:28 -0800)]
Added scanning of other directories for cached packages.

12 years agoCacheManager scans the cache directory during initialization.
Cameron Dale [Tue, 15 Jan 2008 05:55:22 +0000 (21:55 -0800)]
CacheManager scans the cache directory during initialization.

Actually the scanning is done after a successful join so that
files can be added to the DHT.

The HashObjects can now hash a file in a separate thread.

The donwload cache is scanned for new files too, causing the
decompressed versions of compressed Packages files to be
hashed and added to the DHT. This isn't currently consistent
since they are not added when they are first downloaded
and decompressed.

12 years agoRemove a completed TODO item.
Cameron Dale [Tue, 15 Jan 2008 01:29:38 +0000 (17:29 -0800)]
Remove a completed TODO item.

12 years agoRemove the mirror name from the shared HTTP directory.
Cameron Dale [Mon, 14 Jan 2008 07:24:39 +0000 (23:24 -0800)]
Remove the mirror name from the shared HTTP directory.

12 years agoReset the HTTPServer subdirectories when a new cache directory is created.
Cameron Dale [Mon, 14 Jan 2008 06:58:46 +0000 (22:58 -0800)]
Reset the HTTPServer subdirectories when a new cache directory is created.

12 years agoUse FilePath everywhere and create new CacheManager module.
Cameron Dale [Mon, 14 Jan 2008 06:49:16 +0000 (22:49 -0800)]
Use FilePath everywhere and create new CacheManager module.

All cache related things moved from MirrorManager to the new
CacheManager.

12 years agokhashmir's store value takes the origination date.
Cameron Dale [Sun, 13 Jan 2008 22:02:59 +0000 (14:02 -0800)]
khashmir's store value takes the origination date.

12 years agoAdd the ability to bencode datetime values.
Cameron Dale [Sun, 13 Jan 2008 21:49:27 +0000 (13:49 -0800)]
Add the ability to bencode datetime values.

12 years agobencode module now uses new BencodeError instead of ValueError.
Cameron Dale [Sun, 13 Jan 2008 21:11:44 +0000 (13:11 -0800)]
bencode module now uses new BencodeError instead of ValueError.

12 years agoAdded unittests to bencode module for boolean and unicode types.
Cameron Dale [Sun, 13 Jan 2008 21:05:21 +0000 (13:05 -0800)]
Added unittests to bencode module for boolean and unicode types.

12 years agoChanged bencode's test functions to trial unittests.
Cameron Dale [Sun, 13 Jan 2008 20:55:23 +0000 (12:55 -0800)]
Changed bencode's test functions to trial unittests.

12 years agoConvert the new bencode module to use apt-dht logging.
Cameron Dale [Sun, 13 Jan 2008 19:38:17 +0000 (11:38 -0800)]
Convert the new bencode module to use apt-dht logging.

12 years agoCopied the more advanced bencode module from DebTorrent.
Cameron Dale [Sun, 13 Jan 2008 19:19:04 +0000 (11:19 -0800)]
Copied the more advanced bencode module from DebTorrent.

12 years agoMove the main util functions to the new main util module.
Cameron Dale [Sat, 12 Jan 2008 19:28:13 +0000 (11:28 -0800)]
Move the main util functions to the new main util module.

12 years agoMoved the finding IP address function to the util module.
Cameron Dale [Sat, 12 Jan 2008 18:16:32 +0000 (10:16 -0800)]
Moved the finding IP address function to the util module.

12 years agoUse the new DB in the main code.
Cameron Dale [Sat, 12 Jan 2008 18:06:11 +0000 (10:06 -0800)]
Use the new DB in the main code.

12 years agoHTTPServer now gets it's dictionary of subdirectories externally.
Cameron Dale [Sat, 12 Jan 2008 17:53:38 +0000 (09:53 -0800)]
HTTPServer now gets it's dictionary of subdirectories externally.

12 years agoMade the main DB track url top-level directories.
Cameron Dale [Sat, 12 Jan 2008 08:34:48 +0000 (00:34 -0800)]
Made the main DB track url top-level directories.

Need to change HTTPServer to use the DB, and the main code as
well.

Don't forget to call reconcileDirectories periodically and
then if changed update the HTTPServer's directories.

12 years agoMain database finished for now, including unittests.
Cameron Dale [Sat, 12 Jan 2008 04:14:53 +0000 (20:14 -0800)]
Main database finished for now, including unittests.

12 years agoAdded new database module for the main code.
Cameron Dale [Sat, 12 Jan 2008 00:44:38 +0000 (16:44 -0800)]
Added new database module for the main code.

Still needs more functions, and the unittests are just
copied from khashmir's database.

12 years agoMake khashmir's db backwards compatible to pysqlite 2.0.x
Cameron Dale [Fri, 11 Jan 2008 22:58:18 +0000 (14:58 -0800)]
Make khashmir's db backwards compatible to pysqlite 2.0.x

12 years agoUpgrade the security in khashmir by using longer TIDs.
Cameron Dale [Fri, 11 Jan 2008 18:51:32 +0000 (10:51 -0800)]
Upgrade the security in khashmir by using longer TIDs.

12 years agoAdded some things to do for the future.
Cameron Dale [Fri, 11 Jan 2008 04:49:48 +0000 (20:49 -0800)]
Added some things to do for the future.

12 years agoWhen peer downloads fail, try direct downloads.
Cameron Dale [Fri, 11 Jan 2008 04:34:21 +0000 (20:34 -0800)]
When peer downloads fail, try direct downloads.

Also don't cache failed download results, just return them.

12 years agoAdds downloaded hashed files to the DHT.
Cameron Dale [Fri, 11 Jan 2008 03:53:47 +0000 (19:53 -0800)]
Adds downloaded hashed files to the DHT.

12 years agoAdded complicated testing to find our IP address.
Cameron Dale [Fri, 11 Jan 2008 03:31:06 +0000 (19:31 -0800)]
Added complicated testing to find our IP address.

12 years agoNew DHT method 'join' like 'ping' but returns our IP and port.
Cameron Dale [Fri, 11 Jan 2008 01:07:23 +0000 (17:07 -0800)]
New DHT method 'join' like 'ping' but returns our IP and port.

12 years agoMake downloaded files accessible via the HTTP server.
Cameron Dale [Fri, 11 Jan 2008 00:40:23 +0000 (16:40 -0800)]
Make downloaded files accessible via the HTTP server.

MirrorManager calls back to the main code, which adds an
appropriate ~ (tilde) directory to the web server.

12 years agoFixed the storage of binary strings in the database.
Cameron Dale [Thu, 10 Jan 2008 23:47:43 +0000 (15:47 -0800)]
Fixed the storage of binary strings in the database.

Errors were caused when a string stored in the database had
a NUL character in it. Now the DB uses base64 encoding for
all hash values, with automatic conversion using adapters
and converters.

Also added some unittests to the DB.

12 years agoUse function for sending krpc responses, and add spew parameter.
Cameron Dale [Thu, 10 Jan 2008 20:45:04 +0000 (12:45 -0800)]
Use function for sending krpc responses, and add spew parameter.

Sending a response in the krpc module now calls a function.

There is a new 'SPEW' config variable that causes the krpc
module to spew out all sent and received packet data
(previously called noisy).

Also changed all the tests to use failUnlessEqual instead
of assertEqual, and all assert() commands no longer use
brackets.

Updated some of the downloading tests for the new file sizes
on camrdale.org (need a better solution here).

12 years agoFixed some bugs in the new hashing scheme and tests.
Cameron Dale [Thu, 10 Jan 2008 07:07:24 +0000 (23:07 -0800)]
Fixed some bugs in the new hashing scheme and tests.

12 years agoPass the new HashObjects around everywhere (untested).
Cameron Dale [Thu, 10 Jan 2008 00:54:20 +0000 (16:54 -0800)]
Pass the new HashObjects around everywhere (untested).

12 years agoModified the HashObject to do dummy hashing if the hash has already been verified.
Cameron Dale [Thu, 10 Jan 2008 00:53:34 +0000 (16:53 -0800)]
Modified the HashObject to do dummy hashing if the hash has already been verified.

12 years agoAdded a new Hash Object for tracking lots of hash details.
Cameron Dale [Thu, 10 Jan 2008 00:15:32 +0000 (16:15 -0800)]
Added a new Hash Object for tracking lots of hash details.

12 years agoAdded more debug printing to noisy krpc protocol.
Cameron Dale [Wed, 9 Jan 2008 05:49:34 +0000 (21:49 -0800)]
Added more debug printing to noisy krpc protocol.

12 years agoProxyFileStream also calculates hash while downloading.
Cameron Dale [Wed, 9 Jan 2008 05:47:48 +0000 (21:47 -0800)]
ProxyFileStream also calculates hash while downloading.

Needs a better implementation with a HashObject type.

12 years agoMake the MultiDHT test_store work for large number of nodes.
Cameron Dale [Wed, 9 Jan 2008 04:45:21 +0000 (20:45 -0800)]
Make the MultiDHT test_store work for large number of nodes.

It generated errors before as all the requests for get/store
were sent out simultaneously, and some got lost (probably due
to network queue lengths). Instead, send each request and
wait for it to complete.

12 years agoImprove the creation of nodes and move all to the main khashmir class.
Cameron Dale [Wed, 9 Jan 2008 03:11:59 +0000 (19:11 -0800)]
Improve the creation of nodes and move all to the main khashmir class.

Nodes are now initialized with their id, host and port
(optionally stored in a dict) instead of needing 2 steps.

All other classes now call the main khashmir Node() constructor
which adds the udp connection and table.

Previous actions that called insertNode on the table now
call it on the main khashmir class, so that if buckets are
full they can have a chance of being added.

12 years agoImprove the stopping of the krpc protocol so no timeouts are left.
Cameron Dale [Wed, 9 Jan 2008 03:08:21 +0000 (19:08 -0800)]
Improve the stopping of the krpc protocol so no timeouts are left.

12 years agoAvoid repetition, use a function for updating nodes when they contact us.
Cameron Dale [Tue, 8 Jan 2008 23:08:18 +0000 (15:08 -0800)]
Avoid repetition, use a function for updating nodes when they contact us.

12 years agoSwitch from the time module to the datetime module.
Cameron Dale [Tue, 8 Jan 2008 22:48:21 +0000 (14:48 -0800)]
Switch from the time module to the datetime module.

12 years agoUpgrade pysqlite from 1.0 to 2.x (now uses sqlite3).
Cameron Dale [Tue, 8 Jan 2008 21:40:44 +0000 (13:40 -0800)]
Upgrade pysqlite from 1.0 to 2.x (now uses sqlite3).

12 years agoNo need to look at the DEFAULT section when parsing the config.
Cameron Dale [Tue, 8 Jan 2008 20:38:58 +0000 (12:38 -0800)]
No need to look at the DEFAULT section when parsing the config.

12 years agoAdd back the updatedFile method to the MirrorManager.
Cameron Dale [Tue, 8 Jan 2008 20:38:18 +0000 (12:38 -0800)]
Add back the updatedFile method to the MirrorManager.

12 years agoMove all the khashmir database operations to a separate module.
Cameron Dale [Tue, 8 Jan 2008 20:33:18 +0000 (12:33 -0800)]
Move all the khashmir database operations to a separate module.

12 years agoMove the translating of hashes from hex into the MirrorManager.
Cameron Dale [Mon, 7 Jan 2008 19:37:41 +0000 (11:37 -0800)]
Move the translating of hashes from hex into the MirrorManager.

12 years agoDHT normalizes key lengths by chopping or adding 0.
Cameron Dale [Mon, 7 Jan 2008 19:28:14 +0000 (11:28 -0800)]
DHT normalizes key lengths by chopping or adding 0.

12 years agoDecompress needed files while downloading them.
Cameron Dale [Mon, 7 Jan 2008 08:00:57 +0000 (00:00 -0800)]
Decompress needed files while downloading them.

Decompress the files so they can be tracked by AptPackages
and used to generate hashes.

12 years agoCall the AptPackages file_updated when files complete downloading.
Cameron Dale [Mon, 7 Jan 2008 05:10:21 +0000 (21:10 -0800)]
Call the AptPackages file_updated when files complete downloading.

Added a defer to the strem proxy to file to facilitate.
Moved the modtime updating out of the proxy as it makes
more sense to do it in the MirrorManager.

12 years agoFix an error that occurred when there were no Sources in AptPackages.
Cameron Dale [Mon, 7 Jan 2008 05:08:17 +0000 (21:08 -0800)]
Fix an error that occurred when there were no Sources in AptPackages.

Also add an errback to the findHash call to trap future errors.

12 years agoUnhexify the returned hashes from the AptPackages lookup.
Cameron Dale [Mon, 7 Jan 2008 05:06:48 +0000 (21:06 -0800)]
Unhexify the returned hashes from the AptPackages lookup.

12 years agoAdded modification time tracking to the MirrorManager cache.
Cameron Dale [Mon, 7 Jan 2008 04:20:09 +0000 (20:20 -0800)]
Added modification time tracking to the MirrorManager cache.

Had to switch the HTTPServer to use renderHTTP() instead of
render() so that the last-modified HTTP header would be set.