]> git.mxchange.org Git - quix0rs-apt-p2p.git/log
quix0rs-apt-p2p.git
16 years agoMade the main DB track url top-level directories.
Cameron Dale [Sat, 12 Jan 2008 08:34:48 +0000 (00:34 -0800)]
Made the main DB track url top-level directories.

Need to change HTTPServer to use the DB, and the main code as
well.

Don't forget to call reconcileDirectories periodically and
then if changed update the HTTPServer's directories.

16 years agoMain database finished for now, including unittests.
Cameron Dale [Sat, 12 Jan 2008 04:14:53 +0000 (20:14 -0800)]
Main database finished for now, including unittests.

16 years agoAdded new database module for the main code.
Cameron Dale [Sat, 12 Jan 2008 00:44:38 +0000 (16:44 -0800)]
Added new database module for the main code.

Still needs more functions, and the unittests are just
copied from khashmir's database.

16 years agoMake khashmir's db backwards compatible to pysqlite 2.0.x
Cameron Dale [Fri, 11 Jan 2008 22:58:18 +0000 (14:58 -0800)]
Make khashmir's db backwards compatible to pysqlite 2.0.x

16 years agoUpgrade the security in khashmir by using longer TIDs.
Cameron Dale [Fri, 11 Jan 2008 18:51:32 +0000 (10:51 -0800)]
Upgrade the security in khashmir by using longer TIDs.

16 years agoAdded some things to do for the future.
Cameron Dale [Fri, 11 Jan 2008 04:49:48 +0000 (20:49 -0800)]
Added some things to do for the future.

16 years agoWhen peer downloads fail, try direct downloads.
Cameron Dale [Fri, 11 Jan 2008 04:34:21 +0000 (20:34 -0800)]
When peer downloads fail, try direct downloads.

Also don't cache failed download results, just return them.

16 years agoAdds downloaded hashed files to the DHT.
Cameron Dale [Fri, 11 Jan 2008 03:53:47 +0000 (19:53 -0800)]
Adds downloaded hashed files to the DHT.

16 years agoAdded complicated testing to find our IP address.
Cameron Dale [Fri, 11 Jan 2008 03:31:06 +0000 (19:31 -0800)]
Added complicated testing to find our IP address.

16 years agoNew DHT method 'join' like 'ping' but returns our IP and port.
Cameron Dale [Fri, 11 Jan 2008 01:07:23 +0000 (17:07 -0800)]
New DHT method 'join' like 'ping' but returns our IP and port.

16 years agoMake downloaded files accessible via the HTTP server.
Cameron Dale [Fri, 11 Jan 2008 00:40:23 +0000 (16:40 -0800)]
Make downloaded files accessible via the HTTP server.

MirrorManager calls back to the main code, which adds an
appropriate ~ (tilde) directory to the web server.

16 years agoFixed the storage of binary strings in the database.
Cameron Dale [Thu, 10 Jan 2008 23:47:43 +0000 (15:47 -0800)]
Fixed the storage of binary strings in the database.

Errors were caused when a string stored in the database had
a NUL character in it. Now the DB uses base64 encoding for
all hash values, with automatic conversion using adapters
and converters.

Also added some unittests to the DB.

16 years agoUse function for sending krpc responses, and add spew parameter.
Cameron Dale [Thu, 10 Jan 2008 20:45:04 +0000 (12:45 -0800)]
Use function for sending krpc responses, and add spew parameter.

Sending a response in the krpc module now calls a function.

There is a new 'SPEW' config variable that causes the krpc
module to spew out all sent and received packet data
(previously called noisy).

Also changed all the tests to use failUnlessEqual instead
of assertEqual, and all assert() commands no longer use
brackets.

Updated some of the downloading tests for the new file sizes
on camrdale.org (need a better solution here).

16 years agoFixed some bugs in the new hashing scheme and tests.
Cameron Dale [Thu, 10 Jan 2008 07:07:24 +0000 (23:07 -0800)]
Fixed some bugs in the new hashing scheme and tests.

16 years agoPass the new HashObjects around everywhere (untested).
Cameron Dale [Thu, 10 Jan 2008 00:54:20 +0000 (16:54 -0800)]
Pass the new HashObjects around everywhere (untested).

16 years agoModified the HashObject to do dummy hashing if the hash has already been verified.
Cameron Dale [Thu, 10 Jan 2008 00:53:34 +0000 (16:53 -0800)]
Modified the HashObject to do dummy hashing if the hash has already been verified.

16 years agoAdded a new Hash Object for tracking lots of hash details.
Cameron Dale [Thu, 10 Jan 2008 00:15:32 +0000 (16:15 -0800)]
Added a new Hash Object for tracking lots of hash details.

16 years agoAdded more debug printing to noisy krpc protocol.
Cameron Dale [Wed, 9 Jan 2008 05:49:34 +0000 (21:49 -0800)]
Added more debug printing to noisy krpc protocol.

16 years agoProxyFileStream also calculates hash while downloading.
Cameron Dale [Wed, 9 Jan 2008 05:47:48 +0000 (21:47 -0800)]
ProxyFileStream also calculates hash while downloading.

Needs a better implementation with a HashObject type.

16 years agoMake the MultiDHT test_store work for large number of nodes.
Cameron Dale [Wed, 9 Jan 2008 04:45:21 +0000 (20:45 -0800)]
Make the MultiDHT test_store work for large number of nodes.

It generated errors before as all the requests for get/store
were sent out simultaneously, and some got lost (probably due
to network queue lengths). Instead, send each request and
wait for it to complete.

16 years agoImprove the creation of nodes and move all to the main khashmir class.
Cameron Dale [Wed, 9 Jan 2008 03:11:59 +0000 (19:11 -0800)]
Improve the creation of nodes and move all to the main khashmir class.

Nodes are now initialized with their id, host and port
(optionally stored in a dict) instead of needing 2 steps.

All other classes now call the main khashmir Node() constructor
which adds the udp connection and table.

Previous actions that called insertNode on the table now
call it on the main khashmir class, so that if buckets are
full they can have a chance of being added.

16 years agoImprove the stopping of the krpc protocol so no timeouts are left.
Cameron Dale [Wed, 9 Jan 2008 03:08:21 +0000 (19:08 -0800)]
Improve the stopping of the krpc protocol so no timeouts are left.

16 years agoAvoid repetition, use a function for updating nodes when they contact us.
Cameron Dale [Tue, 8 Jan 2008 23:08:18 +0000 (15:08 -0800)]
Avoid repetition, use a function for updating nodes when they contact us.

16 years agoSwitch from the time module to the datetime module.
Cameron Dale [Tue, 8 Jan 2008 22:48:21 +0000 (14:48 -0800)]
Switch from the time module to the datetime module.

16 years agoUpgrade pysqlite from 1.0 to 2.x (now uses sqlite3).
Cameron Dale [Tue, 8 Jan 2008 21:40:44 +0000 (13:40 -0800)]
Upgrade pysqlite from 1.0 to 2.x (now uses sqlite3).

16 years agoNo need to look at the DEFAULT section when parsing the config.
Cameron Dale [Tue, 8 Jan 2008 20:38:58 +0000 (12:38 -0800)]
No need to look at the DEFAULT section when parsing the config.

16 years agoAdd back the updatedFile method to the MirrorManager.
Cameron Dale [Tue, 8 Jan 2008 20:38:18 +0000 (12:38 -0800)]
Add back the updatedFile method to the MirrorManager.

16 years agoMove all the khashmir database operations to a separate module.
Cameron Dale [Tue, 8 Jan 2008 20:33:18 +0000 (12:33 -0800)]
Move all the khashmir database operations to a separate module.

16 years agoMove the translating of hashes from hex into the MirrorManager.
Cameron Dale [Mon, 7 Jan 2008 19:37:41 +0000 (11:37 -0800)]
Move the translating of hashes from hex into the MirrorManager.

16 years agoDHT normalizes key lengths by chopping or adding 0.
Cameron Dale [Mon, 7 Jan 2008 19:28:14 +0000 (11:28 -0800)]
DHT normalizes key lengths by chopping or adding 0.

16 years agoDecompress needed files while downloading them.
Cameron Dale [Mon, 7 Jan 2008 08:00:57 +0000 (00:00 -0800)]
Decompress needed files while downloading them.

Decompress the files so they can be tracked by AptPackages
and used to generate hashes.

16 years agoCall the AptPackages file_updated when files complete downloading.
Cameron Dale [Mon, 7 Jan 2008 05:10:21 +0000 (21:10 -0800)]
Call the AptPackages file_updated when files complete downloading.

Added a defer to the strem proxy to file to facilitate.
Moved the modtime updating out of the proxy as it makes
more sense to do it in the MirrorManager.

16 years agoFix an error that occurred when there were no Sources in AptPackages.
Cameron Dale [Mon, 7 Jan 2008 05:08:17 +0000 (21:08 -0800)]
Fix an error that occurred when there were no Sources in AptPackages.

Also add an errback to the findHash call to trap future errors.

16 years agoUnhexify the returned hashes from the AptPackages lookup.
Cameron Dale [Mon, 7 Jan 2008 05:06:48 +0000 (21:06 -0800)]
Unhexify the returned hashes from the AptPackages lookup.

16 years agoAdded modification time tracking to the MirrorManager cache.
Cameron Dale [Mon, 7 Jan 2008 04:20:09 +0000 (20:20 -0800)]
Added modification time tracking to the MirrorManager cache.

Had to switch the HTTPServer to use renderHTTP() instead of
render() so that the last-modified HTTP header would be set.

16 years agoAdded caching of downloaded files to the MirrorManager.
Cameron Dale [Mon, 7 Jan 2008 02:55:31 +0000 (18:55 -0800)]
Added caching of downloaded files to the MirrorManager.

Also made the MirrorManager work with full URLs (http://...).
Also added some log messages to the HTTPDownloader.

16 years agoLots of log messages and blocked non-local access to server.
Cameron Dale [Fri, 4 Jan 2008 02:58:53 +0000 (18:58 -0800)]
Lots of log messages and blocked non-local access to server.

Remote peers can still access the shared files, but not make
requests that result in lookups or see the stats.

Also fixed a bug in the main findHash routines.
Also changed to use splitHostPort rather than manual splitting.

16 years agoMade the DHT bootstrapping use IPs instead of DNS names.
Cameron Dale [Fri, 4 Jan 2008 01:43:14 +0000 (17:43 -0800)]
Made the DHT bootstrapping use IPs instead of DNS names.

Also added some logging messages.
Also fixed a problem that caused the application to be loaded twice.

16 years agoMove all url parsing into the PeerManager.
Cameron Dale [Fri, 4 Jan 2008 00:01:38 +0000 (16:01 -0800)]
Move all url parsing into the PeerManager.

Also added a lookupHash_done function to the main code.

16 years agoFixed some minor bugs.
Cameron Dale [Thu, 3 Jan 2008 23:51:22 +0000 (15:51 -0800)]
Fixed some minor bugs.

16 years agoMade the get and storeValue DHT functions work.
Cameron Dale [Thu, 3 Jan 2008 02:06:46 +0000 (18:06 -0800)]
Made the get and storeValue DHT functions work.

Also added tests for them.
Had to modify the callback from khashmir's get and storeValue functions
to include the key, and the key and value, respectively so multiple
calls can be tracked.

16 years agoUpdated the DHT join and leave to check if a join is in progress.
Cameron Dale [Wed, 2 Jan 2008 21:52:59 +0000 (13:52 -0800)]
Updated the DHT join and leave to check if a join is in progress.

16 years agoMade the DHT join method work and added tests for it.
Cameron Dale [Wed, 2 Jan 2008 21:45:40 +0000 (13:45 -0800)]
Made the DHT join method work and added tests for it.

16 years agoAdded a shutdown method to the khasmir DHT.
Cameron Dale [Wed, 2 Jan 2008 19:48:37 +0000 (11:48 -0800)]
Added a shutdown method to the khasmir DHT.

16 years agoAdd timeouts to some unittests.
Cameron [Mon, 24 Dec 2007 22:42:08 +0000 (14:42 -0800)]
Add timeouts to some unittests.

16 years agoFixed 2 broken AptPackages unittests.
Cameron [Mon, 24 Dec 2007 22:37:01 +0000 (14:37 -0800)]
Fixed 2 broken AptPackages unittests.

16 years agoDisable the deprecation warning for the khashmir tests.
Cameron Dale [Fri, 21 Dec 2007 03:52:43 +0000 (19:52 -0800)]
Disable the deprecation warning for the khashmir tests.

Need to rewrite the khashmir main functions to use Deferreds
or something.

16 years agoChange all unittests to use twisted's trial.
Cameron Dale [Fri, 21 Dec 2007 03:43:25 +0000 (19:43 -0800)]
Change all unittests to use twisted's trial.

Merge separate "test_*.py" files into their modules.

16 years agoModify khashmir's config system to not use the const module.
Cameron Dale [Fri, 21 Dec 2007 01:54:53 +0000 (17:54 -0800)]
Modify khashmir's config system to not use the const module.

Configuration variables are all passed in as a dictionary where needed.
The NULL_ID config option has been made a constant.
The ConfigParser now stores all keys as upper case.

16 years agoAdd more to khashmir's DHT implementation.
Cameron Dale [Wed, 19 Dec 2007 21:48:25 +0000 (13:48 -0800)]
Add more to khashmir's DHT implementation.

Still needs a massive rewrite for loading config values properly.

16 years agoAdd a MirrorError exception.
Cameron [Wed, 19 Dec 2007 00:24:57 +0000 (16:24 -0800)]
Add a MirrorError exception.

Fixes the deprecated use of calling errback with a string.

16 years agoUpdate all tests to add the deferred as both errback and callback.
Cameron [Wed, 19 Dec 2007 00:17:17 +0000 (16:17 -0800)]
Update all tests to add the deferred as both errback and callback.

16 years agoFix the MirrorManager tests to be dist-agnostic too.
Cameron [Wed, 19 Dec 2007 00:06:32 +0000 (16:06 -0800)]
Fix the MirrorManager tests to be dist-agnostic too.

16 years agoFix some broken AptPackages tests.
Cameron [Wed, 19 Dec 2007 00:01:06 +0000 (16:01 -0800)]
Fix some broken AptPackages tests.

16 years agoMade the AptPackages tests dist-agnostic so it works with Ubuntu.
Cameron [Tue, 18 Dec 2007 23:39:44 +0000 (15:39 -0800)]
Made the AptPackages tests dist-agnostic so it works with Ubuntu.

16 years agoAdded the new DHT interface and a bare implmentation of it.
Cameron Dale [Mon, 17 Dec 2007 02:41:27 +0000 (18:41 -0800)]
Added the new DHT interface and a bare implmentation of it.

16 years agoAdded the DHT starting to the application.
Cameron Dale [Mon, 17 Dec 2007 02:39:43 +0000 (18:39 -0800)]
Added the DHT starting to the application.

16 years agoRemoved the old .cvsignore from khashmir.
Cameron Dale [Sun, 16 Dec 2007 21:19:50 +0000 (13:19 -0800)]
Removed the old .cvsignore from khashmir.

16 years agoAdd some of the fetching logic.
Cameron Dale [Sun, 16 Dec 2007 09:09:41 +0000 (01:09 -0800)]
Add some of the fetching logic.

16 years agoFix up the application to work with then new structure.
Cameron Dale [Sun, 16 Dec 2007 04:02:15 +0000 (20:02 -0800)]
Fix up the application to work with then new structure.

16 years agoMoved the files to appropriate package directories.
Cameron Dale [Sun, 16 Dec 2007 03:53:41 +0000 (19:53 -0800)]
Moved the files to appropriate package directories.

16 years agoMake it also work from the command-line.
Cameron Dale [Sun, 16 Dec 2007 03:32:31 +0000 (19:32 -0800)]
Make it also work from the command-line.

16 years agoAdded the application control files.
Cameron Dale [Sun, 16 Dec 2007 03:27:41 +0000 (19:27 -0800)]
Added the application control files.

16 years agoAdd proper version numbers to the user-agent string.
Cameron Dale [Sun, 16 Dec 2007 03:24:01 +0000 (19:24 -0800)]
Add proper version numbers to the user-agent string.

16 years agoAdded a manager to the HTTPServer for it to call when a file is not found.
Cameron Dale [Sun, 16 Dec 2007 03:12:02 +0000 (19:12 -0800)]
Added a manager to the HTTPServer for it to call when a file is not found.

16 years agoMoved the HTTPDownloader to the new PeerManager.
Cameron Dale [Sat, 15 Dec 2007 22:30:37 +0000 (14:30 -0800)]
Moved the HTTPDownloader to the new PeerManager.

16 years agoAdded a mirror manager frontend for the AptPackages.
Cameron Dale [Sat, 15 Dec 2007 03:58:20 +0000 (19:58 -0800)]
Added a mirror manager frontend for the AptPackages.

16 years agoClean up the imports again.
Cameron Dale [Sat, 15 Dec 2007 02:43:09 +0000 (18:43 -0800)]
Clean up the imports again.

16 years agoAptPackages only takes a single cache directory.
Cameron Dale [Sat, 15 Dec 2007 02:37:19 +0000 (18:37 -0800)]
AptPackages only takes a single cache directory.

Also changed the path specific to use os.path.

16 years agoMoved the index file tracking inside the AptPackages _load method.
Cameron Dale [Sat, 15 Dec 2007 00:02:42 +0000 (16:02 -0800)]
Moved the index file tracking inside the AptPackages _load method.

16 years agoCleanup AptPackages code.
Cameron Dale [Fri, 14 Dec 2007 23:53:54 +0000 (15:53 -0800)]
Cleanup AptPackages code.

Cleanup and order the imports.
Remove unnnecessary parts.
Add comments and docstrings.

16 years agoImplemented cache loading with threads and deferreds.
Cameron Dale [Fri, 14 Dec 2007 23:26:06 +0000 (15:26 -0800)]
Implemented cache loading with threads and deferreds.

Fixed the stdout redirection to work with threads.
Added individual findHash tests and one large multipleFindHash test.

16 years agoAdd a findHash function to AptPackages that searches everywhere for a hash.
Cameron Dale [Fri, 14 Dec 2007 07:56:39 +0000 (23:56 -0800)]
Add a findHash function to AptPackages that searches everywhere for a hash.

Also added a test for it.

16 years agoAdd tracking of index file hashes from Release files.
Cameron Dale [Fri, 14 Dec 2007 06:43:44 +0000 (22:43 -0800)]
Add tracking of index file hashes from Release files.

Also added test for this.

16 years agoAdded support for source package hashes to AptPackages.
Cameron Dale [Fri, 14 Dec 2007 02:57:23 +0000 (18:57 -0800)]
Added support for source package hashes to AptPackages.

Also added a test for it.

16 years agoFix the architecture problem in the AptPackages config system.
Cameron Dale [Fri, 14 Dec 2007 02:19:17 +0000 (18:19 -0800)]
Fix the architecture problem in the AptPackages config system.

16 years agoAdded a unittest to make sure AptPackages gets the right SHA1 hash.
Cameron Dale [Fri, 14 Dec 2007 01:45:33 +0000 (17:45 -0800)]
Added a unittest to make sure AptPackages gets the right SHA1 hash.

16 years agoRemove the apt-proxy specific stuff from AptPackages.
Cameron Dale [Fri, 14 Dec 2007 00:59:16 +0000 (16:59 -0800)]
Remove the apt-proxy specific stuff from AptPackages.

16 years agoSwitched AptPackages to use twisted's logging facility
Cameron Dale [Thu, 13 Dec 2007 23:27:04 +0000 (15:27 -0800)]
Switched AptPackages to use twisted's logging facility

16 years agoRemove some apt-proxy specific stuff.
Cameron Dale [Thu, 13 Dec 2007 08:16:36 +0000 (00:16 -0800)]
Remove some apt-proxy specific stuff.

16 years agoAdded the packages.py file (as AptPackages.py) from apt-proxy.
Cameron Dale [Thu, 13 Dec 2007 02:50:26 +0000 (18:50 -0800)]
Added the packages.py file (as AptPackages.py) from apt-proxy.

16 years agoAdded new HTTPServer that serves static files.
Cameron Dale [Thu, 13 Dec 2007 02:21:29 +0000 (18:21 -0800)]
Added new HTTPServer that serves static files.

16 years agoUpdate the downloader to set the headers itself and add a getRange function.
Cameron Dale [Wed, 12 Dec 2007 22:03:13 +0000 (14:03 -0800)]
Update the downloader to set the headers itself and add a getRange function.

16 years agoRead the response in the unit tests to prevent RST packets.
Cameron Dale [Wed, 12 Dec 2007 21:31:57 +0000 (13:31 -0800)]
Read the response in the unit tests to prevent RST packets.

16 years agoAdd a manager for the client downloads and the tests for it.
Cameron Dale [Wed, 12 Dec 2007 07:08:43 +0000 (23:08 -0800)]
Add a manager for the client downloads and the tests for it.

16 years agoGet it working and add trial unit testing.
Cameron Dale [Wed, 12 Dec 2007 06:16:15 +0000 (22:16 -0800)]
Get it working and add trial unit testing.

Lots of changes that make it work now.

Also added twisted-style unit tests to be run with
the trial program.

16 years agoAdd a unittest to the HTTPDownloader.
Cameron Dale [Tue, 11 Dec 2007 07:27:18 +0000 (23:27 -0800)]
Add a unittest to the HTTPDownloader.

16 years agoFirst attempt (untested) at new HTTP downloader.
Cameron Dale [Mon, 10 Dec 2007 07:20:59 +0000 (23:20 -0800)]
First attempt (untested) at new HTTP downloader.

16 years agoFixed a bug in test that called hash instead of khash.
Cameron Dale [Mon, 10 Dec 2007 06:06:58 +0000 (22:06 -0800)]
Fixed a bug in test that called hash instead of khash.

16 years agoFix a callLater bug introduced in commit 6075497a.
Cameron Dale [Sun, 9 Dec 2007 20:39:47 +0000 (12:39 -0800)]
Fix a callLater bug introduced in commit 6075497a.

switched to callLater from callFromThread but forgot to
include the delay of 0.

16 years agoClean up all the imports.
Cameron Dale [Sun, 9 Dec 2007 04:52:45 +0000 (20:52 -0800)]
Clean up all the imports.

16 years agoSwitch from the entropy definition to using os.urandom.
Cameron Dale [Sun, 9 Dec 2007 04:07:57 +0000 (20:07 -0800)]
Switch from the entropy definition to using os.urandom.

16 years agoFix some errors found by PyLint.
Cameron Dale [Sun, 9 Dec 2007 04:00:03 +0000 (20:00 -0800)]
Fix some errors found by PyLint.

16 years agoRemove the obsolete whrandom module and use random instead.
Cameron Dale [Sun, 9 Dec 2007 03:18:03 +0000 (19:18 -0800)]
Remove the obsolete whrandom module and use random instead.

16 years agoRemoved all unused threading.
Cameron Dale [Sun, 9 Dec 2007 03:16:02 +0000 (19:16 -0800)]
Removed all unused threading.

16 years agoIgnore the pyc and eclipse project files.
Cameron Dale [Sun, 9 Dec 2007 03:01:22 +0000 (19:01 -0800)]
Ignore the pyc and eclipse project files.

16 years agoRemove all the airhook stuff.
Cameron Dale [Sun, 9 Dec 2007 02:41:48 +0000 (18:41 -0800)]
Remove all the airhook stuff.

20 years agoadded a simple tcp over airhook proxy, not fully tested
burris [Sun, 11 Jul 2004 08:57:54 +0000 (08:57 +0000)]
added a simple tcp over airhook proxy, not fully tested

cleaned up airhook a bit

20 years agobroke up khashmir class into base, read, and write classes, since
burris [Thu, 8 Jul 2004 04:02:59 +0000 (04:02 +0000)]
broke up khashmir class into base, read, and write classes, since
having generic read/write methods on deployed DHTs is not a great idea

actions now take an "action" method, since it's not always
"storeValueForKey"

buckets now get refreshed

nodes that change their ID are now handled properly

20 years agonow if you provide a list to storeValueForKey it will use the krpc
burris [Tue, 15 Jun 2004 16:57:05 +0000 (16:57 +0000)]
now if you provide a list to storeValueForKey it will use the krpc
method "store_values" to store multiple values under the same key in
one swoop