--- /dev/null
+%!PS-Adobe-2.0 EPSF-2.0
+%%Title: model_all.fig
+%%Creator: fig2dev Version 3.2 Patchlevel 4
+%%CreationDate: Thu Aug 28 12:20:34 2008
+%%For: camerond@quesnel (Cameron Dale,galat,2008/09)
+%%BoundingBox: 0 0 354 721
+%%Magnification: 1.0000
+%%EndComments
+/$F2psDict 200 dict def
+$F2psDict begin
+$F2psDict /mtrx matrix put
+/col-1 {0 setgray} bind def
+/col0 {0.000 0.000 0.000 srgb} bind def
+/col1 {0.000 0.000 1.000 srgb} bind def
+/col2 {0.000 1.000 0.000 srgb} bind def
+/col3 {0.000 1.000 1.000 srgb} bind def
+/col4 {1.000 0.000 0.000 srgb} bind def
+/col5 {1.000 0.000 1.000 srgb} bind def
+/col6 {1.000 1.000 0.000 srgb} bind def
+/col7 {1.000 1.000 1.000 srgb} bind def
+/col8 {0.000 0.000 0.560 srgb} bind def
+/col9 {0.000 0.000 0.690 srgb} bind def
+/col10 {0.000 0.000 0.820 srgb} bind def
+/col11 {0.530 0.810 1.000 srgb} bind def
+/col12 {0.000 0.560 0.000 srgb} bind def
+/col13 {0.000 0.690 0.000 srgb} bind def
+/col14 {0.000 0.820 0.000 srgb} bind def
+/col15 {0.000 0.560 0.560 srgb} bind def
+/col16 {0.000 0.690 0.690 srgb} bind def
+/col17 {0.000 0.820 0.820 srgb} bind def
+/col18 {0.560 0.000 0.000 srgb} bind def
+/col19 {0.690 0.000 0.000 srgb} bind def
+/col20 {0.820 0.000 0.000 srgb} bind def
+/col21 {0.560 0.000 0.560 srgb} bind def
+/col22 {0.690 0.000 0.690 srgb} bind def
+/col23 {0.820 0.000 0.820 srgb} bind def
+/col24 {0.500 0.190 0.000 srgb} bind def
+/col25 {0.630 0.250 0.000 srgb} bind def
+/col26 {0.750 0.380 0.000 srgb} bind def
+/col27 {1.000 0.500 0.500 srgb} bind def
+/col28 {1.000 0.630 0.630 srgb} bind def
+/col29 {1.000 0.750 0.750 srgb} bind def
+/col30 {1.000 0.880 0.880 srgb} bind def
+/col31 {1.000 0.840 0.000 srgb} bind def
+
+end
+save
+newpath 0 721 moveto 0 0 lineto 354 0 lineto 354 721 lineto closepath clip newpath
+-44.6 718.7 translate
+1 -1 scale
+
+/cp {closepath} bind def
+/ef {eofill} bind def
+/gr {grestore} bind def
+/gs {gsave} bind def
+/sa {save} bind def
+/rs {restore} bind def
+/l {lineto} bind def
+/m {moveto} bind def
+/rm {rmoveto} bind def
+/n {newpath} bind def
+/s {stroke} bind def
+/sh {show} bind def
+/slc {setlinecap} bind def
+/slj {setlinejoin} bind def
+/slw {setlinewidth} bind def
+/srgb {setrgbcolor} bind def
+/rot {rotate} bind def
+/sc {scale} bind def
+/sd {setdash} bind def
+/ff {findfont} bind def
+/sf {setfont} bind def
+/scf {scalefont} bind def
+/sw {stringwidth} bind def
+/tr {translate} bind def
+/tnt {dup dup currentrgbcolor
+ 4 -2 roll dup 1 exch sub 3 -1 roll mul add
+ 4 -2 roll dup 1 exch sub 3 -1 roll mul add
+ 4 -2 roll dup 1 exch sub 3 -1 roll mul add srgb}
+ bind def
+/shd {dup dup currentrgbcolor 4 -2 roll mul 4 -2 roll mul
+ 4 -2 roll mul srgb} bind def
+ /DrawEllipse {
+ /endangle exch def
+ /startangle exch def
+ /yrad exch def
+ /xrad exch def
+ /y exch def
+ /x exch def
+ /savematrix mtrx currentmatrix def
+ x y tr xrad yrad sc 0 0 1 startangle endangle arc
+ closepath
+ savematrix setmatrix
+ } def
+
+/$F2psBegin {$F2psDict begin /$F2psEnteredState save def} def
+/$F2psEnd {$F2psEnteredState restore end} def
+
+$F2psBegin
+10 setmiterlimit
+0 slj 0 slc
+ 0.06299 0.06299 sc
+%
+% Fig objects follow
+%
+%
+% here starts figure with depth 50
+% Ellipse
+7.500 slw
+n 2520 2790 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+2340 2880 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3015 3375 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+2835 3465 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3600 2745 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+3420 2835 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3960 3330 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+3780 3420 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 4500 1260 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+4275 1350 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 4905 2070 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+4680 2160 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 5895 2070 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+5670 2160 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 5535 1305 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+5310 1395 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 2520 6750 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+2340 6840 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3015 7335 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+2835 7425 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3600 6705 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+3420 6795 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3960 7290 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+3780 7380 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 4500 5220 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+4275 5310 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 4905 6030 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+4680 6120 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 5895 6030 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+5670 6120 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 5535 5265 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+5310 5355 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 3046 5362 425 425 0 360 DrawEllipse gs col0 s gr
+
+% Polyline
+n 1080 4815 m 1935 4815 l 1935 5940 l 1080 5940 l
+ cp gs col0 s gr
+% Polyline
+ [60] 0 sd
+n 4140 4545 m 6255 4545 l 6255 6390 l 4140 6390 l
+ cp gs col0 s gr [] 0 sd
+% Polyline
+gs clippath
+2880 4965 m 2970 4965 l 2970 4819 l 2925 4939 l 2880 4819 l cp
+eoclip
+n 2925 4590 m
+ 2925 4950 l gs col0 s gr gr
+
+% arrowhead
+n 2880 4819 m 2925 4939 l 2970 4819 l 2880 4819 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+4247 5264 m 4242 5174 l 4096 5183 l 4219 5221 l 4101 5272 l cp
+eoclip
+n 3465 5265 m
+ 4230 5220 l gs col0 s gr gr
+
+% arrowhead
+n 4101 5272 m 4219 5221 l 4096 5183 l 4101 5272 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+5275 5311 m 5284 5221 l 5138 5208 l 5254 5264 l 5130 5297 l cp
+eoclip
+n 4770 5220 m
+ 5265 5265 l gs col0 s gr gr
+
+% arrowhead
+n 5130 5297 m 5254 5264 l 5138 5208 l 5130 5297 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+5769 5836 m 5852 5801 l 5795 5666 l 5801 5795 l 5712 5702 l cp
+eoclip
+n 5670 5490 m
+ 5805 5805 l gs col0 s gr gr
+
+% arrowhead
+n 5712 5702 m 5801 5795 l 5795 5666 l 5712 5702 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+5160 5985 m 5160 6075 l 5306 6075 l 5186 6030 l 5306 5985 l cp
+eoclip
+n 5625 6030 m
+ 5175 6030 l gs col0 s gr gr
+
+% arrowhead
+n 5306 5985 m 5186 6030 l 5306 6075 l 5306 5985 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+3419 5487 m 3391 5573 l 3531 5617 l 3431 5538 l 3559 5531 l cp
+eoclip
+n 4680 5940 m
+ 3420 5535 l gs col0 s gr gr
+
+% arrowhead
+n 3559 5531 m 3431 5538 l 3531 5617 l 3559 5531 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+n 2535 4050 m 2430 4050 2430 4485 105 arcto 4 {pop} repeat
+ 2430 4590 3585 4590 105 arcto 4 {pop} repeat
+ 3690 4590 3690 4155 105 arcto 4 {pop} repeat
+ 3690 4050 2535 4050 105 arcto 4 {pop} repeat
+ cp gs col0 s gr
+/Helvetica-Bold ff 240.00 scf sf
+1125 5445 m
+gs 1 -1 sc (Server) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+4860 4815 m
+gs 1 -1 sc (DHT) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+945 6210 m
+gs 1 -1 sc 90.0 rot (Package Lookups) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+945 7380 m
+gs 1 -1 sc 90.0 rot (Phase 2:) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+2745 5535 m
+gs 1 -1 sc (Proxy) col0 sh gr
+/Helvetica ff 210.00 scf sf
+2205 4815 m
+gs 1 -1 sc (5 Pkg?) col0 sh gr
+/Helvetica ff 210.00 scf sf
+2745 5265 m
+gs 1 -1 sc (6 Hash) col0 sh gr
+/Helvetica ff 210.00 scf sf
+3465 5130 m
+gs 1 -1 sc (7 Hash?) col0 sh gr
+/Helvetica ff 210.00 scf sf
+3375 5895 m
+gs 1 -1 sc (8 Peers) col0 sh gr
+/Helvetica ff 210.00 scf sf
+4725 5130 m
+gs 1 -1 sc (Hash?) col0 sh gr
+/Helvetica ff 210.00 scf sf
+5040 5715 m
+gs 1 -1 sc (Hash?) col0 sh gr
+/Helvetica ff 210.00 scf sf
+5085 6345 m
+gs 1 -1 sc (Hash?) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+2790 4365 m
+gs 1 -1 sc (User) col0 sh gr
+% Ellipse
+n 2520 10575 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+2340 10665 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3015 11160 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+2835 11250 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3600 10530 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+3420 10620 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 3960 11115 242 242 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+3780 11205 m
+gs 1 -1 sc (Peer) col0 sh gr
+% Ellipse
+n 4500 9045 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+4275 9135 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 4905 9855 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+4680 9945 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 5895 9855 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+5670 9945 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 5535 9090 262 262 0 360 DrawEllipse gs col0 s gr
+
+/Helvetica-Bold ff 180.00 scf sf
+5310 9180 m
+gs 1 -1 sc (Node) col0 sh gr
+% Ellipse
+n 3046 9187 425 425 0 360 DrawEllipse gs col0 s gr
+
+% Polyline
+n 1080 8640 m 1935 8640 l 1935 9765 l 1080 9765 l
+ cp gs col0 s gr
+% Polyline
+n 2535 7875 m 2430 7875 2430 8310 105 arcto 4 {pop} repeat
+ 2430 8415 3585 8415 105 arcto 4 {pop} repeat
+ 3690 8415 3690 7980 105 arcto 4 {pop} repeat
+ 3690 7875 2535 7875 105 arcto 4 {pop} repeat
+ cp gs col0 s gr
+% Polyline
+ [60] 0 sd
+n 4140 8370 m 6255 8370 l 6255 10215 l 4140 10215 l
+ cp gs col0 s gr [] 0 sd
+% Polyline
+gs clippath
+1920 9000 m 1920 9090 l 2066 9090 l 1946 9045 l 2066 9000 l cp
+eoclip
+n 2655 9045 m
+ 1935 9045 l gs col0 s gr gr
+
+% arrowhead
+n 2066 9000 m 1946 9045 l 2066 9090 l 2066 9000 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+2670 9360 m 2670 9270 l 2524 9270 l 2644 9315 l 2524 9360 l cp
+eoclip
+n 1935 9315 m
+ 2655 9315 l gs col0 s gr gr
+
+% arrowhead
+n 2524 9360 m 2644 9315 l 2524 9270 l 2524 9360 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+3195 8400 m 3105 8400 l 3105 8546 l 3150 8426 l 3195 8546 l cp
+eoclip
+n 3150 8775 m
+ 3150 8415 l gs col0 s gr gr
+
+% arrowhead
+n 3195 8546 m 3150 8426 l 3105 8546 l 3195 8546 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+2472 10350 m 2557 10378 l 2604 10239 l 2524 10339 l 2519 10210 l cp
+eoclip
+n 2790 9540 m
+ 2520 10350 l gs col0 s gr gr
+
+% arrowhead
+n 2519 10210 m 2524 10339 l 2604 10239 l 2519 10210 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+2972 9582 m 2885 9558 l 2846 9699 l 2922 9596 l 2933 9723 l cp
+eoclip
+n 2700 10395 m
+ 2925 9585 l gs col0 s gr gr
+
+% arrowhead
+n 2933 9723 m 2922 9596 l 2846 9699 l 2933 9723 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+3610 10337 m 3692 10299 l 3629 10167 l 3640 10295 l 3548 10205 l cp
+eoclip
+n 3285 9540 m
+ 3645 10305 l gs col0 s gr gr
+
+% arrowhead
+n 3548 10205 m 3640 10295 l 3629 10167 l 3548 10205 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+3183 9551 m 3103 9591 l 3168 9722 l 3155 9595 l 3248 9682 l cp
+eoclip
+n 3510 10305 m
+ 3150 9585 l gs col0 s gr gr
+
+% arrowhead
+n 3248 9682 m 3155 9595 l 3168 9722 l 3248 9682 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+4248 9105 m 4243 9015 l 4097 9024 l 4220 9062 l 4102 9113 l cp
+eoclip
+n 3466 9106 m
+ 4231 9061 l gs col0 s gr gr
+
+% arrowhead
+n 4102 9113 m 4220 9062 l 4097 9024 l 4102 9113 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+5276 9165 m 5285 9075 l 5139 9062 l 5255 9118 l 5131 9151 l cp
+eoclip
+n 4771 9074 m
+ 5266 9119 l gs col0 s gr gr
+
+% arrowhead
+n 5131 9151 m 5255 9118 l 5139 9062 l 5131 9151 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+5803 9621 m 5886 9586 l 5829 9451 l 5835 9580 l 5746 9487 l cp
+eoclip
+n 5704 9275 m
+ 5839 9590 l gs col0 s gr gr
+
+% arrowhead
+n 5746 9487 m 5835 9580 l 5829 9451 l 5746 9487 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+5160 9810 m 5160 9900 l 5306 9900 l 5186 9855 l 5306 9810 l cp
+eoclip
+n 5625 9855 m
+ 5175 9855 l gs col0 s gr gr
+
+% arrowhead
+n 5306 9810 m 5186 9855 l 5306 9900 l 5306 9810 l cp gs 0.00 setgray ef gr col0 s
+/Helvetica-Bold ff 240.00 scf sf
+2790 8190 m
+gs 1 -1 sc (User) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+1125 9270 m
+gs 1 -1 sc (Server) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+4860 8640 m
+gs 1 -1 sc (DHT) col0 sh gr
+/Helvetica ff 210.00 scf sf
+3195 8730 m
+gs 1 -1 sc (14 Pkg) col0 sh gr
+/Helvetica ff 210.00 scf sf
+3330 9675 m
+gs 1 -1 sc (11 Pkg?) col0 sh gr
+/Helvetica ff 210.00 scf sf
+2745 10350 m
+gs 1 -1 sc (12 Pkg) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+2745 9360 m
+gs 1 -1 sc (Proxy) col0 sh gr
+/Helvetica ff 210.00 scf sf
+1935 8955 m
+gs 1 -1 sc (11 Pkg?) col0 sh gr
+/Helvetica ff 210.00 scf sf
+1935 9540 m
+gs 1 -1 sc (12 Pkg) col0 sh gr
+/Helvetica ff 210.00 scf sf
+2655 9135 m
+gs 1 -1 sc (13 Hash) col0 sh gr
+/Helvetica ff 210.00 scf sf
+1845 10080 m
+gs 1 -1 sc (11 Pkg?) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+945 10170 m
+gs 1 -1 sc 90.0 rot (Package Downloads) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+945 11250 m
+gs 1 -1 sc 90.0 rot (Phase 3:) col0 sh gr
+/Helvetica ff 210.00 scf sf
+3420 9000 m
+gs 1 -1 sc (15 Store) col0 sh gr
+/Helvetica ff 210.00 scf sf
+4770 9000 m
+gs 1 -1 sc (Store) col0 sh gr
+/Helvetica ff 210.00 scf sf
+5130 10125 m
+gs 1 -1 sc (Store) col0 sh gr
+/Helvetica ff 210.00 scf sf
+5220 9540 m
+gs 1 -1 sc (Store) col0 sh gr
+% Ellipse
+n 3046 1402 425 425 0 360 DrawEllipse gs col0 s gr
+
+% Polyline
+n 1080 855 m 1935 855 l 1935 1980 l 1080 1980 l
+ cp gs col0 s gr
+% Polyline
+n 2535 90 m 2430 90 2430 525 105 arcto 4 {pop} repeat
+ 2430 630 3585 630 105 arcto 4 {pop} repeat
+ 3690 630 3690 195 105 arcto 4 {pop} repeat
+ 3690 90 2535 90 105 arcto 4 {pop} repeat
+ cp gs col0 s gr
+% Polyline
+ [60] 0 sd
+n 4140 585 m 6255 585 l 6255 2430 l 4140 2430 l
+ cp gs col0 s gr [] 0 sd
+% Polyline
+gs clippath
+2880 1005 m 2970 1005 l 2970 859 l 2925 979 l 2880 859 l cp
+eoclip
+n 2925 630 m
+ 2925 990 l gs col0 s gr gr
+
+% arrowhead
+n 2880 859 m 2925 979 l 2970 859 l 2880 859 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+1920 1215 m 1920 1305 l 2066 1305 l 1946 1260 l 2066 1215 l cp
+eoclip
+n 2655 1260 m
+ 1935 1260 l gs col0 s gr gr
+
+% arrowhead
+n 2066 1215 m 1946 1260 l 2066 1305 l 2066 1215 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+2670 1575 m 2670 1485 l 2524 1485 l 2644 1530 l 2524 1575 l cp
+eoclip
+n 1935 1530 m
+ 2655 1530 l gs col0 s gr gr
+
+% arrowhead
+n 2524 1575 m 2644 1530 l 2524 1485 l 2524 1575 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+gs clippath
+3195 615 m 3105 615 l 3105 761 l 3150 641 l 3195 761 l cp
+eoclip
+n 3150 990 m
+ 3150 630 l gs col0 s gr gr
+
+% arrowhead
+n 3195 761 m 3150 641 l 3105 761 l 3195 761 l cp gs 0.00 setgray ef gr col0 s
+% Polyline
+ [15 45] 45 sd
+n 720 3870 m
+ 6300 3870 l gs col0 s gr [] 0 sd
+% Polyline
+ [15 45] 45 sd
+n 720 7695 m
+ 6300 7695 l gs col0 s gr [] 0 sd
+/Helvetica-Bold ff 240.00 scf sf
+2790 405 m
+gs 1 -1 sc (User) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+1125 1485 m
+gs 1 -1 sc (Server) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+4860 855 m
+gs 1 -1 sc (DHT) col0 sh gr
+/Helvetica ff 210.00 scf sf
+2205 855 m
+gs 1 -1 sc (1 File?) col0 sh gr
+/Helvetica ff 210.00 scf sf
+1980 1710 m
+gs 1 -1 sc (3 File) col0 sh gr
+/Helvetica ff 210.00 scf sf
+3195 945 m
+gs 1 -1 sc (4 File) col0 sh gr
+/Helvetica ff 210.00 scf sf
+2025 1170 m
+gs 1 -1 sc (2 File?) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+945 2700 m
+gs 1 -1 sc 90.0 rot (Proxying File Requests) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+945 3735 m
+gs 1 -1 sc 90.0 rot (Phase 1:) col0 sh gr
+/Helvetica-Bold ff 240.00 scf sf
+2745 1575 m
+gs 1 -1 sc (Proxy) col0 sh gr
+% here ends figure;
+$F2psEnd
+rs
+showpage
--- /dev/null
+#FIG 3.2
+Landscape
+Center
+Metric
+A4
+100.00
+Single
+-2
+1200 2
+6 2250 2520 2790 3060
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 2520 2790 242 242 2520 2790 2610 3015
+4 0 0 50 -1 18 12 0.0000 4 135 405 2340 2880 Peer\001
+-6
+6 2745 3105 3285 3645
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3015 3375 242 242 3015 3375 3105 3600
+4 0 0 50 -1 18 12 0.0000 4 135 405 2835 3465 Peer\001
+-6
+6 3330 2475 3870 3015
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3600 2745 242 242 3600 2745 3690 2970
+4 0 0 50 -1 18 12 0.0000 4 135 405 3420 2835 Peer\001
+-6
+6 3690 3060 4230 3600
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3960 3330 242 242 3960 3330 4050 3555
+4 0 0 50 -1 18 12 0.0000 4 135 405 3780 3420 Peer\001
+-6
+6 4230 990 4770 1530
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4500 1260 262 262 4500 1260 4635 1485
+4 0 0 50 -1 18 12 0.0000 4 135 450 4275 1350 Node\001
+-6
+6 4635 1800 5175 2340
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4905 2070 262 262 4905 2070 5040 2295
+4 0 0 50 -1 18 12 0.0000 4 135 450 4680 2160 Node\001
+-6
+6 5625 1800 6165 2340
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 5895 2070 262 262 5895 2070 6030 2295
+4 0 0 50 -1 18 12 0.0000 4 135 450 5670 2160 Node\001
+-6
+6 5265 1035 5805 1575
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 5535 1305 262 262 5535 1305 5670 1530
+4 0 0 50 -1 18 12 0.0000 4 135 450 5310 1395 Node\001
+-6
+6 765 4050 6255 7605
+6 2250 6480 2790 7020
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 2520 6750 242 242 2520 6750 2610 6975
+4 0 0 50 -1 18 12 0.0000 4 135 405 2340 6840 Peer\001
+-6
+6 2745 7065 3285 7605
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3015 7335 242 242 3015 7335 3105 7560
+4 0 0 50 -1 18 12 0.0000 4 135 405 2835 7425 Peer\001
+-6
+6 3330 6435 3870 6975
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3600 6705 242 242 3600 6705 3690 6930
+4 0 0 50 -1 18 12 0.0000 4 135 405 3420 6795 Peer\001
+-6
+6 3690 7020 4230 7560
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3960 7290 242 242 3960 7290 4050 7515
+4 0 0 50 -1 18 12 0.0000 4 135 405 3780 7380 Peer\001
+-6
+6 4230 4950 4770 5490
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4500 5220 262 262 4500 5220 4635 5445
+4 0 0 50 -1 18 12 0.0000 4 135 450 4275 5310 Node\001
+-6
+6 4635 5760 5175 6300
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4905 6030 262 262 4905 6030 5040 6255
+4 0 0 50 -1 18 12 0.0000 4 135 450 4680 6120 Node\001
+-6
+6 5625 5760 6165 6300
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 5895 6030 262 262 5895 6030 6030 6255
+4 0 0 50 -1 18 12 0.0000 4 135 450 5670 6120 Node\001
+-6
+6 5265 4995 5805 5535
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 5535 5265 262 262 5535 5265 5670 5490
+4 0 0 50 -1 18 12 0.0000 4 135 450 5310 5355 Node\001
+-6
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3046 5362 425 425 3046 5362 3406 5587
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 1080 4815 1935 4815 1935 5940 1080 5940 1080 4815
+2 2 1 1 0 7 50 -1 -1 4.000 0 0 -1 0 0 5
+ 4140 4545 6255 4545 6255 6390 4140 6390 4140 4545
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 2925 4590 2925 4950
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 3465 5265 4230 5220
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 4770 5220 5265 5265
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 5670 5490 5805 5805
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 5625 6030 5175 6030
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 4680 5940 3420 5535
+2 4 0 1 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 3690 4590 3690 4050 2430 4050 2430 4590 3690 4590
+4 0 0 50 -1 18 16 0.0000 4 180 750 1125 5445 Server\001
+4 0 0 50 -1 18 16 0.0000 4 180 510 4860 4815 DHT\001
+4 0 0 50 -1 18 16 1.5708 4 240 2070 945 6210 Package Lookups\001
+4 0 0 50 -1 18 16 1.5708 4 180 990 945 7380 Phase 2:\001
+4 0 0 50 -1 18 16 0.0000 4 240 675 2745 5535 Proxy\001
+4 0 0 50 -1 16 14 0.0000 4 195 660 2205 4815 5 Pkg?\001
+4 0 0 50 -1 16 14 0.0000 4 150 675 2745 5265 6 Hash\001
+4 0 0 50 -1 16 14 0.0000 4 150 795 3465 5130 7 Hash?\001
+4 0 0 50 -1 16 14 0.0000 4 150 735 3375 5895 8 Peers\001
+4 0 0 50 -1 16 14 0.0000 4 150 615 4725 5130 Hash?\001
+4 0 0 50 -1 16 14 0.0000 4 150 615 5040 5715 Hash?\001
+4 0 0 50 -1 16 14 0.0000 4 150 615 5085 6345 Hash?\001
+4 0 0 50 -1 18 16 0.0000 4 180 540 2790 4365 User\001
+-6
+6 765 7830 6255 11430
+6 2250 10305 2790 10845
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 2520 10575 242 242 2520 10575 2610 10800
+4 0 0 50 -1 18 12 0.0000 4 135 405 2340 10665 Peer\001
+-6
+6 2745 10890 3285 11430
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3015 11160 242 242 3015 11160 3105 11385
+4 0 0 50 -1 18 12 0.0000 4 135 405 2835 11250 Peer\001
+-6
+6 3330 10260 3870 10800
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3600 10530 242 242 3600 10530 3690 10755
+4 0 0 50 -1 18 12 0.0000 4 135 405 3420 10620 Peer\001
+-6
+6 3690 10845 4230 11385
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3960 11115 242 242 3960 11115 4050 11340
+4 0 0 50 -1 18 12 0.0000 4 135 405 3780 11205 Peer\001
+-6
+6 4230 8775 4770 9315
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4500 9045 262 262 4500 9045 4635 9270
+4 0 0 50 -1 18 12 0.0000 4 135 450 4275 9135 Node\001
+-6
+6 4635 9585 5175 10125
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 4905 9855 262 262 4905 9855 5040 10080
+4 0 0 50 -1 18 12 0.0000 4 135 450 4680 9945 Node\001
+-6
+6 5625 9585 6165 10125
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 5895 9855 262 262 5895 9855 6030 10080
+4 0 0 50 -1 18 12 0.0000 4 135 450 5670 9945 Node\001
+-6
+6 5265 8820 5805 9360
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 5535 9090 262 262 5535 9090 5670 9315
+4 0 0 50 -1 18 12 0.0000 4 135 450 5310 9180 Node\001
+-6
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3046 9187 425 425 3046 9187 3406 9412
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 1080 8640 1935 8640 1935 9765 1080 9765 1080 8640
+2 4 0 1 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 3690 8415 3690 7875 2430 7875 2430 8415 3690 8415
+2 2 1 1 0 7 50 -1 -1 4.000 0 0 -1 0 0 5
+ 4140 8370 6255 8370 6255 10215 4140 10215 4140 8370
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 2655 9045 1935 9045
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 1935 9315 2655 9315
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 3150 8775 3150 8415
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 2790 9540 2520 10350
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 2700 10395 2925 9585
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 3285 9540 3645 10305
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 3510 10305 3150 9585
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 3466 9106 4231 9061
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 4771 9074 5266 9119
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 5704 9275 5839 9590
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 5625 9855 5175 9855
+4 0 0 50 -1 18 16 0.0000 4 180 540 2790 8190 User\001
+4 0 0 50 -1 18 16 0.0000 4 180 750 1125 9270 Server\001
+4 0 0 50 -1 18 16 0.0000 4 180 510 4860 8640 DHT\001
+4 0 0 50 -1 16 14 0.0000 4 195 660 3195 8730 14 Pkg\001
+4 0 0 50 -1 16 14 0.0000 4 195 780 3330 9675 11 Pkg?\001
+4 0 0 50 -1 16 14 0.0000 4 195 660 2745 10350 12 Pkg\001
+4 0 0 50 -1 18 16 0.0000 4 240 675 2745 9360 Proxy\001
+4 0 0 50 -1 16 14 0.0000 4 195 780 1935 8955 11 Pkg?\001
+4 0 0 50 -1 16 14 0.0000 4 195 660 1935 9540 12 Pkg\001
+4 0 0 50 -1 16 14 0.0000 4 150 795 2655 9135 13 Hash\001
+4 0 0 50 -1 16 14 0.0000 4 195 780 1845 10080 11 Pkg?\001
+4 0 0 50 -1 18 16 1.5708 4 240 2340 945 10170 Package Downloads\001
+4 0 0 50 -1 18 16 1.5708 4 180 990 945 11250 Phase 3:\001
+4 0 0 50 -1 16 14 0.0000 4 150 810 3420 9000 15 Store\001
+4 0 0 50 -1 16 14 0.0000 4 150 510 4770 9000 Store\001
+4 0 0 50 -1 16 14 0.0000 4 150 510 5130 10125 Store\001
+4 0 0 50 -1 16 14 0.0000 4 150 510 5220 9540 Store\001
+-6
+1 3 0 1 0 7 50 -1 -1 0.000 1 0.0000 3046 1402 425 425 3046 1402 3406 1627
+2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
+ 1080 855 1935 855 1935 1980 1080 1980 1080 855
+2 4 0 1 0 7 50 -1 -1 0.000 0 0 7 0 0 5
+ 3690 630 3690 90 2430 90 2430 630 3690 630
+2 2 1 1 0 7 50 -1 -1 4.000 0 0 -1 0 0 5
+ 4140 585 6255 585 6255 2430 4140 2430 4140 585
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 2925 630 2925 990
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 2655 1260 1935 1260
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 1935 1530 2655 1530
+2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2
+ 1 1 1.00 90.00 120.00
+ 3150 990 3150 630
+2 1 2 1 0 7 50 -1 -1 3.000 0 0 -1 0 0 2
+ 720 3870 6300 3870
+2 1 2 1 0 7 50 -1 -1 3.000 0 0 -1 0 0 2
+ 720 7695 6300 7695
+4 0 0 50 -1 18 16 0.0000 4 180 540 2790 405 User\001
+4 0 0 50 -1 18 16 0.0000 4 180 750 1125 1485 Server\001
+4 0 0 50 -1 18 16 0.0000 4 180 510 4860 855 DHT\001
+4 0 0 50 -1 16 14 0.0000 4 150 645 2205 855 1 File?\001
+4 0 0 50 -1 16 14 0.0000 4 150 525 1980 1710 3 File\001
+4 0 0 50 -1 16 14 0.0000 4 150 525 3195 945 4 File\001
+4 0 0 50 -1 16 14 0.0000 4 150 645 2025 1170 2 File?\001
+4 0 0 50 -1 18 16 1.5708 4 240 2655 945 2700 Proxying File Requests\001
+4 0 0 50 -1 18 16 1.5708 4 180 990 945 3735 Phase 1:\001
+4 0 0 50 -1 18 16 0.0000 4 240 675 2745 1575 Proxy\001
\r
\begin{document}\r
\r
-\title{\texttt{apt-p2p}: A Peer-to-Peer Distributor for Free Software Package Release and Update}\r
+\title{\texttt{apt-p2p}: A Peer-to-Peer Distribution System for Software Package Releases and Updates}\r
\author{\IEEEauthorblockN{Cameron Dale}\r
\IEEEauthorblockA{School of Computing Science\\\r
Simon Fraser University\\\r
requests from users. However, there is an excellent opportunity to\r
use peer-to-peer techniques to reduce the cost of much of this\r
distribution, especially due to the altruistic nature of many of\r
-these users. There are no existing solution suitable for this\r
+these users. There are no existing solutions suitable for this\r
situation, so we present a new technique for satisfying the needs of\r
-this P2P distribution, which is generally applicable to many of\r
+this P2P distribution which is generally applicable to many of\r
these distributors' systems. Our method makes use of a DHT for\r
storing the location of peers, using the cryptographic hash of the\r
-package as a key. To show the simplicity and functionality, we\r
+package as a key. To show the simplicity and functionality of our model, we\r
implement a solution for the distribution of Debian software\r
-packages, including the many DHT customizations needed. Finally, we\r
+packages, including details on the DHT customizations and system optimizations needed. Finally, we\r
analyze our system to determine how it is performing and what effect\r
it is having.\r
\end{abstract}\r
demands. It is based on many previous implementations of successful\r
peer-to-peer protocols, especially distributed hash tables (DHT) and\r
BitTorrent. The model relies on the pre-existence of cryptographic\r
-hashes of the packages, which should uniquely identify it for a\r
+hashes of the packages, which uniquely identify it for a\r
request from other peers. If the peer-to-peer download fails, then\r
the original request to the server is used as a fallback to prevent\r
any dissatisfaction from users. The peer can then share this new\r
distribution system. This implementation will be used by a large\r
number of users, and serves as an example for other free software\r
distributors of the opportunity that can be met with such a system.\r
+Finally, we analyze our currently deployed implementation to\r
+determine how effective it is at meeting our goals, and to see what\r
+effect it is having on the Debian package distribution system.\r
\r
-The rest of this paper is organized as follows. The background and motivation are presented in Section~\ref{situation}, and we analyze BitTorrent's use for this purpose in Section~\ref{bittorrent}. We propose\r
+The rest of this paper is organized as follows. The background and motivation are presented in Section~\ref{situation}, including an analysis of BitTorrent's use for this purpose in Section~\ref{bittorrent}. We propose\r
our solution in Section~\ref{opportunity}. We then detail our sample\r
implementation for Debian-based distributions in Section~\ref{implementation},\r
including an in-depth look at our DHT\r
-customizations in Section~\ref{custom_dht}. Its performance is evaluated in Section~\ref{analysis}. We examine some related work in Section~\ref{related}, and then\r
+customizations in Section~\ref{custom_dht}. The performance of our implementation is evaluated in Section~\ref{analysis}. We examine some related work in Section~\ref{related}, and then\r
Section~\ref{conclusions} concludes the paper and offers some future directions.\r
\r
%%%%%%%%%%%%%%%%%%%%%%%%%%% Section %%%%%%%%%%%%%%%%%%%%%%%%%%%%\r
\r
In the free software society, there are a large number of groups using the Internet to \r
collaboratively develop and release their software. The ever increasing power of\r
-modern programming languages and operating systems has made these software, like commercial software, extremely large and complex, which often\r
-consists of many small units (packages). Together with their popularity among users, \r
-an efficient and reliable management and distribution of these packages over the Internet has become an daunting task. In this section, we offer concrete examples illustrating the \r
+modern programming languages and operating systems has made this software, like commercial software, extremely large and complex, though it is often\r
+distributed in many small units (packages). Together with their popularity among users, \r
+an efficient and reliable management and distribution of these packages over the Internet has become a daunting task. In this section, we offer concrete examples illustrating the \r
unique challenges in this context. \r
\r
-\subsection{Free Software Package Distributor: Examples}\r
+\subsection{Free Software Package Distributors}\r
\label{examples}\r
\r
-\r
Most Linux distributions use a software package management system\r
that fetches packages to be installed from an archive of packages\r
hosted on a network of mirrors. The Debian project, and other\r
files. Cygwin provides many of the\r
standard Unix/Linux tools in a Windows environment, using a\r
package management tool that requests packages from websites. There\r
-are two software distribution systems for Mac OSX, fink and\r
+are two software distribution systems for software that runs on the Macintosh OS, fink and\r
MacPorts, that also retrieve packages in this way.\r
\r
Direct web downloading is also a common way, often coupled with a hash\r
verification file to be downloaded next to the desired\r
-file. The hash file usually have the same file name, but with an\r
+file. The hash file usually has the same file name, but with an\r
added extension identifying the hash used (e.g. \texttt{.md5} for\r
the MD5 hash). This type of file downloading and verification is\r
typical of free software hosting facilities that are open to anyone\r
to use, such as SourceForge.\r
\r
\r
-Given the free nature of these software, there are often a number of users \r
-motivated by altruism to want to help out with their distribution. \r
-This is particularly true considering that many of these software are used by\r
+Given the free nature of this software, there are often a number of users \r
+motivated by altruism to want to help out with the distribution.\r
+This is particularly true considering that many of this software is used by\r
groups that are staffed mostly, or sometimes completely, by\r
volunteers. They are thus motivated to contribute their network resources, so as to promote the healthy development \r
-of the volunteer community that released the software. As a matter of fact,\r
-we have see many free mirror sites hosting these software packages for downloading. \r
+of the volunteer community that released the software.\r
We also naturally expect that peer-to-peer distribution can be implementation in \r
this context, which scale well with large user bases and can easily explore the network resources made available by\r
the volunteers. \r
512~KB, some of the packages are hundreds of megabytes. The entire\r
archive consists of 22,298 packages and is approximately 119,000 MB\r
in size. Most of the packages are to be installed in any computer environment, but there are \r
-also OS- or architecture-specific packages, as shown by the adjusted sizes based on popularity of the packages. ((((more words on how the size is adjusted by popularity))))\r
+also OS- or architecture-specific packages.\r
\r
\subsubsection{Package Updates}\r
\r
\begin{figure}\r
\centering\r
\includegraphics[width=\columnwidth]{size-quarter.eps}\r
-\caption{The amount of data in the Debian archive that is updated\r
-each day, broken down by architecture.}\r
+\caption{The amount of data in the 119,000 MB Debian archive that is\r
+updated each day, broken down by architecture.}\r
\label{update_size}\r
\end{figure}\r
\r
For example, Figure~\ref{update_size} shows the amount of data in\r
the Debian archive that was updated each day over a period of 3\r
-months. In every single day, approximately 1.5\% of the 119,000 MB archive is\r
-updated with new versions of packages. Note that this frequency is much higher than\r
-that of most commercial software, mainly because many free software are\r
-developed in a loosely management environment with developers working\r
-asynchronously from worldwide. \r
+months. Every single day, approximately 1.5\% of the 119,000 MB archive is\r
+updated with new versions of packages. This frequency is much higher than\r
+that of most commercial software, mainly because much of free software is\r
+developed in a loosely managed environment of developers working\r
+asynchronously on a worldwide scale.\r
\r
\subsubsection{Limited Interest}\r
\r
by everyone, 80\% of the packages are installed by less than 1\% of\r
users.\r
\r
+\subsubsection{Interactive Users}\r
\r
-%%%%%%%%%%%%%%%%%%%%%%%%%%% Section %%%%%%%%%%%%%%%%%%%%%%%%%%%%\r
+The package management software that downloads packages displays\r
+some kind of indication of speed and completeness for users to\r
+watch. Since previous client-server downloads occurred in a sequential\r
+fashion, the package management software also measures the speed\r
+based on sequential downloading. This requires the peer-to-peer\r
+solution to be reasonably responsive at retrieving packages\r
+sequentially.\r
\r
\subsection{Why BitTorrent Doesn't Work Well}\r
\label{bittorrent}\r
\r
-Recently, many distributors make their software available using\r
-BitTorrent \cite{COHEN03}, in particular, for the distribution of CD\r
-images. This straightforward use however is far ineffective, as it requires the\r
+Many distributors make their software available using\r
+BitTorrent \cite{COHEN03}, in particular for the distribution of CD\r
+images. This straightforward use however can be very ineffective, as it requires the\r
peers to download large numbers of packages that they are not\r
interested in, and prevents them from updating to new packages\r
without downloading another image containing a lot of the same\r
packages they already have. \r
\r
-An alternative is to create torrents tracking individual packages. Unfortunately, we find that this enhancement can be\r
+An alternative is to create torrents tracking smaller groups of packages. Unfortunately, we find that this enhancement can be\r
quite difficult given the unique characteristic of free software packages. \r
-\r
First, there is no obvious way to divide the packages into torrents.\r
Most of the packages are too small, and there are too many packages\r
in the entire archive to create individual torrents for each one.\r
old torrent.\r
\r
Other issues also prevent BitTorrent from being a good solution to\r
-this problem. In particular, BitTorrent's fixed piece sizes (?KB) that disregard file\r
+this problem. In particular, BitTorrent's fixed piece sizes (usually 512 KB) that disregard file\r
boundaries are bigger than many of the packages in the archive. This\r
will waste peers' downloading bandwidth as they will end up\r
downloading parts of other packages just to get the piece that\r
contains the package they do want. \r
-\r
Finally, note that BitTorrent downloads files\r
randomly, which does not work well with the interactive package\r
-management tools expectation of sequential downloads. On the other hand, with altruistic peers, incentives to share (upload) \r
-become a less important issue, and the availability of seeds are not critical, either, as the mirror sites \r
-can serve in that capacity.\r
-\r
+management tools expectation of sequential downloads.\r
\r
+On the other hand, there are aspects of BitTorrent that are no\r
+longer needed in a system such as this one. With altruistic peers\r
+and all files being available without uploading, incentives to share\r
+become a less important issue. Also, the availability of seeds are\r
+not critical either, as the mirror sites serve in that capacity\r
+already.\r
\r
%%%%%%%%%%%%%%%%%%%%%%%%%%% Section %%%%%%%%%%%%%%%%%%%%%%%%%%%%\r
\r
\section{Peer-to-Peer Assisted Distributor: An Overview}\r
\label{opportunity}\r
\r
-The situation described in Section~\ref{situation} presents a clear\r
-opportunity to use some form of peer-to-peer file-sharing protocol\r
-to allow willing users to contribute upload bandwidth. This sparse\r
+We assume that the user is attempting to download packages from a\r
+server, using package management software to find and retrieve the\r
+packages. Further, we assume that the requests from the user to the\r
+server are able to be proxied by our peer-to-peer program, and that\r
+the server is always available and has all of the package files.\r
+Finally, the cryptographic hash of the packages must be available\r
+separately from the package itself, and is usually contained in an\r
+\emph{index} file which also contains all the packages' names,\r
+locations and sizes.\r
+\r
+\begin{figure}\r
+\centering\r
+\includegraphics[width=\columnwidth]{model_simple.eps}\r
+\caption{The different phases of functionality of our peer-to-peer distribution model.}\r
+\label{model}\r
+\end{figure}\r
+\r
+Our model for using P2P to enhance such a system is shown in\r
+Figure~\ref{model}. As shown in Phase~1, our program will act as a\r
+proxy (1,2), downloading (3) and caching all files communicated\r
+between the user and the server (4). It will therefore also have\r
+available the index files containing the cryptographic hashes all\r
+packages. Later, in Phase~2, upon receiving a request from the user\r
+to download a package (5), our program will search the index files\r
+for the package being requested and find its hash (6). This hash can\r
+then be looked up recursively in the DHT (7), which will return a\r
+list of peers that have the package already (8). The package can\r
+then be downloaded from the peers (11,12), it can be verified using\r
+the hash (13), and if valid can be returned to the user (14). The\r
+current nodes location is also added to the DHT for that hash (15),\r
+as it is now a source for others to download from.\r
+\r
+In steps (11,12), the fact that this package is also available to download for free\r
+from a server is very important to our proposed model. If the package hash\r
+can not be found in the DHT, the peer can then fallback to\r
+downloading from the original location (i.e. the network of\r
+mirrors). The mirrors thus, with no modification to their\r
+functionality, serve as seeds for the packages in the peer-to-peer\r
+system. Any packages that have just been updated, or that are very\r
+rare, and so don't have any peers available, can always be found on\r
+the mirror. Once the peer has completed the download from the mirror\r
+and verified the package, it can then add itself to the DHT as the\r
+first peer for the new package, so that future requests for the package\r
+will not need the mirror.\r
+\r
+This sparse\r
interest in a large number of packages undergoing constant updating\r
is well suited to the functionality provided by a Distributed Hash\r
Table (DHT). DHTs require unique keys to store and retrieve strings\r
peer will add its entry to the DHT indicating that it now has the\r
package.\r
\r
-The fact that this package is also available to download for free\r
-from a server is very important to our proposal. If the package hash\r
-can not be found in the DHT, the peer can then fallback to\r
-downloading from the original location (i.e. the network of\r
-mirrors). The mirrors thus, with no modification to their\r
-functionality, serve as seeds for the packages in the peer-to-peer\r
-system. Any packages that have just been updated, or that are very\r
-rare, and so don't have any peers available can always be found on\r
-the mirror. Once the peer has completed the download from the mirror\r
-and verified the package, it can then add itself to the DHT as the\r
-first peer for the new package, so that future requests from peers\r
-will not need the mirror.\r
-\r
-The trust of the package is also always guaranteed through the use\r
+Note that, despite downloading the package from untrustworthy peers,\r
+the trust of the package is always guaranteed through the use\r
of the cryptographic hashes. Nothing can be downloaded from a peer\r
until the hash is looked up in the DHT, so a hash must first come\r
-from a trusted source (i.e. a mirror). Most distributors use index\r
+from a trusted source (i.e. the distributors server). Most distributors use index\r
files that contain hashes for a large number of the packages in\r
their archive, and which are also hashed. After retrieving the\r
index's hash from the mirror, the index file can be downloaded from\r
possible to implement the P2P aspect as an almost standard HTTP\r
caching proxy. This proxy will run as a daemon in the background,\r
listening for requests from the package management tool for package\r
-files. It will get uncached requests first from the P2P system, or\r
-falling back to the normal HTTP request from a server should it not\r
+files, as well as serving (uploading) cached package to other peers.\r
+It will get uncached requests first from the P2P system, or\r
+fall back to the normal HTTP request from a server should it not\r
be found. For methods that don't use HTTP requests, other types of\r
proxies may also be possible.\r
\r
addition to storing the file download location in the DHT (which\r
would still be used for small files), a peer will store a\r
\emph{torrent string} containing the peer's hashes of the pieces of\r
-the larger files. These piece hashes could be compared ahead of time\r
+the larger files. These piece hashes are compared ahead of time\r
to determine which peers have the same piece hashes (they all\r
should), and then used during the download to verify the pieces of\r
the downloaded package.\r
We have created a sample implementation that functions as described\r
in section \ref{opportunity}, and is freely available for other\r
distributors to download and modify \cite{apt-p2p}. This software,\r
-called \texttt{apt-p2p}, interacts with the \texttt{apt} tool, which\r
+called \texttt{apt-p2p}, interacts with the \texttt{apt} tool which\r
is found in most Debian-based Linux distributions. \texttt{apt} uses\r
SHA1 hashes to verify most downloaded files, including the large\r
index files that contain the hashes of the individual packages. We\r
available for analyzing the popularity of the software packages\r
\cite{popcon}.\r
\r
-Since all requests from apt are in the form of HTTP downloads from a\r
+Since all requests from \texttt{apt} are in the form of HTTP downloads from a\r
server, the implementation takes the form of a caching HTTP proxy.\r
Making a standard \texttt{apt} implementation use the proxy is then\r
as simple as prepending the proxy location and port to the front of\r
the mirror name in \texttt{apt}'s configuration file (i.e.\r
-``localhost:9977/mirrorname.debian.org/\ldots'').\r
+``http://localhost:9977/mirrorname.debian.org/\ldots'').\r
\r
We created a customized DHT based on Khashmir \cite{khashmir}, which\r
is an implementation of Kademlia \cite{kademlia} using methods\r
downloads from other peers. All peers support HTTP/1.1, both in the\r
server and the client, which allows for pipelining of multiple\r
requests to a peer, and the requesting of smaller pieces of a large\r
-file using the Range request header.\r
+file using the HTTP Range request header.\r
\r
%%%%%%%%%%%%%%%%%%%%%%%%%%% Section %%%%%%%%%%%%%%%%%%%%%%%%%%%%\r
\r
-\section{Customized DHT}\r
+\section{Customized DHT / System Optimization}\r
\label{custom_dht}\r
\r
A large contribution of our work is in the customization and use of\r
incorporated some improvements from BitTorrent's tracker-less DHT\r
implementation.\r
\r
-\subsection{Kademlia Background}\r
-\label{kademlia}\r
+\subsection{DHT Background}\r
+\label{dht}\r
+\r
+DHT's operate by storing (\emph{key}, \emph{value}) pairs in a\r
+distributed fashion such that no node will, on average, store more\r
+than any other node. They support two primitive operations:\r
+\emph{put}, which takes a key and a value and stores it in the DHT;\r
+and \emph{get}, which takes a key and returns a value (or values)\r
+that was previously stored with that key. These operations are\r
+recursive, as each node does not know about all the other nodes in a\r
+DHT, and so must recursively search for the correct node to put to\r
+or get from.\r
\r
The Kademlia DHT, like most other DHTs, assigns IDs to peers from\r
the same space that is used for keys. The peers with IDs closest to\r
close to a key. The queried nodes will return a list of the nodes\r
they know about that are closest to the key, allowing the querying\r
node to quickly traverse the DHT to find the nodes close to the\r
-desired key. The only difference between them is that the\r
+desired key. The only difference between \texttt{find\_node} and \texttt{find\_value} is that the\r
\texttt{find\_value} query will cause a node to return a value, if\r
it has one for that key, instead of a list of nodes.\r
\r
downloads, i.e. HTTP), will cause the peer to return the torrent\r
string.\r
\r
-Figure \ref{size_CDF} shows the package size of the 22,298 packages\r
+Figure \ref{size_CDF} shows the size of the 22,298 packages\r
available in Debian in January 2008. We can see that most of the\r
packages are quite small, and so most will therefore not require\r
piece hash information to download. We have chosen a piece\r
\end{figure}\r
\r
To test our changes during development, we ran the customized DHT\r
-for several hours after each major change on 300 PlanetLab nodes\r
+for several hours after each major change on over 300 PlanetLab nodes\r
\cite{planetlab}. Though the nodes are not expected to be firewalled\r
or NATted, some can be quite overloaded and so consistently fail to\r
respond within a timeout period, similar to NATted peers. The\r
\subsection{Multiple Values}\r
\label{multiple_values}\r
\r
-The original design of Kademlia specified that each keywould have\r
+The original design of Kademlia specified that each key would have\r
only a single value associated with it. The RPC to find this value\r
was called \texttt{find\_value} and worked similarly to\r
\texttt{find\_node}, iteratively finding nodes with ID's closer to\r
While this works well for single values, it can cause a problem when\r
there are multiple values. If the responding node is no longer one\r
of the closest to the key being searched for, then the values it is\r
-returning will probably be the staler ones in the system, and it\r
+returning will probably be the staler ones in the system, as it\r
will not have the latest stored values. However, the search for\r
-closer nodes will stop here, as the queried node only returned the\r
+closer nodes will stop here, as the queried node only returned\r
values and not a list of nodes to recursively query. We could have\r
the request return both the values and the list of nodes, but that\r
would severely limit the size and number of the values that could be\r
-returned.\r
+returned in a single UDP packet.\r
\r
Instead, we have broken up the original \texttt{find\_value}\r
operation into two parts. The new \texttt{find\_value} request\r
it could choose to only request values from the nodes that are\r
closest to the key being searched for.\r
\r
-\subsection{BitTorrent's Improvements}\r
+\subsection{\textbf{OPTIONAL}: BitTorrent's Improvements}\r
\label{bittorrent_dht}\r
\r
-In the many years that some BitTorrent clients have been using a\r
-Kademlia based DHT for tracker-less operation, the developers have\r
+In the several years that some BitTorrent clients have been using a\r
+Kademlia-based DHT for tracker-less operation, the developers have\r
made many enhancements which we can take advantage of. One of the\r
most important is a security feature added to stop malicious nodes\r
from subscribing other nodes as downloaders. When a node issues a\r
The value is supposed to be large enough such that any given\r
\emph{k} nodes are unlikely to fail within an hour of each other,\r
which is very unlikely in our system given the long uptimes of\r
-nodes. We also increased the number of concurrent outstanding\r
+nodes (see Figure~\ref{duration_online_1} in Section~\ref{analysis}.\r
+We also increased the number of concurrent outstanding\r
requests allowed from 3 to 6 to speed up the recursive key finding\r
processes.\r
\r
-\subsection{Other Changes}\r
+\subsection{\textbf{OPTIONAL}: Other Changes}\r
\label{other_changes}\r
\r
We added one other new RPC request that nodes can make:\r
The peer population is very steady, with just over 50 regular users\r
participating in the DHT at any time. We also note that we find 100\r
users who connect regularly (weekly), and we have found 186 unique\r
-users in the 2 months of our analysis. We determined which users are\r
+users in the 2 months of our analysis.\r
+\r
+We also determined which users are\r
behind a firewall or NAT, which is one of the main problems of\r
implementing a peer-to-peer network. These peers will be\r
unresponsive to DHT requests from peers they have not contacted\r
recently, which will cause the peer to wait for a timeout to occur\r
-(currently set at 9 seconds) before moving on. They will also be\r
+(currently 9 seconds) before moving on. They will also be\r
unable to contribute any upload bandwidth to other peers, as all\r
requests for packages from them will also timeout. From\r
Figure~\ref{walker_peers}, we see that approximately half of all\r
long a connection from a peer can be expected to last. Due to our\r
software being installed as a daemon that is started by default\r
every time their computer boots up, peers are expected to stay for a\r
-long period in the system. 50\% of connections last longer than 5\r
+long period in the system.\r
+Indeed, we find that 50\% of connections last longer than 5\r
hours, and 20\% last longer than 10 hours. These connections are\r
much longer than those reported by Saroiu et. al. \cite{saroiu2001}\r
for other P2P systems, which had 50\% of Napster and Gnutella\r
\begin{figure}\r
\centering\r
\includegraphics[width=\columnwidth]{AptP2PDuration-ind_peers.eps}\r
-\caption{The CDF of the average time individual peers stay in the\r
+\caption{\textbf{OPTIONAL}: The CDF of the average time individual peers stay in the\r
system.}\r
\label{duration_ind_peers}\r
\end{figure}\r
\r
-We also examined the average time each individual peer spends in the\r
+\textbf{OPTIONAL}: We also examined the average time each individual peer spends in the\r
system. Figure~\ref{duration_peers} shows the cumulative\r
distribution of how long each individual peer remains in the system.\r
Here we see that 50\% of peers have average stays in the system\r
Since our DHT is based on Kademlia, which was designed based on the\r
probability that a node will remain up another hour, we also\r
analyzed our system for this parameter.\r
-Figure~\ref{duration_online_1} shows the fraction of peers will\r
+Figure~\ref{duration_online_1} shows the fraction of peers that will\r
remain online for another hour, as a function of how long they have\r
been online so far. Maymounkov and Mazieres found that the longer a\r
node has been online, the higher the probability that it will stay\r
\begin{figure}\r
\centering\r
\includegraphics[width=\columnwidth]{AptP2PDuration-online_6.eps}\r
-\caption{The fraction of peers that, given their current duration in\r
+\caption{\textbf{OPTIONAL}: The fraction of peers that, given their current duration in\r
the system, will stay online for another 6 hours.}\r
\label{duration_online_6}\r
\end{figure}\r
\r
-Since our peers are much longer-lived than other P2P systems, we\r
+\textbf{OPTIONAL}: Since our peers are much longer-lived than other P2P systems, we\r
also looked at the fraction of peers that stay online for another 6\r
hours. Figure~\ref{duration_online_6} shows that over 60\% of peers\r
that are online for 10 hours will stay online for another 6.\r
Our walker can extract this information if the peer is not\r
firewalled or NATted, it has not disabled this functionality, and if\r
it uses the same port for both its DHT (UDP) requests and download\r
-(TCP) requests (which is also a configuration parameter).\r
+(TCP) requests (which is also the default configuration parameter).\r
\r
\begin{figure}\r
\centering\r
\begin{figure}\r
\centering\r
\includegraphics[width=\columnwidth]{AptP2PDownloaded-bw.eps}\r
-\caption{The bandwidth of data that the contacted peers have\r
+\caption{\textbf{OPTIONAL}: The bandwidth of data that the contacted peers have\r
downloaded and uploaded.}\r
\label{down_bw}\r
\end{figure}\r
anticipated. We do have plans to improve this through better\r
informing of users of their NATted status, the use of STUN\r
\cite{STUN} to circumvent the NATs, and by better exclusion of\r
-NATted peers from the DHT (which will not prevent them from using\r
+NATted peers from the DHT (which does not prevent them from using\r
the system).\r
\r
We were also concerned that the constant DHT requests and responses,\r
\section{Related Work}\r
\label{related}\r
\r
-There have also been preliminary attempts to implement peer-to-peer distributors for\r
+There have been other preliminary attempts to implement peer-to-peer distributors for\r
software packages. apt-torrent \cite{apttorrent} creates torrents\r
for some of the larger packages available, but it ignores the\r
smaller packages, which are often the most popular. DebTorrent\r
considers all the files available to users to download, and makes\r
use of the existing infrastructure unmodified.\r
\r
-Others have also used DHTs to support this type of functionality.\r
+\textbf{OPTIONAL}: Others have also used DHTs to support this type of functionality.\r
Kenosis \cite{kenosis} is a P2P Remote Procedure Call\r
client also based on the Kademlia DHT, but it is meant as a P2P\r
primitive system on which other tools can be built, and so it has no\r
\r
There are a number of works dedicated to developing a collaborative\r
content distribution network (CDN) using peer-to-peer techniques.\r
-Freedman et. al. developed Coral \cite{coral} using a distrbitued\r
+Freedman et. al. developed Coral \cite{coral} using a distributed\r
\emph{sloppy} hash table to speed request times. Pierre and van\r
Steen developed Globule \cite{globule} which uses typical DNS and\r
HTTP redirection techniques to serve requests from a network of\r
\r
The most similar works to ours are by Shah et. al. \cite{shah08} and\r
Shark by Annapureddy et. al. \cite{shark}.\r
-Shah's system, in addition to the drawbacks mentioned previously,\r
+Shah's system, in addition to the drawbacks mentioned above,\r
is not focused on the interactivity of downloads, as\r
half of all requests were required ``to wait between 8 and 15\r
minutes.'' In contrast, lookups in our system take only seconds to\r
system to update files and propagate those changes to others. Our\r
system is well-tailored to the application of disseminating the\r
unchanging software packages from the authoritative sources to all\r
-users unchanged.\r
+users.\r
\r
\r
%%%%%%%%%%%%%%%%%%%%%%%%%%% Section %%%%%%%%%%%%%%%%%%%%%%%%%%%%\r
may port our existing system to some of the other groups for them to\r
try.\r
\r
-One aspect missing from our model is the removal of old packages\r
+\textbf{OPTIONAL}: One aspect missing from our model is the removal of old packages\r
from the cache. Since our implementation is still relatively young,\r
we have not had to deal with the problems of a growing cache of\r
obsolete packages consuming all of a user's hard drive. We plan to\r
projects / quix0rs-apt-p2p.git / commitdiff