[plug] Fastest way to transfer files over Internet.

[Kirk Turner]

Ahh I remember that lecture :) Actually it was a quote from good ol' 
Tanenbaum.

"Never underestimate the bandwidth of a station wagon full of tapes 
hurtling down the highway"


When I was in Amsterdam a couple of months ago I heard a company talking 
about how they solved such a problem (video distribution between tv 
networks) and using courier's doesn't work so well when you end up with 
3 hour (60km) traffic jams and you have multiple destination... it was 
simply laying private optical cable all over the country...  but I think 
that is probably out of most people's reach :)

Kirk


William Kenworthy wrote:
> If you start now, you should have them zipped and mostly burnt to CD by
> the time I arrive to pick them up ...
> 
> Seriously, as one of my lecturers said, its hard to beat a station wagon
> loaded with CD's or backup tapes for bandwidth across town.  I think you
> will have to be a bit more specific (how far, bandwidth (BB or modem),
> protocol(s) available and so on.
> 
> BillK
> 
> On Sun, 2003-05-11 at 20:33, Scott Middleton wrote:
> 
>>What would you use?
>>1 to transfer 1x600MB file.
>>2 to transfer 600x1MB files.
>>3 to transfer 1200x512k files.
>>
>>>From 1 Linux box to another.
>>
>>On Sat, 2003-05-10 at 09:04, James Devenish wrote:
>>
>>>In message <1052526265.29102.13.camel at virgo>
>>>on Sat, May 10, 2003 at 08:24:25AM +0800, Scott Middleton wrote:
>>>
>>>>Whats the general consensus for Linux users on the fastest way to
>>>
>>>Fast in what sense: Setup time? Connection initiation? Fewest keys to
>>>press? Bulk transfer rate? Number of small files per unit time?
>>>
>>>
>>>>transfer files from one computer to another over the Internet.
>>>
>>>UDP.
>>>
>>>
>>>>There must be some security but it's not really essential.
>>>
>>>Huh? "Must be...but not...essential"! What security are you talking
>>>about? Are you talking about authentication or encryption and
>>>validation?
>>>
>>>(I would say the easiest way would be to stick the file into a web- or
>>>FTP-shared directory and then wget it on the other computer. No time
>>>spent authenticating, no time spent encrypting, works across operating
>>>systems, may include compression by default if your server and client
>>>can handle it, etc. But presumably we can exclude such things in your
>>>situation?)
>>>
>>>
>>>>Preferably a 1 liner.
>>>
>>>So you're talking about transferring just a single file per invocation?
>>>
>>>If you have rcp (a long-standing UNIX rsh facility) set up with
>>>address-based authorisation then you would have minimal authentication
>>>time and no overhead of encryption. If you needed encryption you could
>>>set up IPsec with a fast, loose cipher (obviously that only applies if
>>>you are continually doing transfers between a particular set of
>>>machines).
>>>
>>>If you have SSH set up with public key authentication and no passphrase
>>>on your private key, then yes scp would be very convenient. But of
>>>course it's slow to get started. So it wouldn't fit the general
>>>definition of "fastest" because it would be slow to do many. On that
>>>note, transfers of large numbers of small files may be faster if you
>>>archive the files on the sending machine and dearchive them on the
>>>receiving machine at a later time. That way, the *Internet* part of the
>>>transfer would be faster. If you have several files to copy, but not all
>>>at exactly the same moment, you could open an SFTP connection and send
>>>through files as and when you realise you need them transferred. That
>>>way you only have to initiate the connection once.
>>>

-- 
Kirk Turner <kirk.turner at wagoonline.com>
Software Engineer
WA Go Online Pty. Ltd.
Suite 5, 1 Sarich Way
Bentley WA 6102
Australia

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

mQGiBD6RSIgRBACRm7YZuE3hss6monmfVnGtOmDW2esXRp53AOiU0XAUnEQFtQCA
Iyhw+SwY5sbd94SzUTPveYOKRmLBSdaZqXrh5m1Kxe8f686bf6AKs1cvfOuzRPim
IozAso//ouq3G50PLWjIYTy7JYdz9MYCOG1K3loT3FhZh+2JO04+B71LgwCgzyP3
jWDynm7TrtUoY1XtDfAMkIED/in2XZqhuPU0zfwrpy3JGwk6Blj6+9RwwLCZI/ZV
kSQi37ftycNHxpFHtGdw9rTQqNuO3SHiuwIvtEa20m19QTOcd7OQV3zqAZDHNCim
07htCmVEzSSIVXcjEhLZl+wK5n474wgl9ASy+JnYj7vJEs6GehRTnwPF6pVTGzEX
Z0TpA/9NgzB9sRFDYr/D2fbcO2+0H1KX+x/kxoI2pfIQjt4c94EsQhX6F/5Q4FYA
iAVpcfR9/iSVOeldTT/Z5FlnRx8gIdDnT/dqccUXyahbFeZEr8pyCcj1sNb7Omox
iOoQuk2dK3qG+Qd4RfOpvpwyZoJOKN/QnyD1XSxNp76sSqImPbQ8S2lyayBUdXJu
ZXIgKFNvZnR3YXJlIEVuZ2luZWVyKSA8a2lyay50dXJuZXJAd2Fnb29ubGluZS5j
b20+iFcEExECABcFAj6RSIgFCwcKAwQDFQMCAxYCAQIXgAAKCRDAXmWE4BNAB0VB
AKCZQKI7pKW8aGshkYxIqcie+vILWQCfTyzCBiPPbnCqgdk8WfSZqlwp/LC5AQ0E
PpFIihAEALSMzB6GiCRh36bwg+/+o+N3RBlx2NhxNvRXiFVLNyeUsq19ZwK8sdX+
mrc/2e75Y5iakdOkIu5hQ5qEcp/OBm+RCP0kUgNmwSjydbYYNlgH/j2CNIaxPp1n
OxmtFaPfYN0eP67oUOpfSgViJh4HWgR7421WjPuZ0+lN0Weaoq8fAAMFA/4uJpt2
2AIqRYUAhLn4ag1X6uwwL8Y2zRMJKKE9mHz/nWqpL4QvMceVcGa3OKI6oR9sXONi
9szHpY1htVnieO3vXShVukmMW6eAK1KgnOXihTf++Z2CHgKBj9ueatOA7kxOYwsF
ZmVB4xo5ja3xxdfuj9Ec/GT18HgT17vOPYZGR4hGBBgRAgAGBQI+kUiKAAoJEMBe
ZYTgE0AHf7IAoKl67G9GT6RWJsjpOMeb89Sudb85AJ9Bl4nvDKTIVBYsczg5oZru
ORxHaA==
=XU88
-----END PGP PUBLIC KEY BLOCK-----