Best utilizing fat long pipes and large file transfer

[robert at tellurian.com (Robert Boyle)]

At 06:37 PM 6/12/2008, you wrote:
>I'm looking for input on the best practices for sending large files 
>over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
>I'd like to avoid modifying TCP windows and options on end hosts 
>where possible (I have a lot of them). I've seen products that work 
>as "transfer stations" using "reliable UDP" to get around the 
>windowing problem.
>
>I'm thinking of setting up servers with optimized TCP settings to 
>push big files around data centers but I'm curious to know how 
>others deal with LFN+large transfers.

In our experience, you can't get to line speed with over 20-30ms of 
latency using TCP regardless of how much you tweak it. We transfer 
files across the US with 60-70ms at line speeds with UDP based file 
transfer programs. There are a number of open source projects out 
there designed for this purpose.

-Robert



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin