Search this site with

End User Tools and Projects
- NextINet
- Advanced Applications
Database
- DAST Projects/Tools
- Network Performance
and Measurement Tools

End User Support
- Getting Started Guide
- Networking Glossary
- Other Projects/Organizations
- Funding Opportunities

Documents
- Guides/Tutorials
- Papers/Articles
- Presentations
- Reference Books

WebCT Courses
- Tuning Applications

Events
- NLANR/DAST Training
- NLANR Packets Calendar
- Idesk Travel Schedule

News
- Press Releases
- Alliance Data Link
- I2 Newswire Archives

Reports & Statistics
- Monthly Updates and QSRs
- Abilene "Weather Map"
- Web Server Stats

Auto Tuning Enabled FTP Client And Server, v2.0

Released: November 13, 2001

NLANR applications support  | http://dast.nlanr.net/  | <dast@nlanr.net>

[ Introduction | Download Source | Compiling | Usage | OS | Bugs and Comments | FAQ]

Update: Autobuf v_2.0

By using a newly added compile time option (AUTO_UDP), the client no longer needs to be compiled as 'root' (setuid root), which should make system administrators much happier.

We have also compiled a FAQ since the last release which we encourage you to read, especially to see why you would want to use Autobuf instead of other ftp tools available. Autotuning with Autobuf has been shown to improve file transfer speeds (goodput), and is therefore an extremely useful tool for speeding up large file transfers.

Introduction

The Client is compatible with most other FTP Servers.

For the technical background that led to this work, please see our writeup at Automatic TCP Window Tuning and Applications
Please also see our User's Guide to TCP Windowsfor additional information.

To measure optimal TCP/UDP bandwidth using our bandwidth measurment tool, Iperf, please look at Iperf.

Download Source

Download the Auto Tuning Enabled FTP Server (October 20, 2001)

The Auto Tuning Enabled FTP Client and Server are released as a distribution of the C source.

Compiling

Once you have the gzipped tar files, unpack them using gunzip and tar. That will create a new directory 'auto_ncftp' or 'auto_wuftp' (depending on which file you're untarring) with the source files and documentation.

Client Installation steps.

$gunzip -c auto_ncftp_2.0.tar.gz | tar -xvf - 
$cd auto_ncftp 
$./configure 
$make 
$make install [defaults to /usr/local]

$gunzip -c auto_wuftp_2.0.tar.gz | tar -xvf - 
$cd auto_wuftp 
$./configure 
$make 
$make install [defaults to /usr/local]

One problem likely to happen when you try to install, is "permission denied". Since the program opens BSD RAW sockets, which is a privileged action, a user substitution is necessary. If you get "permission denied", you should login as root and compile and install the program.

If you have problems, please report them to dast@nlanr.net.

Usage

The "-O" option enables Auto Tuning.

The client measures the optimal window (socket buffer) size by exchanging a burst of ICMP echo/reply packets with the server, and sets its sending and receiving socket buffer sizes to this optimal value.

It then informs the server of this optimal buffer size. The server responds by setting its own sending and receiving socket buffer sizes to this optimal value. This optimizes file transfer speeds.

When used at the time of opening an FTP session, the "-O" option enables optimal buffer size measurement automatically throughout the session, once in the beginning, and once before every file transfer (get/put) thereafter, without further ado.

For example,

    $ncftp -u  <user> -O <server>

    $ncftp

    NcFTP 3.0.1 (March 27, 2000) by Mike Gleason (ncftp@ncftp.com).

    ncftp> open -u <user> -O <server>

Both of the above ensure that every file transfer during the duration of this ftp session uses only the optimal sending and receiving window sizes calculated just preceeding the file transfer.

Optionally, if you anticipate that you are not going to be transferring a large number of specially huge files, and that the time spent making optimal buffer size measurements would be an overhead best avoided, then you might omit using the "-O" option at the time of opening a session.

Instead, you might then use the "-O" option along with each individual file transfer as shown below.

    $ncftp -u <user> <server>

    ncftp /home/<user> > ls

    file1

    file2

    ncftp /home/<user> > get -O file1

    Optimal Socket Buffer Size Set To [35280] bytes.

    file3:                    15 MB        1.23 MB/s

    ncftp /home/<user> > !ls

    file3

    ncftp /home/<user> > put -O file3

    Optimal Socket Buffer Size Set To [39012] bytes.

    file3:                    18 MB        1.36 MB/s

    .

    .

Apart from automatic tuning, the "set so-bufsize" option allows manual setting of the socket buffer size.

For example, after making an ftp connection..
 

    ncftp /home/<user> > set so-bufsize

    so-bufsize 35280

    ncftp /home/<user> > set so-bufsize 100000

    Optimal Socket Buffer Size Set To [100000] bytes.

    ncftp /home/<user> > set so-bufsize

    so-bufsize 100000

    ncftp /home/<user> >

    .

    .

As shown above, you can also use the "set so-bufsize" without any argument to find out the socket buffer size being used by the client. [Note: the set so-bufsize command will have no effect if there is no current ftp session]

If the server does not support the SITE BUFSIZE <bufsize> feature, then the client lets the user know. The client however sets its own buffer sizes to this new optimal or user-provided value irrespective of the server's capability.
 

    ncftp /home/<user> > get -O file1

    Optimal Socket Buffer Size Set To [40000] bytes.

    Server [<server>] Could Not Set Optimal Buffer Size

    Continuing normally...

    file1:                    6 MB            1.5 MB/s

    ncftp /home/<user> >

    ncftp /home/<user> > set so-bufsize 200000

    Optimal Socket Buffer Size Set To [200000] bytes.

    Server [<server>] Could Not Set Optimal Buffer Size

    Continuing normally...

    ncftp /home/<user> >

If you have problems, please report them to dast@nlanr.net.

OS

Note: The Kernel must support TCP Large windows (beyond 64KB) in order for the ftp application to benefit from Auto Tuning.

Bugs And Comments