Slowing down large ftp transfers

Oct 18 05:52 UTC 1996

  "ftp" is a file transfer protocol used to move files between
computer systems on the Internet.  Some people use it a lot, which
can noticeably slow Grex down.  An idea to reduce the use and impact
of ftp in transferring large files is to modify Grex's ftpd program
to add a delay between data packets during transfer of large files,
with larger files having longer delays.
 
  This would have two benefits:
 
  1) It would discourage users from transferring large files on
     Grex via ftp.  We ask people not to do this in the newuser
     program, but it isn't banned, because sometimes there's a
     compelling reason to transfer a big file.  If people do it
     once in a while, it's no problem, but Grex has had users
     who ftp files upwards of eight hours a day.  If it took
     several times as long to transfer their files, they might
     decide to stop using Grex in this way.
 
  2) It would help reduce the impact of the file transfers on
     Grex's resources.  For example, rather than causing a major
     slowdown for one minute, it might cause a minor slowdown for
     three minutes, spreading out the resource utilization.
  
  On the downside, a determined and knowledgeable user can run
multiple ftp sessions simultaneously, reducing the effectiveness
of the delay, but some users aren't that knowledgeable, and it's
a bit of a hassle in any case.
 
  I've heard this idea mentioned before, I think the first time from
Marcus, but I'm not sure if it's been discussed in co-op.  What do
people think?

response 2 of 31:

Oct 18 07:32 UTC 1996

I think this is a great idea, I'm just wondering what reasons the
nay-sayers will have for complaining about it.  >8)

response 4 of 31:

Oct 18 16:48 UTC 1996

   We've talked about this a little at staff meetings.  I looked at
the code; it wouldn't be that hard to do.  My thoughts were to allow full 
speed for the first 50K, then start sleeping some in a degrading pattern
for more data.  Maybe the threshold shold be greater than 50K--I'm not
sure.  At any rate, those are easy to change.

response 6 of 31:

Oct 18 19:40 UTC 1996

ummm, yes, FULL notice *before* implementation is a wise idea. even int
the motd screen that is occassionally read by some, alwyas read by
others and ignored totally by a few.
  
doesn't setting the   nice  value achieve *soometing* along the lines
suggested?

response 8 of 31:

Oct 19 08:35 UTC 1996

The problem I see with this plan is that a person will give up after a large
amount of data has been transfered.  When you try again, the first portion
of the file is retransmittedm increasing the total link usage.

response 10 of 31:

Oct 20 05:28 UTC 1996

My understanding is that this plan would not have much impact on staff and
members who mostly ftp from grex.  The ftp client on grex - only the server.
This is designed to discourage people from running ftp clients elsewhere to
connect to grex's ftp server.

response 12 of 31:

Oct 21 02:38 UTC 1996

  Out of curiosity, I just ftp'ed to Grex from another system with a
text-based ftp program to see what message is displayed.  Here's part of
what it says when you log in:

 230-Please be aware that Grex has a very tiny connection to the Internet (an
 230-ordinary 28.8 kbps modem).  FTP file transfers slow down Grex for
 230-everybody who is logged in.
 230-
 230-It is OK to use FTP to transfer small files to and from Grex.  However,
 230-PLEASE DON'T TRANSFER LARGE FILES, such as binaries, graphics, jpegs, etc.
 230-Grex has no capacity to display pictures: all our web pages are text-only.
 230-If you have a binary file to transfer from system A to system B, please
 230-send it directly from A to B without a stopover here on Grex.  Thanks.

  This is an area where we could explain that big files are transferred
more slowly, to deter people from starting transfers, then aborting them
partway through.

  However, when you log in with most graphical FTP program, they don't
display this message to the user.  It just presents icons of files and
folders on the local and remote systems.

response 14 of 31:

Oct 22 16:03 UTC 1996

I suppose a determined user could use ftpmail for big things he wasn't willing
to wait around for in real time...

response 16 of 31:

Oct 22 21:04 UTC 1996

(Once in a while I ftp in something along the lines of a new version of Pine
to install, but that type of activity is a small portion of ftp traffic.)

response 18 of 31:

Oct 23 15:27 UTC 1996

Holy cow, yes!  Could you point me toward a list of these users?  Thanks....

response 20 of 31:

Oct 23 17:21 UTC 1996

Yep, though perhaps some of the top users put files on Grex,
then post the login and password to a group of people to grab
the files from.  Or they may be using simultaneous connections,
so they might only be running FTP 3 hours a day, but with four
connections.  I have a hard time imagining one person doing
twelve non-overlapping hours of transfers every day.

I just sent the list of top users to Valerie.

response 22 of 31:

Oct 24 20:56 UTC 1996

  ftpd, the ftp server program on Grex, is supposed to cut off ftp
connections after 15 minutes of inactivity.  Is there evidence that
this isn't happening?  The ftp log could shed some light.

response 24 of 31:

Nov 2 22:03 UTC 1996

Grex decidedly doesn't cut off idle ftp'ers.  Often a "ps -auwwwwxn" will show
ftp processes left from days ago.