|
Grex > Coop7 > #100: Limits of some sort on FTPing large files? | |
|
| Author |
Message |
steve
|
|
Limits of some sort on FTPing large files?
| 
Sep 20 02:54 UTC 1995 |
Recently, I've been seeing an increase of FTP sessions of
large-ish files. Large-ish being 400K and up, considering
our puny net link.
We haven't really talked about it before much, but I'd like
to see a distinct policy about FTPing files to and from Grex,
something like
Please do not use Grex for the storage or exchange of
files. Grex's Internet link is a 28.8Kbpps modem, and
as such, is completely overloaded. Grex cannot reasonably
handle lots of FTP requests for files. FTPing mail,
which can accumulate, is a reasonable thing to move
by FTP but please don't move over 250K a day this way.
If you are within the local calling area of Grex, it
is strongly suggested that you use one of the 9600bps
dialin lines for this purpose.
Something like that. We are already preventing .GIF files
from being used with Lynx, which is a good thing (or I should
say preventing httpd from that).
The sad fact of the matter is that we have more things that
want to go through Grex than can. Grex's lag gets so bad at
times that we have permently lost people over this.
The single biggest "user" of the link is mail, at some huge
percentage which I do not know. We should calculate that some
time. Peoples telent sessions are almost certainly next, and
FTPing is somewhere back farther down. But, when people start
doing lots of FTPs, everyone coming in by the link can feel
it. I know I was at the point of being able to tell that either
a sendmail storm or FTP barrage was happening just by "feeling"
the link speed.
If we had an ISDN link, I'm not sure it would make much of
a difference--I think we're always going to be flooded with
things. I'd like to get an ISDN link however, so we could
test that thought(!).
I'm curious what others think. At some point we have to
do some work to determine what Grex is, or at least the
primary focus is. We are getting to the point that we're
getting bogged down so much that we're losing good people
because of our speed. The CPU situation will be improved
in a while, which will help a little, but we need to be
thinking of how to manage our scarce resources like link
bandwidth.
|
| 13 responses total. |
rcurl
|
|
response 1 of 13:
|
Sep 20 06:19 UTC 1995 |
The fairest thing is to choose and enforce limits. If the 1 M
filespace limit were enforced, there would be far fewer large
files ftp'd. This has come up a number of times, and I always
hear a chorus of simultaneous complaints about heavy users *and*
an antipathy to implementing any limits. Just appealing to people's
good natures goes only so far (and harms the cooperative more
than the uncooperative).
|
steve
|
|
response 2 of 13:
|
Sep 20 13:14 UTC 1995 |
I don't think we need hard and fast "limits", Rane. The fact
that 99% of our users are below 1M of disk suggests that.
There are reasons why someone might need a lot of space for
a short while, and thats OK. Enacting disk quotas will probably
happen, but I'd like to see them high enough that it becomes a
catch mechanism for harmful activities, rather than a "block"
on people.
But disk isn't the issue (disk is the cheapest resource we have
on Grex right now, is gets cheaper every month), but link bandwidth.
What I'd like us have is better wording at newuser, in the faq and
whereever else that says that Grex isn't so much a place for file
storage as things like computer conferencing.
|
rcurl
|
|
response 3 of 13:
|
Sep 20 16:42 UTC 1995 |
If 99% of our users are below 1 M, then why not enact 1 M limits
immediately? It will stop many/big file transfers, and send those that do
that packing. Also, we talked this out before - and *agreed* that all
limits would be negotiable with staff. That takes care of the temporary
need for more space. Its all well and good to have a lot of guff in
faq/newuser about Grex etiquette, but it is easiest to eliminate the
problem without inconveniencing (hardly) anyone, by installing some
limits.
|
gregc
|
|
response 4 of 13:
|
Sep 20 18:04 UTC 1995 |
We had quotas(limits) in place, but then the disk problem reared it's ugly
head, so we began turning things off in an attempt to isolate the problem.
Quotas do odd things with th efile system, so we're leaving them off until
either we find the problem with the Sun-3, or (more likely) we move to the
SUn-4. The plan is to re-enable quotas from day 1 when we move to the Sun-4.
|
steve
|
|
response 5 of 13:
|
Sep 20 21:27 UTC 1995 |
Because Rane, its darned useful to be able to have people do things
that need a large amount of space, for a limited period of time.
Setting quotas up and making the limit high enough that people don't
normally crash into it, is a good idea however, since we've had a
couple of people attempt to write scripts that generated infinate
output to a file. Having a quota would stop that, which is the
most important part.
I say again: Disk is the easy resource on Grex. It is usage
of the link that is the problem. At this point I am much more
concerned with people doing FTPs every day bogging everything
down than I am the usage of Disk. Also, there are several people
who use lots of disk who are local, so they can use the modei to
download with, which is almost "free" in terms of its impact on
Grex. (Note that *uploading* TO grex at 9600 is not a reasonable
thing to do)
|
rcurl
|
|
response 6 of 13:
|
Sep 20 21:33 UTC 1995 |
What you said in the first pagraph is just what I've been saying. Glad
we agree.
We agree on the seond paragraph too. I'm not suggesting the quota to
save diskspace - its to block big file transfers.
|
steve
|
|
response 7 of 13:
|
Sep 21 04:06 UTC 1995 |
Ah, OK; glad to see that we agree. ;-)
The problem is still with the link, and not disk quotas. If
we had quotas of 1M (I believe we were talking about 2M when
we first talked about limits, but that doesn't matter for this
conversation), we'd still have the problem of people moving 500K
GIFs/JPGs/tar files over to and from Grex.
Thats our problem--people thinking of Grex as this free
rest area on the info superhighway.
I'll also point out that those who would want to get around
our disk quotas could just take out account (or another seven)
to hold the files desired.
So, in this case, technical solutions don't quite work the
way we might want them to.
|
gregc
|
|
response 8 of 13:
|
Sep 21 05:09 UTC 1995 |
OTOH, Steve and I have constantly disagreed on this one. My argument is that
if 99+% of the people never go past 1meg, then let's just set it at 1 or 2
meg and be done with it. ANybody that needs extra space can request it any
time they want. If any body needs a "large amount of disk space for a
short amount of time" as Steve argues, then let them use /tmp. That is
what it's there for.
|
srw
|
|
response 9 of 13:
|
Sep 21 12:27 UTC 1995 |
I agree with Greg on this. I think 1 or 2 megs is a reasonable setting for
a quota. I thought we were planning on setting it at 1Meg, and allowing
people to request more.
|
steve
|
|
response 10 of 13:
|
Sep 21 12:58 UTC 1995 |
I argued and wrangled with Greg on this one, and got the limit
to 2M, thus keeping the requests for more space down a little more.
|
popcorn
|
|
response 11 of 13:
|
Sep 21 14:26 UTC 1995 |
Politically, I like the idea of turning on disk quotas.
Technically, I'd like us to talk to the Greater Detroit Freenet before
we do it: they had bad experiences with disk quotas on a Sun (a SPARC 20).
Every time the system writes to disk, the system needs to do a lookup (in a
list of 8000 users) to be sure that it's OK to write to the disk. This
adds a lot of overhead, slowing the system down considerably.
I don't think our overtaxed Sun 3 is ready to do this right now.
|
lilmo
|
|
response 12 of 13:
|
Sep 22 17:04 UTC 1995 |
I'm wondering if FTP really is that much of a drain on the link. Didn't I
hear some time ago that FTP was lowest on the link pecking order, that ftp
went trhough only if there was "room" on the link?
|
ajax
|
|
response 13 of 13:
|
Sep 22 22:07 UTC 1995 |
STeve, it's useful that you noticed this problem, but your proposed
policy sounds an awful lot like the "Limitations of Grex" policy you
already put in newuser. Excerpts from the last time I ran newuser:
> We ARE NOT AN INTERNET PROVIDER. This means we aren't a good place
> to store files, do FTPs from, etc.
>...
> We need your help to keep Grex's Intenet connection working at
> anything close to a reasonable speed. There are three main areas
> of concern--we'd really appreciate it if:
>...
> - You do not store, mail or FTP GIF, JPG or other data files here.
>...
> Because of our slow Internet link, each transfer slows everything
> else down just a little more.
I think adding a longer policy like in #0 to newuser would cause more
people to skip over it. If you recall, I proposed a one-screen alternative
to the two-screen Limitations, so I view adding a third as even worse :-).
However, I also don't think adding a policy that says "please don't..."
will impact the problem much. To change ftp usage, I'd agree with trying
disk quotas. Not a direct solution, but it might help.
Ideally, like if Grex had hundreds of programmers just waiting for new
ideas to implement, it might be neat to automatically send a policy/request
to people who ftp more than 500k in a day or something.
|