|
Grex > Coop8 > #59: Policy on the importing of large files to/from Grex? | |
|
| Author |
Message |
steve
|
|
Policy on the importing of large files to/from Grex?
| 
Apr 24 18:30 UTC 1996 |
I just saw the load average climb to 60 a little bit ago; it
was due to a person attempting to FTP three large files here at
once. After I killed the FTP processes the averages fell back
to about 8.
This isn't the first time I've seen the load avarage shoot
up, because of various problems with getting data into Grex.
The reasons for the problems are technical, but I'm not
interested in that at the moment. I'm more thinking about
policy issues about this right now.
We've never really had a policy about what people can store
here; that was deliberate in the beginning. I'm beginning
to think that we do need to come up with some guidelines about
things people can store here.
Give that we're always goinh to have a limited Internet
link, and that more and more people seem to be using Grex,
I think that we need to have some general policies about
importing and storing files here.
Right now we have about 50M of .gif, .jpg, .wav and .au
files on Grex. Some people have attempted to bring over
really large .au (audio) files--about 18M each. Most are
in the 100 - 200K range, mostly. Having looked at the FTP
logs about every other day, I would estimate that the number
of files being sent is down slightly, but the size of those
files is larger than before. I wish I had kept the data I
gathered about 6 months ago on this, but I didn't.
So, we get back to the "What is Grex For" question. We're
trying to tell people that Grex isn't an ISP, but I don't think
its working, or rather the wording isn't right.
Now, obviously we need to work on the technical side of things
and make it such that FTP doesn't eat as much of the system as
it does now, but the more important question is, should we try
and discourage people from keeeping gifs and other largeish
files here, or keep things the way they are?
More and more people are hearing about Grex it seems. That,
with the fact that fewer people have any real idea about things
like "network bandwidth" make me think that we have a problem
here.
We currently disallow the usage of graphics on web pages here
becuase of the bandwith problem, so we've started a policy on this.
|
| 35 responses total. |
dang
|
|
response 1 of 35:
|
Apr 24 19:27 UTC 1996 |
I say discourage storing picture and sound files here. If people want them,
let them download them to their won computers. That's what I do.
|
kerouac
|
|
response 2 of 35:
|
Apr 24 20:04 UTC 1996 |
coulid simply disallow the exporting of files from mail into a user's
home directory. This would surely cut down on the abuse.
|
adbarr
|
|
response 3 of 35:
|
Apr 24 22:24 UTC 1996 |
STeve -- think about the mail system, and the Unabomber. It is instructive
for the solution to your quandry. A great free system, open to nuts. What
to do? Look at the federal laws about mail. This is not unique.
|
carson
|
|
response 4 of 35:
|
Apr 25 01:15 UTC 1996 |
re #2: I think STeve is referring to the actual FTP process, not
mail-to-FTP gateways. otherwise, you might have the solution. ;)
|
ajax
|
|
response 5 of 35:
|
Apr 25 06:17 UTC 1996 |
I've had a couple "helper" questions about how best to get and store
big files on Grex. I recommend they try nether.net (much better net
connection) or arbornet.org (will have a better net connection Real
Soon Now). How about modify the ftp and mail programs, so when someone
stores a 100k+ file, they display a helpful advertisement of those
systems, and why they'd be better for their purposes? :-) (Just
kidding, mostly.)
|
rcurl
|
|
response 6 of 35:
|
Apr 25 07:47 UTC 1996 |
Are we implementing a max memory allocation for users (yet)? Or a
max transfer limit? Some high but reasonable limits on both of
those would send the transfer/memory hogs away.
|
gregc
|
|
response 7 of 35:
|
Apr 25 09:25 UTC 1996 |
There are a couple of technical solutions we could implement:
1.) We can turn on quotas and limit total storage under a user's home
directory.
2.) We can set a max filesize limit that would prevent a user from
creating a file bigger than, say, 2meg for instance.
3.) We can put additional restrictions on FTP. This is what I would be
in favor of. and it makes the most sense.
Currently, a user can ftp a file to Grex 2 different ways:
1.) He can log onto Grex and execute an "ftp foo.com" command to connect
to the ftp server at foo.com and then do a "get {filename}" command
to transfer {filename} to Grex. Only a member can do this because
outgoing Internet services are limited to members only.
2.) He can log onto another machine somewhere on the Internet and execute
The command "ftp grex.cyberspace.org" and do a "put {filename}" to
transfer {filename} to grex. Unfortunately, *anybody* who has a login
to Grex can do this.
I'd like to change the ftp server so that #2 above is either turned off
completely, or is at least limited to members-only. My reasons are as
follows:
1.) It is a non-orthogonal system, unlike incoming telnet, incoming ftp
doesn't "bring new people to the community". If we only allow certain
users to use it one way, we should also only allow the same subset of
users to use it the other way.
2.) If a user has another system on the Internet that they can attach to
then presumably they can download info directly from that machine to
their home machine. There is no need to use Grex as a stepping stone,
so there is no need to send the file to Grex in the first place.
3.) Having restrictions on both sides of FTP gives us some power over
enforcement of it's use. Currently, only members can use outgoing
ftp. If we put in place some rules that say "No more than XX megs
per day" or "No files over XX megs in size", and then a user refues
to follow those rules, as a last resort we have the ability to revoke
a members ability to use this service. On the other hand, if a
non-member keeps abusing incoming ftp, they can pretty much thumb their
noses at us, becuase they can keep creating new accounts.
4.) I think this would already be set up this way now if it wern't for
technical considerations. The original SunOS ftp daemon wasn't very
bright. It didn't have any way to implement what I'm suggesting. But
we long ago replaced the stock SunOS ftpd with the better one written
by Washington University, the Wu-Ftp daemon. This daemon has much
better security and is *very* configurable. One of the things that
can be configured is an access list of *who* is allowed to ftp into
the system.
We havn't scratched the surface of the capabilities of the Wu-ftp server.
One of the things we can also configure is how many simultaineous incoming
connections are allowed. Users get a message to the effect: "I'm sorry,
there are currently XX ftp sessions, please try again later.". This limit
can be set to different values for different times of the day. I *think*
we can also limit the number of simultaineous connections a given user
has also.
|
gregc
|
|
response 8 of 35:
|
Apr 25 09:47 UTC 1996 |
I just checked the config file. We are currently limiting incoming ftp
sessions to 20. Under the circumstances, I think that's too many. I think
we should change that to 5 from 7:00am to 2:00am and maybe open it up to
10 during the night.
|
steve
|
|
response 9 of 35:
|
Apr 25 17:44 UTC 1996 |
Arnold, I'm not quite sure what you're saying. Can you restate it?
Rane, although disk isn't free, disk space is such that since 98%
of the users here keep only a very little here, I'm not really worried
about more than 2 or three 3 persons disk usage. It's the transfer
via the net that I'm thinking of, here.
I agree with Greg that there is a lot we can do here from the
technical side. At the moment, I'm curious how we come to come
conclusion as to how to set up some kind of limit.
I think that newuser needs to tell people very explicitly that
we aren't good for file storage, etc. The FAQ should contain that
too. Not sure what other documents need to reflect this.
|
arthurp
|
|
response 10 of 35:
|
Apr 25 21:54 UTC 1996 |
I think the limiting of inbound ftp to members is not unreasonable.
|
dang
|
|
response 11 of 35:
|
Apr 26 00:21 UTC 1996 |
I agree with limiting it to members, but I don't think it should go away
entirely. I find it useful on occasion, for getting/dtoring things like text
files, and as my telnet doesn't have z-modem, that's the only way, short of
retyping it in, that I can send files.
|
gregc
|
|
response 12 of 35:
|
Apr 26 00:44 UTC 1996 |
Just to prevent any confusion: I wasn't suggesting that *all* ftp access
be turned off, just the ability to initiate an ftp session on a remote machine
and then connect *to* Grex. You could still log into Grex and run ftp locally
and connect *to* the remote machine.
|
steve
|
|
response 13 of 35:
|
Apr 26 00:52 UTC 1996 |
But the problem with excluding non-members from using FTP is that
we'd stop folks from being able to easily take mail, etc from the
system to elsewhere. Now, granted that I don't like seeing people
take out huge files from Grex, but there are lots of people who FTP
out small files all the time.
I'd like to see this be as simple as policy as we can. Perhaps
a statement asking for no audio/video/graphics files, in places
were all new people can see it is the right thing to do.
|
gregc
|
|
response 14 of 35:
|
Apr 26 01:34 UTC 1996 |
If they have another Internet location, why are they getting mail on Grex
in the first place? And second, mail can always be forwarded off of Grex
using the normal mail mechanisms.
|
dang
|
|
response 15 of 35:
|
Apr 26 04:05 UTC 1996 |
re 12:
I was objecting to turning off FTP initiated from another system. What if
they don't have another mail site? What if they have slip access, but no mail
site, and want to ftp their mail? Last I knew, Grex wasn't running a POP
server...
|
popcorn
|
|
response 16 of 35:
|
Apr 26 04:35 UTC 1996 |
I'd like to see us turn off incoming ftp of big files for non-members.
Outgoing ftp, and *maybe* incoming ftp of small files, would still be
available to everybody. I have no idea if our ftpd could be configured
to do this, though.
|
ajax
|
|
response 17 of 35:
|
Apr 26 04:53 UTC 1996 |
Greg, I'm always wondering "why?" when I hear about convoluted data
movement, too, but while the reasons aren't obvious, people usually have
them, or even if they don't, humans are under no requirement to behave
rationally :-). As a "for instance" to your question, a person might
have an account elsewhere that won't receive mail messages bigger than
32K, so they have mail sent to Grex, then ftp it to the account elsewhere.
STeve, I like the gist of your suggestion, but feel that file size is
a better limit than file content.
Another technical solution might be to add a delay between FTP packets,
so that FTPing becomes much slower. This would cause fewer people to use
it, and when they did use it, it wouldn't be as big a burden to Grex.
(I'm still puzzled how three FTP processes can grind the Sun 4 to a halt).
|
rcurl
|
|
response 18 of 35:
|
Apr 26 07:04 UTC 1996 |
I oppose using membership as a criterion for adding restrictions on use of
the system. We just took steps in the other direction, as a matter of
policy if not philosphy. The fair thing, I think, is to limit file size
for ftp, number of ftp sessions, etc., for everyone. I use ftp by
preference, usually directly (via PPP), for most file transfer, but I have
no reason to transfer very big files.
|
scg
|
|
response 19 of 35:
|
Apr 26 07:10 UTC 1996 |
Right. The impact created by somebody ftping things in or out isn't really
dependant in any way on whether they're a member or not. If we're going to
restrict FTP, we should do it for everybody.
There are legitimate uses for ftping things in and out that don't have
anything to do with mail. Consider, for example, somebody who composes their
conference responses on their own computersj so that they don't have to type
with netlag, and then has to get the response to Grex somehow.
|
tsty
|
|
response 20 of 35:
|
Apr 26 09:24 UTC 1996 |
is there any way to permit only a single ftp process per login at one time?
yes, i realize eventaht could be abused .... tehre will *always* be
abusers somewhere, finding thinngs to abuse, includng Grex.
i'm glad steve killed the simultanwous stuff - whether or not there
was "intent to abuse" or just simple ignrance of the environment, the
result was teh same .... load average climbing to the moon.
Grex does exert control on mail traffic with some sort of load average
sensitivity ... adn that is a good thing, overall, for all users, and
the overall responsivenes for all uses.
can some sort of load-average sensitivity be configured into ftp? And
have it also set within gregc's idea of TOD sensitivity?
As far as storage of huge files is concerned - if space is not a
terrible problem - thre might be a general direction established (i'm
avoiding the "P" word here) taht would keep somewhat track of how
long a huge file sits here - say 12-18 days e.g. - and then polite
inquiries are made as to the disposition of the HugeFile (tm).
ftping a 2 meg file and then d/l the same can take, in total, quite
a chunk out of someone's day. But that may be the only way to
achieve the transfer (short of an email enclosure ...heh-heh). But,
for temporary storage, i think Grex is flexible enough to accomodate
that.
except for the newbies who try to ftp a telnet client (is that still
ahppenning??) and compile it here (or worse, a binary and try to
run it) my swag is that the file transfer is benign but a nuisance
at times.
Slamming the door on everyne doesn't seem like the right thing to do.
|
albaugh
|
|
response 21 of 35:
|
Apr 26 17:53 UTC 1996 |
One advantage that ftp has over mail is that it allows transfer of binary,
compressed files. That cuts down on disk space usage and bandwidth during
transfer. But if things are big, they're big...
|
steve
|
|
response 22 of 35:
|
Apr 26 18:58 UTC 1996 |
Right.
I'd rather see the limits on the kind of files still, rather
than the size. Why? Because its usually the case that the large
files are graphics/audio types of files. There have been a few
people who've moved other types of large files through Grex, but
really pretty small.
I certainly don't want to see us get rid of FTP. Doing that
would be the "easy" way to fix the problem, but that isn't in
the spirit of Grex.
Someone asked about why would someone get mail here, and then
FTP it elsewhere. Well, it seems that more and more ISP's are
offering a cheap PPP/SLIP service, and only provide the "pipe",
sans just about all other services save for DNS. I think we're
going to encounter this type of user more and more in the future.
The other reason I'd like to see the types of files "banned"
as opposed to numeric limits is that then we can avoid coming
up with numbers. If we ask people to simply not stuff graphics
or audio files here, thats easy(er) for people to comprehend.
If we say "only nnn hundred K per day" to people, we're going
to get a *LOT* of people who will significantly go over those
bounds, becuase they have *no clue* about the size of the things
they're moving across the net.
The era of people using the net as an applicance is here.
Shortly (within 3 years) we can expect the percentage of people
who actually know about the technical side of their computers
to roughly match the number of people who know about the inner
workings of their sterio radios.
|
kerouac
|
|
response 23 of 35:
|
Apr 26 19:24 UTC 1996 |
In fact with the popularity of the web, there is a line of new televisions
being produced that will come with computer chips built in. You'll
get a keyboard and a mouse, and be able to netsurf through your tv.
But the tv's wont have drives in them, so on-line file storage will
become the norm. In ten years, probably less, keeping a bulky pc hard
drive in use in one's home will be as archaic as having a rotary dial
phone or a typewriter is today.
I'd think grex would be better off in the long run if e-mail and file
storage are both eventually shifted to a second machine, where there
wouldnt be such competition for resources and thus less of a need for
limiting ftp processes..
|
steve
|
|
response 24 of 35:
|
Apr 26 20:17 UTC 1996 |
Interesting to hear, about the TV. I suppose it was only a
matter of time.
It isn't the disk space that really matters--its how the data
gets to the disk that I'm concerned about.
For example, once we're on the terminal server, it should be
possible to upload/download faster than ever before. Espically
in the case of downloading, if someone wanted to sit there and
transfer a bunch of stuff to their PC from Grex, it wouldn't
take much in the way of system resources. The user has a dedicated
channel (their modem) to use as they see fit, essentially. So
a big download via a dialin won't affect us much (uploading may
still be a problem, I can't speak to that yet).
But network users on the other hand, still share this limited
resource we call the Internet. Even if people only FTP'd little
tiny files (but did so often), we'd have a problem. We have a
pipe of a certain 'data-diameter', and we can't (easily) expand
that much.
When we bought the 2G disk we're currently using, it cost us
$1000, and that was a "deal"; the place we got it from had gotten
some units from somewhere that couldn't sell them so we got it
at a good price. Today, that same $1000 will get us 4G of disk,
which will be a little faster, and possibly significantly cooler
(ie, less power consumption). Thats only going to continue--so
once we've used up this 2G disk, we can get another. We could
get another Micropolis 2G disk from the same source for $400.
|