|
Grex > Coop6 > #35: Mail: the final resource hog | |
|
| Author |
Message |
steve
|
|
Mail: the final resource hog
| 
Nov 11 17:26 UTC 1994 |
Now that Grex is as busy as Grand Central, we need to talk more
about some things that we'd been able to not deal with before.
This item deals with what is Grex's ultimate hog: mail.
Lately, I've gotten the feeling that mail is the #1 resource
consumer, both in terms of CPU and internet link bandwidth. We're
handling more than 2,500 pieces of mail every day now, such that Grex
is getting near to handling a million pieces of mail every year.
Mostly, we're dealing with it. Smail, while a reliable system,
isn't too good at throttling itself back when lots of mail needs
to be dealt with, and spawns seemingly endless copies of itself
at times and really slows things down. Smail needs to be replaced,
and sooner or later we've got to deal with that. But this is a
technical issue, and that isn't the important part.
The important part is how to set social limits on mail
usage. Specifically, how do we deal with people who subscribe
to massive mailing lists, and then don't delete them? The
second question is how to we deal with people who set up
mailing lists, by creating an account with a .forward file
that specifies 50 people?
From the beginning of Grex, we've had people whose eyes
have gotten bigger than their reading abilities, it seems.
Back in the oldren days when netmeg would deliver the mail,
we kinda new that there was a problem when netmeg had to stay
on an extra hour because of someones request for a zillion
files via mail. Not so any more; with our taking mail
automatically, constantly (and endlessly), in a sense its easy
to forget about peoples pigishness. But nowdays, we have
enough people using Grex for mail that we're running into the
problems we had long ago, in terms of running out of space
on /var/spool/mail.
Our second (potential) problem are mailing lists. The
most blatent case was a person heavily involved with the
cypherpunks, who set up an anonymous remailer system here,
and then publicized it on the cyperpunks mailing list without
ever asking or consulting us! When a friend at the UM showed
me that mail, I went over and blasted it. Nothing more ever
came of that.
But now, we're getting these 'little' mailing lists in
various .forward files. Some time ago I went looking for
them and doscovered two of them that had in excess of 40
members each. I renamed the .forward file, and put a
message in the account asking them not to create such lists
without consulting staff first. One of them disregarded my
request, and is in operation again.
So, then, what do we do about this? What is the obligation
of Grex towards mail to the electronic community of the world?
It's funny in an ironic way: we're running the risk of becomming
the postmaster to the world because of being on the internet.
Have a problem holding mail on your country in eastern europe?
No problem: set up an account on a system in the midwestern
United States, and they'll handle it for you. I was told this
by two greatful users in what used to be the Soviet Union.
I would live to set up some (ugh) rules or guidelines (that
latter term makes me feel better) with regard to mail. Having
the numbers of people come through our gates has pretty much
forced us to do this (unless someone has a brilliant idea).
I'm thinking of the following things:
- ask users not to create and use .forward files on Grex unless
there is a specific reason to do so. Usage of a Grex .forward
file for remailing through another system for 'anonomnity' is
specifically frowned upon.
- People who want to create mailing lists should ask staff about
it, such that the size of traffic (not content of the mail) can
be discuessed. People should have the ability to create mailing
lists, but only low volume things that won't hurt all of Grex.
- Ask people to keep their incomming mail to XXX K. (Whatever
XXX is).
- More formally inform people of the existing policy on huge
mail boxes, namely their deletion if truly corpulent and
there is little space on the system, and unsubscribing people
when it appears that they've gone away.
As much as I dislike setting up rules, I think some general
guidelines need to be in place. Probably a screen in newuser
is the only way to disseminate this to all.
|
| 43 responses total. |
scg
|
|
response 1 of 43:
|
Nov 11 22:46 UTC 1994 |
Those look like good guidelines for the most part, but how do we define
"specific reason" for creating a .forward file. "Specific reason" could
include anything from setting up a .forward file to consolodate all of a
person's mail in one place to setting up a 1000 line forward file
specifically for the purpose of jamming our Internet connection.
|
robh
|
|
response 2 of 43:
|
Nov 12 00:28 UTC 1994 |
Well, I've got a .forward file for the very specific reason of
using the filter program, I hope that counts. >8)
Goddess Athena, I remember when I was the biggest mail hog on
the system. Sounds like I've been surpassed.
|
kentn
|
|
response 3 of 43:
|
Nov 12 01:07 UTC 1994 |
If you want to attack this via guidelines and friendly nudges rather than
hard and fast rules, you need to get the word out. You've started that
process here, STeve, but I doubt that the co-op cf is big on the Hit
Parade of conferences, and conferences in general don't seem to be
the major attraction of Grex anymore, anyway. I hate to say it, but
maybe Grex needs to get in the e-mail reminder business. You do need
to keep reminding people...
This "pigishness" is perhaps another problem--people that really
don't care even if they are reminded and are just out to grab out of
all proportion to any idea of fairness. These are the people who bring
about the need for hard and fast rules (and to be fair rules, they end up
being applied to everyone, perhaps for the wrong reasons).
Grex has already (via STeve's examples) shown that there *are* limits
to how pigish people can be. If gentle reminders and calls for civic-
mindedness do not avail, the staff swoop down and zap the offender.
That seems to be about as open a system as we can be without sinking
the lifeboat, so to speak. Are you just worried that your idea of
excessive usage is unfair, STeve? (It seems reasonable to me).
|
steve
|
|
response 4 of 43:
|
Nov 12 02:27 UTC 1994 |
My thoughts on the "specific reason" for .forward files applies to
people using Grex as a mail stop. Rob's local use of a .forward file
doesn't do anything to the link, so thats always fine.
It's the folks using us as a mail stop that is beginning to
worry me, in terms of link bandwidth usage. I'm just now beginning
to get a handle on being able to figure out what type of packets
we're dealing with on Grex--there isn't any really good software for
this, that I know of. But I'm getting the impression that mail is
*really* impacting us.
As far as the PF (Piggishness Factor) goes, I think its a case
of ignorance (hopefully) half the time.
In response to your last question Kent, I'm hoping that what
I'm saying is reasonable, but thats why I'm putting it here: to
see if all the others here think its OK, or if I need another
head-realignment.
|
popcorn
|
|
response 5 of 43:
|
Nov 12 04:48 UTC 1994 |
For added data in the discussion: The mail spool directory has been
gradually creeping toward full, regularly, in the last few weeks.
Generally when it gets past about 90% full I'll make a list of all
the mailboxes that are larger than a megabyte. Then, if I think the
user logs on regularly, I'll move their mailbox to a file called
"inbox", possibly compressed, in their home directory, and I'll send
e-mail saying how to access this mail. If it seems like the user has
gone away for good, I'll do a grep on the From: and To: lines in
the message headers to get an idea of what mailing lists they're
subscribed to. I send a message to root on those systems asking to
unsubscribe the user from their mailing lists. I send a copy of
the message to the user, and (before mailing it) I delete the user's
incoming mailbox.
I feel uncomfortable about manually Doing Things to people's mailboxes.
Re reaching people: I think the idea is to discuss some proposed
guidelines here and come up with something more definite, and *then*
try to reach people with the more definite listing.
|
tsty
|
|
response 6 of 43:
|
Nov 12 05:49 UTC 1994 |
In the last few weeks, and only inthelast few weeks, I've been
getting about 1 chat-help per day for info on a .forward file.
Before I answer any direct questions, I ask why/where/howmany and
that Grex isn't around just for being a maildrop spot in the ether.
In all bu one case, the replies included "yeh, but Grex's mailer is
so slowwwwwwww, I want my mail forwareded to a system where I
can read it faster."
So maybe the situation is snowballing itself.
To allow for the maximum flexibility I would propose that for
perns who want a .forward, it needs to be permed at least 444
and shouldn't/can't have more than 2 addresses in it.
|
srw
|
|
response 7 of 43:
|
Nov 12 07:45 UTC 1994 |
I think it is reasonable to limit the number of lines in a .forward, but
you should only count the lines that point off of grex.
(and I think 1 is a reasonable limit under that method.)
I think it is legitimate to have a .forward if and only if you receive
no list mail to be forwarded. That's hard to catch people at, but it
is a guideline that makes sense.
I don't have a problem with people using .forward as long as their
volume is low. It would be better if they could avoid ending their mail
through our internet link twice, but I'm not ready to ban it altogether.
It is an important service to some who are not abusers.
|
rcurl
|
|
response 8 of 43:
|
Nov 12 08:31 UTC 1994 |
We had been discussing a policy to have a limit on filespace (adjustable
with permission of staff); would it be reasonable to have a limit on total
mail exchanged over the link? Then, there would be no need for a judgement
of the use of .forward or e-mail. Such a limit would automtically
interrupt big mailers, remailers and mail-bombs, without any need for
case-by-case interpretation (except for requested exceptions). (If it can
be done, of course!)
|
popcorn
|
|
response 9 of 43:
|
Nov 12 13:36 UTC 1994 |
Unlike disk quotas, I don't think there's any software to do that.
Re .forward files: we should probably talk about the number of
forwarding addresses in a .forward file, not the number of lines.
You can put several addresses on a single line.
|
jlewis
|
|
response 10 of 43:
|
Nov 12 14:37 UTC 1994 |
I use my mailbox to recieve a magazine file (Amiga Reports) which usually runs
about 100k or so. I understand that some users may abuse a system that has
no hard and fast rules on usage, but one of the things that brought me to GREX
is the no limit on mailbox size (within reason of course). I do try to keep my
directory clear of useless stuff.
|
remmers
|
|
response 11 of 43:
|
Nov 13 12:56 UTC 1994 |
Re limiting the number of addresses or lines in a .forward file: That
wouldn't catch the more knowledgeable and clever offenders. A one-line,
one-entry .forward could forward mail to a program in one's directory
that does anything you want, like forwarding it to a several hundred
recipients. You'd have to go snooping in users' directories to find out
what the thing was doing, and even then it might not be easy to tell.
As a first step, I'd suggest better informing users of what Grex's link
can and can't handle in the way of mail usage. A message in newuser
sounds good.
|
srw
|
|
response 12 of 43:
|
Nov 13 16:53 UTC 1994 |
While I agree that education is valuable, the vast majority of .forward files
contain one or more addresses. I don't think we're looking for a solution
that is foolproof, but rather a set of guidelines for what is acceptable.
|
remmers
|
|
response 13 of 43:
|
Nov 13 21:17 UTC 1994 |
Was #12 a response to #11?
|
rcurl
|
|
response 14 of 43:
|
Nov 13 21:34 UTC 1994 |
Isn't there some place in the system where a user's mail can be "counted",
so that when it exceeds so many bytes, it just gets cleared? A trapdoor
like that would drive excessive users off the system. If it is feasible,
it should of course be made known to newusers, and included in whatever is
the "use guidelines" available to all. (I have asked about this approach
elsewhere, and maybe here too, but keep coming back to it as I have been
given this impression that one can do *anything* with data manipulations
on computers.)
|
steve
|
|
response 15 of 43:
|
Nov 14 03:25 UTC 1994 |
Indeed, there are logs that will help us. So it is the gathering
tools that we don't have, mostly.
|
srw
|
|
response 16 of 43:
|
Nov 14 03:57 UTC 1994 |
Re #13. #12 was intended to be a response to #11.
#11 sounded like it was pooh-poohing the idea of counting off-site addresses
in a .forward file, but I still think it is a reasonable thing to do,
and that it is reasonable to tell people what the limit is.
|
rcurl
|
|
response 17 of 43:
|
Nov 14 07:04 UTC 1994 |
Is a .forward file with 30 addresses used once a month worse than a
.forward file with one address, used daily (except in February)?
|
tsty
|
|
response 18 of 43:
|
Nov 14 10:11 UTC 1994 |
actually, I think, yes it is - for that one day ...
|
rcurl
|
|
response 19 of 43:
|
Nov 14 23:15 UTC 1994 |
In the "big scheme of things"? How many e-mail messages are sent out on
the net per day?
|
steve
|
|
response 20 of 43:
|
Nov 16 09:07 UTC 1994 |
No Rane, a mailing list with 30 names that is used once a month
is hardly a drain on the system. It all boils down to the amount
of traffic generated. In the case of the two 30+ entry mailing lists
I encountered, I know that there was a fair amount of traffic generated,
and, one of them disregarded my request to not use a .forward file,
and set things up a second time anyway. So what I'd like to see is
simply people asking first.
IN talking with people who've accumulated large amount of mail,
I've noticed time and time again that they had no idea what a lot
of mail might be a strain on us. or anyone for that matter. There
are a LOT of people that have no idea at all of the technology
behind their mail, and what that mail might due to a system like
ours.
|
rcurl
|
|
response 21 of 43:
|
Nov 16 14:24 UTC 1994 |
I understand, STeve. What my questions are driving at is that it is the
total consumption of a particular resource that matters, and not
especially how it is used (given its legal, etc). Thus, we intend to apply
a filespace limit, not a limit on this or that kind of files. Likewise, it
would be fairest to set a mail transfer limit, not a limit on how or when.
This is a form of rationing, but from what I gather, it is more of a
"cap", for excessive use that harms the system health.
|
popcorn
|
|
response 22 of 43:
|
Nov 16 14:37 UTC 1994 |
One tough question in this case is that you can control how much file
space you're using, but mail comes from other people and you can't always
control it. If someone got mailbombed, their mailbox filled up, and we
started rejecting their mail, it wouldn't really be their fault, and
we'd be rejecting their legitimate mail along with the mail bombs.
It's like the reason why the caller, and not the callee, pays for
telephone calls.
|
srw
|
|
response 23 of 43:
|
Nov 16 15:55 UTC 1994 |
There are two resources that require shepherding here,
(1) internet bandwidth, and (2) disk space in the mail spool.
We currently have problems with both, and we need limits on both.
Limits on .forward files are not directly addressing either of these,
but are indirectly addressing the internet bandwidth problem,
which is a difficult resource to limit. Rane certainly pointed out
examples of how the .forward approach fails to be a good method
of shepherding that resource.
Some people do things which simultaneously impact both resources,
e.g., if you subcribe to high volume mailing list and don't read it.
Just subscribing impacts the internet, not reading impacts the spool.
It sure would be nice if we could get user-by-user stats on amount
of data transferred in/out, so as to be able to think clearly about
possible ways to control this. Will sendmail provide a better handle on
this than smail?
|
remmers
|
|
response 24 of 43:
|
Nov 16 18:32 UTC 1994 |
STeve proposed a while back, and re-proposed in this item, having a
message in newuser on mail uses that are inappropriate due to their
impact on our resources. We could go ahead and put that in now,
while we're still thinking about what if any limits and guidelines
to impose. Putting a message in the motd for a while might also
be appropriate, to inform current users who don't read this
conference.
If we give educational measures like that a little time to work, they
might give us some clarity on what further measures are needed.
|