|
Grex > Oldcoop > #376: The problems with Grex, e-mail and spam | |
|
| Author |
Message |
| 25 new of 480 responses total. |
remmers
|
|
response 175 of 480:
| 
Dec 11 14:35 UTC 2006 |
To recapitulate, and to slightly rephrase a response I entered in the
Agora system problems item: There is currently a system-wide
spam filter running - spamd, the "daemonized" version of SpamAssassin.
It works like this: Operating in conjunction with procmail, it reads
configuration files from the user's home directory (.procmailrc and a
few other things) to decide what to do with incoming mail messages for
that user. In particular, the user can decide how aggressive the
filtering should be. SpamAssassin also has Bayesian filtering features.
Sindi has it exactly right: What I've volunteered to do is write a
simple menu-style interface that makes it easy for a user to set this up
for his or her account.
The problem with this approach - and I can certainly understand Rane's
and others' concern - is that if you own the configuration files that
control how your spam filtering is done, you also own the job of keeping
them up-to-date. The fear that this could be a bothersome task is a
legitimate one. I've been experimenting the last few days with
SpamAssassin-based filtering in my own account, and I must say that my
experience contradicts Sindi's. The same filtering options that caught
over 90% of my spam a few months ago are now catching less than 20% of
it. I woke up this morning with over 100 uncaught spam messages that
had accumulated over the last day. If I'm going to make my filtering
more effective, I'll have to tune my SpamAssassin options, or add stuff
to my .procmailrc, or something. Now, I'm something of a computer geek
and enjoy tinkering, so I might find that a bit fun, even. But I can
certainly understand why people don't want to bother with it.
So that raises the question: What can Grex do on a global level,
independent of any user preferences, to control the spam problem? And
do we have the resources - computing power and personnel - to do it?
Offhand, I don't know the answer to that.
|
cross
|
|
response 176 of 480:
|
Dec 11 16:10 UTC 2006 |
Computing power? Yes. Personnel? No. Culture? No.
|
remmers
|
|
response 177 of 480:
|
Dec 11 16:38 UTC 2006 |
Boy, we really suck, don't we.
|
tod
|
|
response 178 of 480:
|
Dec 11 18:16 UTC 2006 |
I'd venture to say that so long as Cindy is tweaking her script then it
wouldn't be too much scripting to propagate those rulesets globally.
Computing power? I dunno..I think grex email is a bad idea altogether.
|
keesan
|
|
response 179 of 480:
|
Dec 11 18:44 UTC 2006 |
Remmers, have you set spamassassin to three points instead of the default five
points? I also set my .procmailrc to put anything on the spamcop list in a
/spam folder, after running spamd, along with anything with 2 points (which
is sometimes real mail). Some real mail ends up in /spam folder.
My tweaks are aimed at the residual 20% or so. I change the stock spam filter
every couple of days, for instance, and throw out HTML with Windows charsets
or 3D or embedded images (some of which comes from people I know).
|
keesan
|
|
response 180 of 480:
|
Dec 11 18:45 UTC 2006 |
Remmers, can you set up a script that will automatically update .procmailrc
for everyone using spamd, if you do want to keep tweaking? Or just include
that option in a change program (update spam filter)?
|
rcurl
|
|
response 181 of 480:
|
Dec 11 20:11 UTC 2006 |
You still have to check all your mail, spam and all, before deleting, don't
you? Or do you send any of it just to dev/null without checking?
|
keesan
|
|
response 182 of 480:
|
Dec 11 20:14 UTC 2006 |
I send things to /dev/null but keep a non-verbose log which I check every few
days in case my filter threw out something it should not. It is much more
aggressive than spamassassin itself, which has never thrown out anything it
should not, using 3 points. I send 2-point mail to a spam folder, and about
1% of the time that is real mail. Ditto for anything on sorbs or spamcop
lists. My aggressive filter is somewhat tempered by a long whitelist of any
friend whose mail got dumped previously by the filter. But you can just use
spamassassin without a log file, set to 3 points, and probably catch 90%.
|
keesan
|
|
response 183 of 480:
|
Dec 11 20:19 UTC 2006 |
The log file is a list of where the mail came from (The 'From' line, which
spammers often fake), the subject line (spammers tend to use creative lines
like 'Hi' or 'Re: Re: Re:'), and where it went (/var/mail/keesan, or
/dev/null, or /var/mail/keesan/spam). Every few days I check over the
/dev/null lines quickly and then delete the log file, at which time it is
around 50K (pages and pages). If you are curious why things go to /dev/null
you can set the log file to verbose and it will tell you which filter(s) the
mail passed through and where it was caught. I used to do that until the spam
count increased. If you notice a lot of the same spam slipping through for
a few days, you can add your own filter before spamassassin.
|
rcurl
|
|
response 184 of 480:
|
Dec 11 20:19 UTC 2006 |
But you still scan the subject of everything, don't you? That's all I do.
I suppose your system just prevents your inbox from filling - or, does it?
I don't expect a perfect system is possible, but so much spam is so obviously
in *classes* of spam, and getting rid of those immediately would be a great
help. Users that want to could keep tweaking a separate filter for themselves.
|
keesan
|
|
response 185 of 480:
|
Dec 11 20:25 UTC 2006 |
My system sends 90% of spam to /dev/null and most of the rest to a spam
folder. I scan really quickly, just what went to /dev/null and came in less
than 2 copies. Nothing to move or delete or save afterward, it is already
gone. This week I got one false positive, since I filter on anything
that looks like a webpage and the sender designed it to look like a webpage
but will consider next time sending out holiday greetings as a text message
with the link to a website.
|
keesan
|
|
response 186 of 480:
|
Dec 11 20:44 UTC 2006 |
I just revised my simple filter and added lots of explanations. See
procmail.simple for instructions how to set up a spam filter using spamc.
Copy one line to .forward and the rest of the file to .procmailrc after
editing out my name and putting in your own. All your mail will be forwarded
to procmail, which runs your filter, sends any whitelisted mail directly to
the inbox, assigns spam points to the rest, dumps anything with 3 points,
sticks anything with 2 points in a spam folder, and puts the rest in inbox.
It keeps a logfile in the mail directory of where mail came from, subject
line, and where it ended up, which you can read and delete every few days.
You should also read the spam folder and delete it once in a while. Adding
additional filters catches more spam but sometimes also gets real mail.
|
cmcgee
|
|
response 187 of 480:
|
Dec 11 21:20 UTC 2006 |
Actually, there are already some system-wide spam filters in effect. Marcus
designed some, and they return an email to the sending system. To track what
filter rule was applied, Marcus put in a cryptic quotation.
I ran across these a few years ago when I was a list admin for a list on a
different system. Some of the mail from that list bounced back from my Grex
address. As admin, I saw the bounced email and made inquiries, but no one
knew what the cryptic quotations actually referred to.
|
tod
|
|
response 188 of 480:
|
Dec 11 21:24 UTC 2006 |
Its a countdown sequence for the alien invasions.
"Haven't you ever wanted to be part of something special?"
|
spooked
|
|
response 189 of 480:
|
Dec 11 21:41 UTC 2006 |
I very much doubt Marcus' quotations/basic spam analysis have been ported
across from SunOS/Sendmail to our current system featuring procmail.
The DEFAULT way to cut spam for Grex (OR SHOULD BE!) is at our mail server
on Grex, BEFORE it reaches user's mailboxes. I have been reading a
technical book dedicated to combatting spam and believe I have the technical
capability to hugely reduce the current epidemic problem of spam facing
Grex user's currently.
The question remains, and the options simple. Does Grex want to move
forward (in anything other than microsteps)? If that is a yes, than staff
should IMMEDIATELY reinstate Dan and myself to staff.
|
mcnally
|
|
response 190 of 480:
|
Dec 11 23:49 UTC 2006 |
re #189: procmail is a local delivery agent. the program which fills
the sendmail niche in our current configuration is exim.
re #187: that sounds typically Marcus. I'm sure if pressed for an
explanation of why he chose cryptic quotations rather than informative
error messages he'd've given a compelling-sounding explanation about
how readable error messages would have helped the spammers when the
reality is that long after he lost interest in the project other
people were reduced to trying to figure out his cryptic and non-standard
configuration choices.
re #189 (again):
> I have been reading a technical book dedicated to combatting spam
> and believe I have the technical capability to hugely reduce the
> current epidemic problem of spam facing Grex user's currently.
I'm all ears. Perhaps you could write a brief summary to tell us
how you would approach the problem, what resources would be required,
what e-mail would be affected, and what the potential downsides of
your approach would be (in your opinion.) If the approach looks
promising I'm betting that the board will support it, but running it
by the user community for comments would be a great first step.
|
spooked
|
|
response 191 of 480:
|
Dec 12 00:18 UTC 2006 |
Yes, 'exim' is our current MTA (Mail Transfer Agent) component - thus my
strong doubts the customisations to 'sendmail' that Marcus made on our last
machine would have been ported over to this machine with a new MTA.
'procmail' is a Mail Delivery Agent (MDA) component (sitting between the
MTA and user mailbox software, aka Mail User Agent (MUA) component, albeit
a very powerful one. Customisations to 'exim' and server-interface
'procmail' components is the way to go.
I would require staff re-instatement before I committed my services any
further. It appears as though they do not value my input and technical
capabilities, however.
|
mcnally
|
|
response 192 of 480:
|
Dec 12 01:55 UTC 2006 |
I was under the impression that you resigned from staff. Its not
surprising if nobody has reinstated you if no request has been
made and you haven't previously indicated a willingness to rescind
your resignation.
I'm not trying to be difficult, though, but if you were evaluating
someone else's proposal, would the "I have a secret plan to reduce spam.
Make me staff and I'll tell you what it is.." be one that you found
persuasive?
|
spooked
|
|
response 193 of 480:
|
Dec 12 02:11 UTC 2006 |
I have requested to be reinstated to staff about 2 weeks ago...
|
keesan
|
|
response 194 of 480:
|
Dec 12 03:17 UTC 2006 |
Perhaps next time you or Cross get disgruntled you could just take a leave
of absence from staff instead of resigning? I sympathize with your reasons
from resigning but wish you would stay unresigned.
|
cross
|
|
response 195 of 480:
|
Dec 12 04:14 UTC 2006 |
Regarding #177; Well, when you don't bother to listen to anyone outside of
your little inner circle then yes, you really do suck.
Regarding #187; Those changes weren't ported to the current platform.
Regarding #192; I've posted a number of suggestions for improving grex's email
situation. I've offered to help. I haven't heard anything back. *Shrug*
|
cross
|
|
response 196 of 480:
|
Dec 12 04:55 UTC 2006 |
Regarding #194; I think people should protest grex staff's practices. If that
means resigning, then so be it.
|
spooked
|
|
response 197 of 480:
|
Dec 12 04:56 UTC 2006 |
re:194: Leave of absence: Oh yes, like Grex staff when any decision is
required or real work needs to be done. Nah, I'd rather do the work or
knick off.
The solution/s (as there are many) to dramatically reducing spam (and
I'm talking 95%+) are not rocket science (by any means!), and there are
mighty fine people on Grex offering to help deliver those solution/s. I'm
sure significantly less talented people have solved this problem many,
many, many times before --- but have one big advantage, access and
delight from the sysadmins.
If I was a Grex member, I would not be happy. Hell, I am just a Grex user
and I am NOT happy with staff apathy and non-leadership by example here.
|
spooked
|
|
response 198 of 480:
|
Dec 12 05:00 UTC 2006 |
Dan slipped-in... but, funnily enough, his thoughts echoed mine entirely.
|
rcurl
|
|
response 199 of 480:
|
Dec 12 06:44 UTC 2006 |
Isn't it the Board that appoints (and dis-appoints) staff? Have you gone to
a board meeting to make your case?
|