response 218 of 248:

Dec 14 17:19 UTC 2002

Thanks for keeping us informed, Marcus.  What a problem!

response 220 of 248:

Dec 14 22:58 UTC 2002

  It's been a long time since I've dealt with something like this but it
  seems at first glance that a simple package like tcp_wrappers should be
  fine for this, I don't think you need full bpf-style flexibility.

response 222 of 248:

Dec 15 01:05 UTC 2002

(I'd say blocking the range is a first, temporary measure, but that's
a discussion for another conference.)

response 224 of 248:

Dec 15 01:32 UTC 2002

Tcp wrappers can't help - they work at the user level; this problem
isn't even visible at the user level, so there's no way to use it to
reject connections.  The DSL router has a limited capability of
filtering packets, but we don't administer it, so that's not useful to
us.  obsd pf would be quite capable of that, but we're not on openbsd
yet, and an obsd pf firewall of some sort would also work, but that's
quite a bit of work.  It may be worth doing this anyways, but it's not
going to happen this weekend.  My current plan is to try to write
something that can "sniff" the wire, detect when grex tries to complete
a connect to a set of bad IP ranges, and send packets to grex to cause
the connections to be aborted.  I tried to do just this last night, but
I'm afraid lack of sleep was catching up fast with me and I didn't
succeed in getting this to work.  I'm going to try to debug this
separately outside of grex where I can better figure out what's
happening, and hope to have better luck making this work with grex
tomorrow.

response 226 of 248:

Dec 15 05:13 UTC 2002

  No doubt my understanding of the problem is completely deficient but
  nothing said so far illustrates to me why this problem can't be dealt
  with by using tcp wrappers to block these IP addresses from connecting
  to just the SMTP port.

  I'm really not understanding what Marcus is saying in the first sentence
  of 224.  My recollecton concerning TCP wrappers is that the major reason
  to run the package is to allow selective blocking of incoming connections
  on certain ports.  I'll admit it's been a long time, though, maybe I'm
  thinking of another package entirely.

response 228 of 248:

Dec 15 19:01 UTC 2002

I have another report of lost mail.  I sent my sister e-mail on 
Wednesday; she tells me by phone today that she received the mail
promptly and replied, but the reply hasn't shown on up on Grex after
about three days.

response 230 of 248:

Dec 16 01:01 UTC 2002

(Three days is the usual "first notification" limit, so she may have gotten
her message back by now, or at least a warning that it has not yet been
delivered.)

response 232 of 248:

Dec 16 02:32 UTC 2002

My guess is that the bulk mailers were in the "process" of being
disconnected; they still had the physical IP connection and could send
packets, but were using addresses that were no longer "routable", so it
was no longer possible to send packets back to them.

The tcp wrappers come with a library and it's possible to compile other
programs (such as sshd) to use the same basic blocking code.  It
wouldn't be useful for sendmail hwever as there are way too many mail
gateways that, when a connection fails for any reason, immediately retry
without even waiting.  So, in fact, you actually want to accept the
connection, wait until they try to send mail, and *then* deliver a 5xx
error code.  That *should* clear the queue entry out at their end and
avoid the retry.  Grex's sendmail isn't great at doing this, but it does
have support and we use it routinely against people we recognize as
spammers.  Last week, we rejected 1904 pieces of mail using this
facility.  The 5xx error also contains a URL, and 11 people read the
file referenced by the URL.

response 234 of 248:

Dec 16 04:36 UTC 2002

  Ahh..  Ok, now I get it.

response 236 of 248:

Dec 16 20:56 UTC 2002

For whatever it's worth:  I send (from off site) several messages a day to
a couple of people on Grex.  From the system where I send them, I see them
hanging in the queue for long periods.  Usually the error message given is
  Deferred: Connection timed out with grex.cyberspace.org.
but sometimes it is
  Deferred: Interrupted system call

Given what's been said, I assume that it times out because there are too many
of these half-open connections.  But I have to wonder if the "Interrupted
system call" error doesn't occur because I somehow get a half-open connection.
At any rate, I thought I'd mention this message.

response 238 of 248:

Dec 17 06:02 UTC 2002

The half-open connections are typically from several *different* IP
addresses.  So, anything that thinks a syn flood comes from one IP
address isn't going to recognize this.  Another technique is just to
randomly close out several connections in the middle of the queue - that
would work better here, but would probably also randomly trash real
connections that were too slow to complete.

"Interrupted system call" isn't something grex can create directly.
This would result from an interrupt caused by some event on the remote
end (such as alarm(2)) that caused a system call to be aborted.
Probably that means something was too slow at grex's end, although that
seems difficult to believe, tcp & standard smtp timeouts are pretty big.
Perhaps it means something quite different.

response 240 of 248:

Dec 19 03:47 UTC 2002

Russ, can you look at the Received: lines and tell where it got hung up?

response 242 of 248:

Dec 19 15:29 UTC 2002

Still running slowly, on and off.  I waited close to a minute to open my
mailbox with Pine (100 plus message, I admit).  BBS is okay.