response 151 of 457:

Apr 26 23:35 UTC 2005

Increasing the space in /var/mail will only help for only a short time.  The
new amount of space will just fill up the same as the smaller space.  And,
we cannot increase the space anywhere until we get the new drive and get it
up and running.  There just isn't any space to allocate to it right now.

response 153 of 457:

Apr 27 00:31 UTC 2005

I tried yahoo mail and hated it.  Webmail is always slow and annoying.

Eventually a systemwide spam filter will drastically reduce the amount of
unread and unwanted mail sitting in unused accounts and free up space.

What is keeping grex from installing a new drive sooner?  There was something
about a new drive being saved for a new BSD installation but what would it
cost to have two new drives?  

response 155 of 457:

Apr 27 00:35 UTC 2005

 I believe that reducing the amount of incoming Spam will dramatically
 reduce the demand for mail storage space and finding ways to block 
 some of it will be an important step towards a sustainable mail service.

response 157 of 457:

Apr 30 08:19 UTC 2005

On Wednesday, April 27, 2005, STeve Andre wrote, in a message:

}  Someone did something to use up all the CPU.  You can start an ssh telnet
} or ftp session but nothing ever starts so its brain dead at the moment.

Attmepts to reboot the machine failed.  When we did get the machine to
reboot, we discovered that "the pw database's in a totally inconsistent
state.  /etc/passwd had about 1137 entries in it, and the database files were
of different lenghts.  This means the passwd system was in chaos, . . . "
(STeve, in a message dated April 29, 2005).  Which is why some people could
log on, but others couldn't.

At or about 02:42 this morning (April 30, 205), STeve wrote: " Grex is up at
the moment.  The damage done with the accounts can be fixed this weekend or
not, but it won't affect all the other accounts.  Better to have Grex up now
for the majority of people on Grex."

There appear to be some problems with newuser, so it has been turned off
until the problems can be investigated and repaired.  The web newuser program
is also disabled.

response 159 of 457:

Apr 30 14:56 UTC 2005

splat the trolls = remove the ribbon

response 161 of 457:

Apr 30 22:18 UTC 2005

It did get updated at 23:00 or so last night.  I'll leave it that way a bit
longer.

response 163 of 457:

May 1 05:48 UTC 2005

   I'm baack. ;-)

   I wish things had gone more evenly over the last three weeks.  It's
been alternately boring and too exciting lately.  I think things are
evening out, thankfully.

   Grex went down on Tuesday because of what I believe to be a fork
bomb, which pretty much absorbed all the system.  Some will remember
that the start of a telnet or ssh session worked and then nothing.
I think that was the damon responding to the socket connect but then
not being able to do anything else.  During that time there were
people trying to create accounts, primarily via the web newuser, and
pretty much all hell broke loose internally.  The new accounts were
created in the /etc/passwd file OK, but the logging of them is just
totally bizarre.  There are multiple entries for many accounts,
with incorrect data associated with them.  I'm still trying to sort
all that out.

   Once I got over to Provide.net it became clear that the passwd file
was really messed up.  Root's password wasn't what it should be, nor
were any others that I knew of.  Booting Grex into single user mode
revealed an /etc/passwd file of about 1100 accounts, not the 24,000
that it should have had.  Worse, the "master.passwd" main passwd file
and associated database (.db) files were messed up as well. The
master.passwd file had a different line count than passwd did, which
is horribly wrong.

   Hours before I went into the hospital on April 6th I discoverd
that the reason why Grex couldn't create accounts was because of a
bad disk spot that was underneath one of the passwd database files
and prevented accounts from being updated properly.  It was at that
point that I made a backup of the /etc directly.  That proved very
useful.

   At Provide.net I finally figured out the stuff about the weirdness
with the passwd files and copied my set over.  That then let me put
Grex back into multi-user mode with an /etc/nologin file so Grex
could start processing mail.  Once Glenda and I were satisfied that
the system seemed to be OK other than the passwd stuff, we left
Provide.

   As an aside, Grex now lives in the Attic.  This is the space onb
the second floor of Provide.net, and with its curved in walls it looks
like an attic.  It's a nice facility by the way, nicely air-conditioned
and several UPS's to feed all the computers.  The highest tech attic
I've yet encountered. ;-)

   Anyway, it was leaving Provide that was a mistake.  By bringing Grex
up and not doing anything special, I wound up destroying a good perfect
copy of the passwd file.  To understand this you have to understand that
Jan wrote a very nice set of shell scripts which make backups of Grex
onto our IDE disk on the /mirror partition.  This occurs every day so
we have a backup of things.  It's already come in handy for me, twice.
The problem is that it only has one level of backups.  The problem was
that I had the passwd stuff in place on /etc (my version from 4/6) and
the backup script ran.  That then overwrote the complete copy of passwd
stuff.  By the time I realized this might happen it was too late.  We
had a copy of the backup passwd in /mirror.

   That was stupid of me and I think I owe about 1,100 apologies to 
the people whose accounts are in limbo.  Their home directories are 
on the disk, but with no associated entries in the passwd file.

   It was then that I discovered the data in the nulogfile (the copy
of newuser runs) was crazy.

   So I'm now in the process of sorting that data out and figuring out
the best way to restore accounts.  If the password information in the
nulogfile is correct I think I can restore some? all? accounts.  We'll
see.

   After the problem on 4/6 I remembered that I needed to create the
scripts I wrote on Sunos, where copies of /etc/passwd were made every
6 hours.  I did not do this, much to my shame.  Had I done this none
of this passwd weirdness would have mattered.  I'm going to fix that
soon, and add something, namely teach a machine at work to sftp into
Grex once a day and grab a copy of master.passwd and group, so we'll
have an off site backup of this data.

   Making a few comments on stuff I've read earlier in this item:
We need to alter the size of some of the partitions on Grex.  In
particular the /var/mail partiton is too small.  We keep on running
into this because of the ever increasing amounts of spam we're getting

   Speaking of spam, it is my *hope* that I'll be able to get back
to working with Exim and spam assasin soon, and talk with other staff
about using it here.  Spam is a *complex* problem, one that has no
easy solutions, but we now have enough raw CPU power to start dealing
with it.  We've not done this for far too long.

   I'm not sure why we haven't ordered the new disk yet; I think
there has been some confusion over this.  I hope thats settled before
long.  This touches on the upgrade we need.  OpenBSD 3.7 has started
to ship, so I think we can upgrade sometime in May.  There are several
things that have been improved, including support for our network card,
which has caused one or two panics.

   So thats it for the moment.

response 165 of 457:

May 1 18:03 UTC 2005

I've heard of such a program from, I think, Symantec. Designed to deal with
the fact that modern hard drives have circuitry which attempts to hide the
fact that some of the disk goes bad thoughout its service life by moving data
around to good sectors and lying about the actual state of the media. I forget
what it's called.

response 167 of 457:

May 1 21:56 UTC 2005

   The disks were bought around May 2003.  Checking disks for problems
is a difficult thing.  There is a system that IBM delevoped for its
own disks that has failed to catch more problems than it has found,
in my usage of it.  Grex munches on disks.  We might want to consider
replacing them every X years, but figuring out what X should be is
interesting.

   I don't see that needing to increase the size of /var as an
indication that we need to stop doing email.  Disk is the one thing
that has dropped in cost over everything else.  For about $250 we
could devote at 36G disk to mail and likely have enough disk for some
time.  Also, with spam filtering our disk needs will slow down.

   If Grex is on some government list, which I wouldn't be surprised
if true, it doesn't matter if we offer mail or not.  The fact that
we're an open system is enough. This is still America and the secret
police aren't quite here yet.  I'm not going to worry about it.

response 169 of 457:

May 2 01:09 UTC 2005

whoa, steVE!  are your lungs ok ?!

response 171 of 457:

May 2 02:59 UTC 2005

I have several friends who use grex ONLY for email and one of them was a paid
member (she may stop paying since she lives in Chelsea) but the others still
appreciate the email.  I told them they should not feel obligated to pay for
light use.  One of them also has an ISP with mail but got used to grex.  

STeve, were these disks bought new in 2003 and only put into service a few
months ago?  If so, would a warranty at least cover them going bad if we got
similar ones new now and they lasted under a year?

response 173 of 457:

May 2 04:05 UTC 2005

,