response 139 of 457:

Apr 26 21:34 UTC 2005

This is also generating messages about new mail and there is never any new
mail, at least not today.

response 141 of 457:

Apr 26 22:10 UTC 2005

Can /var/mail be given a larger drive?  If my calculations are correct it is
occupying about an 8G right now.

response 143 of 457:

Apr 26 22:34 UTC 2005

What does "move on" mean?

Please replace the Grex website's blue ribbon with:
"If you need reliable mail you're not going to get it from Grex."

response 145 of 457:

Apr 26 22:47 UTC 2005

Exactly.  If you NEED mail, get a free account elsewhere.  You'll 
not be happy with what Grex can offer at the moment.

response 147 of 457:

Apr 26 22:58 UTC 2005

Thanks, Mike.  I know your efforts will be appreciated.

response 149 of 457:

Apr 26 23:17 UTC 2005

I think all that mail needs is more space than is available on /var/mail. 
It keeps filling up.  Remmers moved some things off var - if there is nothing
else to be moved, can we add another drive until the next BSD version is
available?

response 151 of 457:

Apr 26 23:35 UTC 2005

Increasing the space in /var/mail will only help for only a short time.  The
new amount of space will just fill up the same as the smaller space.  And,
we cannot increase the space anywhere until we get the new drive and get it
up and running.  There just isn't any space to allocate to it right now.

response 153 of 457:

Apr 27 00:31 UTC 2005

I tried yahoo mail and hated it.  Webmail is always slow and annoying.

Eventually a systemwide spam filter will drastically reduce the amount of
unread and unwanted mail sitting in unused accounts and free up space.

What is keeping grex from installing a new drive sooner?  There was something
about a new drive being saved for a new BSD installation but what would it
cost to have two new drives?  

response 155 of 457:

Apr 27 00:35 UTC 2005

 I believe that reducing the amount of incoming Spam will dramatically
 reduce the demand for mail storage space and finding ways to block 
 some of it will be an important step towards a sustainable mail service.

response 157 of 457:

Apr 30 08:19 UTC 2005

On Wednesday, April 27, 2005, STeve Andre wrote, in a message:

}  Someone did something to use up all the CPU.  You can start an ssh telnet
} or ftp session but nothing ever starts so its brain dead at the moment.

Attmepts to reboot the machine failed.  When we did get the machine to
reboot, we discovered that "the pw database's in a totally inconsistent
state.  /etc/passwd had about 1137 entries in it, and the database files were
of different lenghts.  This means the passwd system was in chaos, . . . "
(STeve, in a message dated April 29, 2005).  Which is why some people could
log on, but others couldn't.

At or about 02:42 this morning (April 30, 205), STeve wrote: " Grex is up at
the moment.  The damage done with the accounts can be fixed this weekend or
not, but it won't affect all the other accounts.  Better to have Grex up now
for the majority of people on Grex."

There appear to be some problems with newuser, so it has been turned off
until the problems can be investigated and repaired.  The web newuser program
is also disabled.

response 159 of 457:

Apr 30 14:56 UTC 2005

splat the trolls = remove the ribbon

response 161 of 457:

Apr 30 22:18 UTC 2005

It did get updated at 23:00 or so last night.  I'll leave it that way a bit
longer.

response 163 of 457:

May 1 05:48 UTC 2005

   I'm baack. ;-)

   I wish things had gone more evenly over the last three weeks.  It's
been alternately boring and too exciting lately.  I think things are
evening out, thankfully.

   Grex went down on Tuesday because of what I believe to be a fork
bomb, which pretty much absorbed all the system.  Some will remember
that the start of a telnet or ssh session worked and then nothing.
I think that was the damon responding to the socket connect but then
not being able to do anything else.  During that time there were
people trying to create accounts, primarily via the web newuser, and
pretty much all hell broke loose internally.  The new accounts were
created in the /etc/passwd file OK, but the logging of them is just
totally bizarre.  There are multiple entries for many accounts,
with incorrect data associated with them.  I'm still trying to sort
all that out.

   Once I got over to Provide.net it became clear that the passwd file
was really messed up.  Root's password wasn't what it should be, nor
were any others that I knew of.  Booting Grex into single user mode
revealed an /etc/passwd file of about 1100 accounts, not the 24,000
that it should have had.  Worse, the "master.passwd" main passwd file
and associated database (.db) files were messed up as well. The
master.passwd file had a different line count than passwd did, which
is horribly wrong.

   Hours before I went into the hospital on April 6th I discoverd
that the reason why Grex couldn't create accounts was because of a
bad disk spot that was underneath one of the passwd database files
and prevented accounts from being updated properly.  It was at that
point that I made a backup of the /etc directly.  That proved very
useful.

   At Provide.net I finally figured out the stuff about the weirdness
with the passwd files and copied my set over.  That then let me put
Grex back into multi-user mode with an /etc/nologin file so Grex
could start processing mail.  Once Glenda and I were satisfied that
the system seemed to be OK other than the passwd stuff, we left
Provide.

   As an aside, Grex now lives in the Attic.  This is the space onb
the second floor of Provide.net, and with its curved in walls it looks
like an attic.  It's a nice facility by the way, nicely air-conditioned
and several UPS's to feed all the computers.  The highest tech attic
I've yet encountered. ;-)

   Anyway, it was leaving Provide that was a mistake.  By bringing Grex
up and not doing anything special, I wound up destroying a good perfect
copy of the passwd file.  To understand this you have to understand that
Jan wrote a very nice set of shell scripts which make backups of Grex
onto our IDE disk on the /mirror partition.  This occurs every day so
we have a backup of things.  It's already come in handy for me, twice.
The problem is that it only has one level of backups.  The problem was
that I had the passwd stuff in place on /etc (my version from 4/6) and
the backup script ran.  That then overwrote the complete copy of passwd
stuff.  By the time I realized this might happen it was too late.  We
had a copy of the backup passwd in /mirror.

   That was stupid of me and I think I owe about 1,100 apologies to 
the people whose accounts are in limbo.  Their home directories are 
on the disk, but with no associated entries in the passwd file.

   It was then that I discovered the data in the nulogfile (the copy
of newuser runs) was crazy.

   So I'm now in the process of sorting that data out and figuring out
the best way to restore accounts.  If the password information in the
nulogfile is correct I think I can restore some? all? accounts.  We'll
see.

   After the problem on 4/6 I remembered that I needed to create the
scripts I wrote on Sunos, where copies of /etc/passwd were made every
6 hours.  I did not do this, much to my shame.  Had I done this none
of this passwd weirdness would have mattered.  I'm going to fix that
soon, and add something, namely teach a machine at work to sftp into
Grex once a day and grab a copy of master.passwd and group, so we'll
have an off site backup of this data.

   Making a few comments on stuff I've read earlier in this item:
We need to alter the size of some of the partitions on Grex.  In
particular the /var/mail partiton is too small.  We keep on running
into this because of the ever increasing amounts of spam we're getting

   Speaking of spam, it is my *hope* that I'll be able to get back
to working with Exim and spam assasin soon, and talk with other staff
about using it here.  Spam is a *complex* problem, one that has no
easy solutions, but we now have enough raw CPU power to start dealing
with it.  We've not done this for far too long.

   I'm not sure why we haven't ordered the new disk yet; I think
there has been some confusion over this.  I hope thats settled before
long.  This touches on the upgrade we need.  OpenBSD 3.7 has started
to ship, so I think we can upgrade sometime in May.  There are several
things that have been improved, including support for our network card,
which has caused one or two panics.

   So thats it for the moment.