response 122 of 286:

Aug 18 03:38 UTC 2004

  
   grex is a spammer, it says here:
  
   REALLY need this fixed,   pp ll ee aa ss ee !!
  
*******  See http://spamblock.outblaze.com/216.93.104.34  <<<<<<<
  
from:
  
The original message was received at Tue, 17 Aug 2004 23:12:48 -0400
from YYYyyy@localhost

   ----- The following addresses had delivery problems -----
XXXXXxxxxxx@lycos.com  (unrecoverable error)

   ----- Transcript of session follows -----
... while talking to lycos-com.mr.outblaze.com.:
>>> RCPT To:<XXXXxxxx@lycos.com>
<<< 554 EMail from mailserver at 216.93.104.34 is refused. 

*******  See http://spamblock.outblaze.com/216.93.104.34  <<<<<<<

554 XXXXxxxx@lycos.com... Service unavailable

   ----- Original message follows -----

--XAA22521.1092798775/grex.cyberspace.org
Content-Type: message/rfc822

 /cry

response 124 of 286:

Aug 19 20:11 UTC 2004

The TTY queue is hosed, as is part of spring agora.

response 126 of 286:

Aug 27 16:23 UTC 2004

 It sure does..

response 128 of 286:

Aug 27 16:30 UTC 2004

Okay, I believe I have /tmp permissioned correctly now.  I'm still a little
groggy, so holler if that's not it.

response 130 of 286:

Aug 27 17:20 UTC 2004

Yeah, there were some problems restoring agora49 (the previous agora to this
one).  A lot of it's files are munged.

response 132 of 286:

Aug 27 17:27 UTC 2004

With the problems that grex experienced, would there have been any way to
display some terse "system down because X" + auto-logout during connection
via telnet or dialup?

response 134 of 286:

Aug 27 18:10 UTC 2004

Onion rings & a chocolate shake, please!  :-)

response 136 of 286:

Aug 27 18:37 UTC 2004

Here's what I know:

  Grex's disk drive zero died.  This disk is used to boot the system, and
  contains most of the operating system (root, /usr, /usr/local) and the
  bbs data (/bbs).  It was still partially working, but not enough to
  boot the system.

  Our last tape backup was many months old.  I forgot how many, but way too
  many.

  STeve and Kip did nearly all the work restoring things.  I was on vacation
  when this started and didn't get involved till late and then only in
  limited ways.

  STeve and Kip's initial problem was that you can't build a new disk on
  a machine that you can't boot.  So they're plan was to take Grex's tape
  drive to one of the machines at Kip's workplace, and build a new boot
  disk for Grex from the backup tape there.  However, they didn't have
  the latest backup tape, which (appropriately) was not stored on site.
  So they got the tape drive hooked up, and discovered that they couldn't
  read the much older tape backups that they had brought from the pumpkin.

  Later, Joe passed the latest backup tape to STeve.

  I came home from vacation, and spent a little time in the pumpkin.  Since
  I did much of the original building of the current Grex system, I
  remembered that we had a CD-drive for Grex (standard ones won't work
  with SunOS) and a 4.1.4 distribution CD which can be booted.  I hooked
  this up and figured out how to boot from the CDrom and documented this.
  Booting from the CDrom gives you an extremely limited set of tools. It
  looked to me to be too limited to actually do anything useful, plus I
  had neither the tape drive, nor the backup tape, so I left things at that.

  Kip and STeve again got together, with tape drive and backup tape.  They
  actually managed to figure out how to do a restore when booted from the
  CDrom.  (I'd still like to know how they did that.)  However, they
  discovered that most of the spare disk drives in the pumpkin were
  unusuable.  Some are differential drives.  Some are too small.  The
  only viable candidates were four 4Gig Conner drives.  They tried two
  of these and found that both were defective.  (I had tried using one
  of these years ago and found it didn't work, but they didn't know that).

  When I saw their emailed report the next day, I went and searched my
  house for some other disk drives.  When we had started putting together
  the new Grex, but didn't have drives yet, I had borrowed some drives from
  the pumpkin that I thought I could use temporarily.  These were 4Gig
  Seagate drives, which had previously been used on a development system
  that we ran for a while called "grease".  I never ended up using them on
  the NextGrex project, and couldn't really remember what I'd done with them.
  I found them in my garage and returned them to the pumpkin.

  STeve and Kip did another late night session, restoring the backups of
  root, /usr, and /usr/local onto one of the Seagate disks.  Years ago,
  STeve had set up a cron process that backed up the /etc/passwd file and
  related files to various other disks periodically, so they had a current
  copy of that.  However, he had not backed up /etc/group or the system
  mail aliases, so new versions of those have to be built.  There is
  probably more work to do updating things that had changed since the last
  backup, but not that much has.

  I believe the got /bbs by reading off the dead drive.  Mostly the /bbs
  partition of the old drive was still readable, but there seems to have been
  some damage.  Items 19 through 58 in oldagora (agora49) where trashed.
  I think someone said they had an offsite backup of this though, so we may
  be able to restore those.

  Overall, we took way too long to get this job done.  We repeatedly allowed
  ourselves to fall into resource deadlocks - first STeve and Kip couldn't
  do much because Joe had the backup tape, then I couldn't do much because
  STeve and Kip had the backup tape and tape drive respectively, then STeve
  and Kip couldn't do much because I had the only good disk drives.  Each
  time STeve and Kip's work got blocked, it took them some time to be able
  to get together again - they both have very busy schedules.

  I think that part of the problem is that knowledge of these old Sun systems
  is thinly scattered.  STeve and Kip are experienced system administrators,
  but I doubt either has done much SunOS work for years.  I'm not a system
  administrator at all, and my knowledge about this stuff is very spotty, but
  I did do a lot of the "recent" work on Grex, so I know more about what
  CD drives and disk drives and bootable CD's we have.  I don't think any
  one of us knows enough to be able to readily do this kind of job on our
  own.  This means that we have to work together on jobs like this, and
  that slows things down, as it is hard to coordinate among us.  Hopefully
  this will be less of an issue with the new machine and operating system
  which are better known to more people.

response 138 of 286:

Aug 27 22:10 UTC 2004

A huge thank you to Kip, STeve and Jan.  While the rest of us
were missing Grex you folks were spending hours of your time 
thrashing though problem after problem.  You are our heroes.

response 140 of 286:

Aug 28 01:06 UTC 2004

Thanks Jan, that was a very good summary of the events.

As for the "trick" to restoring from tape while booting from the CD,
actually after booting from the CD, you have an option to install a miniroot
system (like a modern Linux rescue disk) to one of the swap partitiions on
the system and then boot from that where you can then create mount points and
start to work with the filesystems and eventually restore from the tape. 

A rather nice feature, wouldn't you say?

response 142 of 286:

Aug 28 03:18 UTC 2004

Thanks to Kip, Steve, Jan and all others who resurrected Grex! (charcat does
the snoopy happydance!)

response 144 of 286:

Aug 28 03:44 UTC 2004

One advantage of the new machine, which could be put to use on the current
one, is that much of the documentation, and thus the critical files, are
being stored in CVS, on a separate machine.  

Yes, backups can be done to CD instead of tape.  Backups can also be done
to separate disks.  As disk gets cheaper, many folks are finding it makes
more (economic) sense to mirror to disk than to tape or CD.