response 136 of 286:

Aug 27 18:37 UTC 2004

Here's what I know:

  Grex's disk drive zero died.  This disk is used to boot the system, and
  contains most of the operating system (root, /usr, /usr/local) and the
  bbs data (/bbs).  It was still partially working, but not enough to
  boot the system.

  Our last tape backup was many months old.  I forgot how many, but way too
  many.

  STeve and Kip did nearly all the work restoring things.  I was on vacation
  when this started and didn't get involved till late and then only in
  limited ways.

  STeve and Kip's initial problem was that you can't build a new disk on
  a machine that you can't boot.  So they're plan was to take Grex's tape
  drive to one of the machines at Kip's workplace, and build a new boot
  disk for Grex from the backup tape there.  However, they didn't have
  the latest backup tape, which (appropriately) was not stored on site.
  So they got the tape drive hooked up, and discovered that they couldn't
  read the much older tape backups that they had brought from the pumpkin.

  Later, Joe passed the latest backup tape to STeve.

  I came home from vacation, and spent a little time in the pumpkin.  Since
  I did much of the original building of the current Grex system, I
  remembered that we had a CD-drive for Grex (standard ones won't work
  with SunOS) and a 4.1.4 distribution CD which can be booted.  I hooked
  this up and figured out how to boot from the CDrom and documented this.
  Booting from the CDrom gives you an extremely limited set of tools. It
  looked to me to be too limited to actually do anything useful, plus I
  had neither the tape drive, nor the backup tape, so I left things at that.

  Kip and STeve again got together, with tape drive and backup tape.  They
  actually managed to figure out how to do a restore when booted from the
  CDrom.  (I'd still like to know how they did that.)  However, they
  discovered that most of the spare disk drives in the pumpkin were
  unusuable.  Some are differential drives.  Some are too small.  The
  only viable candidates were four 4Gig Conner drives.  They tried two
  of these and found that both were defective.  (I had tried using one
  of these years ago and found it didn't work, but they didn't know that).

  When I saw their emailed report the next day, I went and searched my
  house for some other disk drives.  When we had started putting together
  the new Grex, but didn't have drives yet, I had borrowed some drives from
  the pumpkin that I thought I could use temporarily.  These were 4Gig
  Seagate drives, which had previously been used on a development system
  that we ran for a while called "grease".  I never ended up using them on
  the NextGrex project, and couldn't really remember what I'd done with them.
  I found them in my garage and returned them to the pumpkin.

  STeve and Kip did another late night session, restoring the backups of
  root, /usr, and /usr/local onto one of the Seagate disks.  Years ago,
  STeve had set up a cron process that backed up the /etc/passwd file and
  related files to various other disks periodically, so they had a current
  copy of that.  However, he had not backed up /etc/group or the system
  mail aliases, so new versions of those have to be built.  There is
  probably more work to do updating things that had changed since the last
  backup, but not that much has.

  I believe the got /bbs by reading off the dead drive.  Mostly the /bbs
  partition of the old drive was still readable, but there seems to have been
  some damage.  Items 19 through 58 in oldagora (agora49) where trashed.
  I think someone said they had an offsite backup of this though, so we may
  be able to restore those.

  Overall, we took way too long to get this job done.  We repeatedly allowed
  ourselves to fall into resource deadlocks - first STeve and Kip couldn't
  do much because Joe had the backup tape, then I couldn't do much because
  STeve and Kip had the backup tape and tape drive respectively, then STeve
  and Kip couldn't do much because I had the only good disk drives.  Each
  time STeve and Kip's work got blocked, it took them some time to be able
  to get together again - they both have very busy schedules.

  I think that part of the problem is that knowledge of these old Sun systems
  is thinly scattered.  STeve and Kip are experienced system administrators,
  but I doubt either has done much SunOS work for years.  I'm not a system
  administrator at all, and my knowledge about this stuff is very spotty, but
  I did do a lot of the "recent" work on Grex, so I know more about what
  CD drives and disk drives and bootable CD's we have.  I don't think any
  one of us knows enough to be able to readily do this kind of job on our
  own.  This means that we have to work together on jobs like this, and
  that slows things down, as it is hard to coordinate among us.  Hopefully
  this will be less of an issue with the new machine and operating system
  which are better known to more people.

response 138 of 286:

Aug 27 22:10 UTC 2004

A huge thank you to Kip, STeve and Jan.  While the rest of us
were missing Grex you folks were spending hours of your time 
thrashing though problem after problem.  You are our heroes.

response 140 of 286:

Aug 28 01:06 UTC 2004

Thanks Jan, that was a very good summary of the events.

As for the "trick" to restoring from tape while booting from the CD,
actually after booting from the CD, you have an option to install a miniroot
system (like a modern Linux rescue disk) to one of the swap partitiions on
the system and then boot from that where you can then create mount points and
start to work with the filesystems and eventually restore from the tape. 

A rather nice feature, wouldn't you say?

response 142 of 286:

Aug 28 03:18 UTC 2004

Thanks to Kip, Steve, Jan and all others who resurrected Grex! (charcat does
the snoopy happydance!)

response 144 of 286:

Aug 28 03:44 UTC 2004

One advantage of the new machine, which could be put to use on the current
one, is that much of the documentation, and thus the critical files, are
being stored in CVS, on a separate machine.  

Yes, backups can be done to CD instead of tape.  Backups can also be done
to separate disks.  As disk gets cheaper, many folks are finding it makes
more (economic) sense to mirror to disk than to tape or CD.

response 146 of 286:

Aug 28 19:31 UTC 2004

The one limitation of backing up to disk is that it would still be in close
proximity to the master, so if a disaster struck the pumpkin there would be
no off-site storage to recover from.

response 148 of 286:

Aug 28 23:51 UTC 2004

        runs on a SUN

response 150 of 286:

Aug 29 09:13 UTC 2004

So what OS did first grex use?  And what hardware?

response 152 of 286:

Aug 29 14:56 UTC 2004

I think the first open source version of BSD, BSD/386 was also released in
1991.  It too would have been horribly inadequate for Grex's needs.  FreeBSD,
NetBSD, and OpenBSD were all years later.

I presume that it was SunOS 4.1.3 on the Sun 2.  The differences between
that and the SunOS 4.1.4 running on this system are entirely unnoticable.
Mostly bug fixes.

At the time, SunOS was clearly the most stable, most capable version of
Unix available in our price range (probably in any price range).  It's
still a remarkably solid piece of software.  For me the main reason to
move off it is that too many of the open source packages that we want to
use (like mysql) no longer compile on SunOS.

response 154 of 286:

Aug 30 17:14 UTC 2004

nice job ... we all appreciate the efforts and results -thank you

response 156 of 286:

Aug 31 20:13 UTC 2004

i'm sorry.

i use to work there.  yuk.

response 158 of 286:

Aug 31 20:29 UTC 2004

i remember some of the grafitti from the torlet:

"The Fleetwood makes me shit PURE WATER SHIT."

 accompanied by a childlike drawing of a screaming
person sitting on a torlet.

the cook use to pork his girlfriend in the storeroom and would
ash his ciggies in the chilipot.

bon appetit!