response 67 of 176:

Nov 30 02:25 UTC 2005

The problem with messages in the motd is that people forget to change them
when they get outdated so we tend to ignore them.  If there were only relevant
messages there I would read them.  I don't really want to know that grex was
down two weeks ago for a day.

response 69 of 176:

Nov 30 04:09 UTC 2005

If not the motd, where?  It was also discussed in at least one item here and
in Agora.  Short of sending email to every account on Grex what are we
supposed to do.  I manage to glance at the motd every time I log on enough
to notice if something new is posted.  It only takes a couple of seconds, it
isn't that long and it has been rather up to date lately.  If you choose it
ignore it, that is more your problem than it is staff's.  Yes, I agree that
outdated things should be removed, but lets get real here.

response 71 of 176:

Nov 30 04:31 UTC 2005

  Now *that* is a good idea, a mailing list for announcements of system
work, downtime, etc.  Excellent.

  The impact of staff sending out mail to every acocunt on Grex would
be 1) to take about 20 minutes of system pounding to deliver about
29,000 emails, 2) consume about 50M of /var/mail space, and 3) would
likely generate a couple hundred emails back with 1/2 asking if this
was real, and the other half asking about why and when the system
would be back up, regardless of what we said in the mail. ;-)

response 73 of 176:

Nov 30 12:20 UTC 2005

Re#67 Sindi, we can certainly remove motd messages more aggressively.
Notices for things such as recent outages tend to stay in motd for
at least a week to insure that folks not regularly logging in or
reading the conferences still will have some idea why the system
may have recently crashed or otherwise been unavailable.

A weeks notice for major notices is a (hopefully) reasonable balance
between those who log in daily and those who hit the system at least
weekly (arguably it is a balance between those who only need to be
told once and those for whom the message may not register until
they've seen it several times).

response 75 of 176:

Nov 30 14:08 UTC 2005

Steve, I was referring to Sindi's complaint that motd has messages
about past crashes and outages too long *after* the event.  I made
no comment as to how much warning there should be before there is
an outage.

I think our current routine of announcing system down time several
days in advance for scheduled downtime, and best effort warning for
anything else is sufficient.

On a related note, I think we should commit to updating the hvcn
page with current system status *before* commencing any system work,
emergency or otherwise, that will keep grex down or unavailable for
more than a few minutes.

response 77 of 176:

Nov 30 14:55 UTC 2005

an operation as large as an os upgrade ought to have had a 
written checklist - and that checkllist could have been discussed
looong before hand in public, agora &/or coop.
  
but then shoulda/coulda/woulda only has recrimination value after the fact.
  
since this type of operation isn't about to happen too often, the
disaster is just  lurking around - AS ALWAYS - waiting for memories
to fade .
  
teh previous upgrades were not as complicated, had much more notice,
and were thought through with more precision - might even have had
a written checklist handy!
  

response 79 of 176:

Nov 30 18:06 UTC 2005

   I'm not really sure how to respond to what you are saying.  I
don't think you understand the nature of the upgrade, in that 
there was no path back.  The op system had problems with both
the networking card and filesystem issues, and as an extra
treat hardware problems.

   This all dances around the critical issue that no one should
keep valuable mail on Grex only.  Keeping valuable mail in the
/var/spool area is even worse; it's an active filesystem, the
most active one on Grex and as such is more prone to failures
than anything else.

response 81 of 176:

Nov 30 19:11 UTC 2005

(hah)

response 83 of 176:

Dec 1 02:15 UTC 2005

   And you don't think I care about Grex?  Good God Mark, I made a mistake.
WHERE do you think that I a) don't feel badly about missing it, and b)
that I don't care?

response 85 of 176:

Dec 1 04:17 UTC 2005

I haven't logged into Grex in the last year or two, but I've been
lurking on the staff mailing list.  If I lost anything in this backup
error, it wasn't anything I cared about.  That said, I, too, am somewhat
puzzled at the procedure that was followed.

I got my start doing Internet stuff as a member of the Grex staff more
than ten years ago, so I remember the constraints we had to work under
then.  Grex was a rare piece of ancient Sun hardware, disks were really
expensive, and none of it was any more reliable than most of the other
stuff running on the Internet in those days.  When something needed to
be done, it often meant taking the system offline, sometimes for a full
weekend in the case of a few major upgrades or disk crashes.  We had a
much bigger staff back then, and for many of us whose social lives
revolved around the Grex community it was a pretty high priority, so
when something needed to be done there were typically lots of people
around to work on it.  I can certainly see how doing things as we did
them then, but with a smaller and less focused on Grex staff, would lead
long periods of downtime.

But I'm puzzled about why I see the same methods being used on Grex now,
when hardware is considerably cheaper and staff time appears to be a
much scarcer resource.  My perspective is arguably a bit skewed.  The
non-profit where I'm now a paid full-time staff member is pretty
impoverished, but still has a budget a couple of orders of magnitude
higher than Grex's, and I tend to come at systems stuff as a manager
rather than as a hands-on sysadmin these days.  Still, it doesn't look
to me like the problems that are being talked about here are difficult
to solve.

If I recall correctly, Grex is now running on PC hardware that's at
least two or three years old.  In other words, getting some equivalent
systems should be cheap (or free, given that that's replacement age at a
lot of places, and Grex is 501(c)3).  Installing new software versions
on new hardware, testing, and then copying over whatever is dynamic at
the last minute, seems pretty obvious.  Falling back to the old system
at that point if something doesn't work is at most a matter of moving an
ethernet cable.  Likewise, having spare systems ready to copy whatever
is dynamic onto is a good way of dealing with hardware failures.

This really, I think, comes down to whether anybody still cares enough
about Grex to make it worth dealing with.  My own view is that the
community I once cared about seems to have gone on to other things, and
the services Grex is providing aren't anything special anymore.  But if
people care about keeping Grex operating, it looks like something needs
to change.

response 87 of 176:

Dec 1 18:29 UTC 2005

I think SCG (hi, Steve!) hits a few important points in his resp:85.
 
In particular, I would like to stress the need to evolve our thinking
from "This computer is Grex," to "This internet service is Grex, and 
these are the hardware components we have to support our service."
Right now, *everything* is a single point of failure for Grex, 
and as we just learned, staff can't back out of an upgrade because
the upgrade is done on top of the old disks.
 
Amazon.com and LiveJournal don't go dark for a week while they 
do upgrades; they acquire the hardware they need so that upgrades
can be rolled into production with a minimum of disruption.

----

Longer term: all of the community-building services Grex offers
are now offered, for free, by large organizations with professional
support staffs.  The one thing which isn't common is the open access
to a shell prompt; but that's also one thing which creates huge 
social/behavior management problems.  It's also unclear to me if that's 
a core function of Cyberspace Communications as it was organized,
rather than the tool towards the community-building goals which 
was available 14 years ago.

response 89 of 176:

Dec 1 18:59 UTC 2005

 You're right that Grex doesn't have as much money to spend as Amazon
 or LiveJournal, though I don't think that point escaped anyone even
 before you explicitly stated it.  A more salient point is that Grex
 has enough money in the bank to afford a backup disk.  We just didn't
 plan to use it for that.