response 136 of 176:

Dec 7 23:44 UTC 2005

I'm seeing a lot of comments here about how things work in commercial
environments, making it sound as if there's one way of doing things in
such places.  In fact, from what I've seen, there's a pretty wide
spectrum.  Commercial organizations have a wide variety of experience,
budgets, resource constraints, contractual obligations, perceived levels
of importance, and operational philosophies, even if they're providing
services that may look quite similar from the outside.

It seems non-useful for people to say, "commercial content providers do
X, therefore Grex should too."  It likewise seems non-useful to say,
"Grex isn't a commercial organization, so it can't do what commercial
organizations do."

It's perhaps worth taking a look at change management procedures in some
of the slowest changing but most stable network operators -- traditional
phone companies.  At the one I worked in the web hosting division of,
nothing could be done without filling out lots of change management
documentation: extensive documentation about the change procedure,
including exact commands that would entered, test procedures, backout
plans, justification of why the change was needed, who was going to be
involved, when it was going to happen, what the impacts were going to be
and to which customers, and so forth.  This all had to go through a
committee, which might approve it a couple of weeks after it was
submitted.  It wasn't fun.  Nobody did anything just because they
thought it might make some small incremental improvement.  Problems were
often left alone until they became emergencies, because the bureaucracy
involved in fixing them would become somewhat easier then.  But at the
same time, human error-caused outages became pretty rare.  The committee
that reviewed these things didn't really know how to do anything other
than see if the questions had been answered, but answering the questions
forced people to think through things carefully.  Adopting a very
stripped down version of that protocol, asking people to answer a list
of standard questions to their own satisfaction before diving into major
changes, gets a lot of the same benefits and doesn't cost much.

There are also the comments I've seen here about enterprise-class
hardware that Grex can't afford.  A lot of commercial sites also can't
afford it, or decide it's not worth the cost.  A lot of services which
the Internet would be perceived as not working without -- some of the
root and top level DNS infrastructure, Akamai caches, Google, etc. --
involves standard off the shelf hardware deployed in large enough
numbers that if some piece of it breaks, end users won't notice in the
few days it may take to fix it.  What sort of hardware to use, how much
of it, and how much support to provide in case it breaks, are
interrelated decisions with costs associated, and different
organizations come up with different answers.

Managing volunteers is different than managing employees.  Managing
employees who are paid less than they could earn elsewhere is different
than managing employees who are paid more than they could earn
elsewhere.  A general question to ask is, "are we getting more out of
this person than we're paying them."  I've dealt with employees who have
been hard to deal with, but who were occasionally doing things that were
really important, and they've seemed worth keeping.  I think I've even
been such an employee at a few former jobs.  At my current non-profit
employer, I've "fired" volunteers who were taking more of my time to
manage than it would have to do the work they were doing.  At the same
time, if somebody isn't doing anything, is known to not be doing
anything, and isn't costing anything, telling them to go away probably
isn't all that useful.  Having volunteers who occasionally do something
that wouldn't otherwise get done can be a very useful thing.  Telling
anybody to go away before you're sure you want them gone can have some
less than desirable consequences.  On the other hand, having somebody be
in charge, with at least the authority to tell voluneers what not to do,
may have more positive impact than its cost in ruffled feathers.

response 138 of 176:

Dec 13 20:35 UTC 2005

Please, that's just deflecting responsibility.  Someone really does need to
be "in charge" of Grex.

Besides, arbornet not having its annual meeting isn't necessarily Todd's
fault.

response 140 of 176:

Dec 14 23:21 UTC 2005

a very Romanian response.

response 142 of 176:

Dec 19 17:09 UTC 2005

Being volunteers doesn't remove them from the responsibility to do quality
work when they decide to use the powers over the system they're given.

The whole backup thing was terribly poor work.  Even the most novice,
inexperienced of system administrators know how important backups are.  The
people involved in the mail mishap are apparently a gaggle of fools with FAKE
pocket protectors.

response 144 of 176:

Dec 19 19:01 UTC 2005

The loss of data wasn't caused by system downtime.

It was caused by people not making proper backups.

Even in a volunteer organization, there must be some work ethic.

response 146 of 176:

Dec 19 19:23 UTC 2005

I don't have to be able to "force" something for it to be the right thing.

response 148 of 176:

Dec 19 23:57 UTC 2005

And how do you suppose you could do better?  Backups were made.  A listing
was made of the said backups to see that all the files were there, the listing
report the mail directory and files were there, it just didn't say how big
it was.  Is the person doing the backups supposed to go in and look at all
the 100s of thousands of files individually to make sure that the sizes are
correct?  When I do backups, I do listings to see that the major files exist,
I usually don't unzip them and look at the size, with that many files there
just isn't enough time to do so, especially when there are time limitations.

response 150 of 176:

Dec 20 00:30 UTC 2005

It's not particularly difficult to compare the size of files in an archive
to the size of files in a directory, though the fact you think it is difficult
speaks to your ignorance of Unix.

It's also not particularly difficult to make sure the backup is done right
in the first place.

response 152 of 176:

Dec 20 03:06 UTC 2005

 re #148:  
 >  And how do you suppose you could do better?

 I've tried to refrain from criticizing STeve's mistake for a number
 of reasons -- (1) it doesn't get the deleted mail back, (2) I suspect
 he feels (or felt) bad enough, and (3) nobody else was stepping up to
 volunteer to get the job done and it's unfair how much of the
 responsibility has devolved onto STeve, but..

 Your defense, while commendable from a family loyalty standpoint,
 is wholly misguided from a technical standpoint.  A couple of really
 serious mistakes were made (chiefly, the backup was badly botched 
 and* the decision had been made to repartition in place.) The results
 turned out to be a minor disaster for many of us, and it's insulting
 to pretend that there was no way it could have been prevented..

 >  Backups were made.
  
 As it turned out, some were, some weren't.  That's the issue.

 >  Is the person doing the backups supposed to go in and look at all
 >  the 100s of thousands of files individually to make sure that the
 >  sizes are correct?

 Actually, it's not that hard to write a program to do that, but even
 if you don't want to go to that much trouble one can get a pretty good
 idea by comparing the size taken up by the backup with the size taken
 up by the originals.

response 154 of 176:

Dec 20 06:37 UTC 2005

Actually, this would have been avoided had Steve used the dump program
instead of tar to do the backups, as I suggested.  Steve wrote something
somewhere that I thought was funny that seemed to indicate he thought it
wouldn't have made a difference; actually, it would.  Dump doesn't go
through the filesystem to get the data it backs up; rather, it looks at
the filesystem data on the raw disk devices.  Tar goes through the file
system; hence when it's sensative to whether the disk was mounted at the
time.  A better way to do the backups would have been to use dump.

But I really don't want to beat up on Steve about this.  I've done the
exact same thing myself (luckily, I only deleted the mail spool of one
user, but he was still pretty pissed off).  Hey, live and learn.

My major concern is with grex as a whole, and the idea that no one really
seems to be in charge, despite claims to the contrary.

response 156 of 176:

Dec 20 15:45 UTC 2005

If we all know how grex operates, and should set our expectations accordingly,
then you *are* suggesting that people don't try to do better next time.  You
are, without a doubt, saying that the status quo is perfectly fine.  I am not.

response 158 of 176:

Dec 20 17:42 UTC 2005

Oh, sorry.  My bad.

response 160 of 176:

Dec 20 18:26 UTC 2005

Well, I made a suggestion that grex solicit a staff member to be `in
charge' in the case of a failure.  Others suggested that a written plan
be made prior to a major change (such as an upgrade).  Both of those
seem like suggestions that could make things better.