|
Grex > Coop13 > #294: Why Grex lost its mail partition | |
|
| Author |
Message |
| 25 new of 176 responses total. |
tod
|
|
response 41 of 176:
| 
Nov 29 16:49 UTC 2005 |
I think it is ridiculous to say that Grex lost the mail partition
because of some great organizational fault.
Let's step back for a second. There was a time when it was considered polite
to notify users of intended downtime due to upgrades. That was so users could
ensure they have their precious data moved offline if they felt the need.
It was also so they'd know not to plan on being online at that time.
Organizational fault is written all over the last "upgrade." The Board is
slacking off by letting a bunch of part time hobbyhorse types take the system
offline without forewarning. The board has a fiduciary responsibility to the
members to keep the system around and available. No accountability in this
organization, imo. My recommendation is that the board seek some fresh blood
for the staff and also take some lessons in diplomacy and accountability.
No need to chant "volunteer organization" at me, neither. That's a dead horse
not worth beating and everyone is tired of that excuse.
|
keesan
|
|
response 42 of 176:
|
Nov 29 17:25 UTC 2005 |
Todd, why don't you start your own bbs and do it right? Our volunteers are
not perfect, they admit to this, they accept suggestions, and they don't need
more complaints. A polite request for a few days notice next time grex is
going down would accomplish more than #41. Are you looking for someone to
volunteer as full-time staff?
|
nharmon
|
|
response 43 of 176:
|
Nov 29 17:31 UTC 2005 |
The function of the staff should be to advise the BoD on technical
issues, implement the decisions made by the BoD, and intervene on their
own initiative in some circumstances. I say should be, because with the
exception of a few board motions, there is not a lot that defines what
staff's duties, responsibilities, requirements, etc. are.
Ideally, there should be one person appointed by the BoD responsible for
supervising the staff. This person would be accountable to the BoD, and
the other staffers accountable to him/her.
Right now it seems there is a hash of staff members who do a good job of
working together but without any real guidance or direction. I think
finding someone with the time and drive to give direction is something
Grex needs desperately.
|
nharmon
|
|
response 44 of 176:
|
Nov 29 17:32 UTC 2005 |
I think it is a shame that someone takes the time to voice their
recommend on how to improve Grex only to be told to leave and start
their own BBS if they don't like how Grex is run.
That sort of attitude is exactly what ruins good organizations like this.
|
tod
|
|
response 45 of 176:
|
Nov 29 18:10 UTC 2005 |
#42 of 44: by Sindi Keesan (keesan) on Tue, Nov 29, 2005 (12:25):
Todd, why don't you start your own bbs and do it right?
I'm a member of Cyberspace, Inc. I like this BBS (when its online.)
Our volunteers are not perfect, they admit to this, they accept suggestions,
and they don't need more complaints.
How do we provide a community service without community feedback?
A polite request for a few days notice next time grex is
going down would accomplish more than #41.
I would request politely if I thought the Board would listen. I suspect
that the Board is 2nd fiddle to janc and STeve's whims, though.
Therefore, I'm using a harsher tone in hopes of a constructive response
and action.
Are you looking for someone to volunteer as full-time staff?
I think the Board seriously should be. The downtimes of the recent past
have been at the full mercy of staffers with Grex nowhere near the
top of their priorities. I don't fault them for it but I do fault the
Board for not seeking additional available and willing staff.
Retaining staff should also include a bit more diplomacy in the
way it treats existing volunteers and members.
#43 of 44: by Nathan Harmon (nharmon) on Tue, Nov 29, 2005 (12:31):
Ideally, there should be one person appointed by the BoD responsible for
supervising the staff. This person would be accountable to the BoD, and
the other staffers accountable to him/her.
I agree with you, Nathan. I'd also interject that the BoD is
ultimately responsible. As members, we should not be told to shutup
when we ask the Board why no one is being accountable for Grex.
|
steve
|
|
response 46 of 176:
|
Nov 29 18:25 UTC 2005 |
Grex has never operated with a chief staff person. It's always been
more of a collective thing. It's worked out at least as good as work
places I've been at which had an official structure.
Tod you know as well I as I do that there isn't going to be a
full-time staff person.
|
mcnally
|
|
response 47 of 176:
|
Nov 29 18:39 UTC 2005 |
> I think it is ridiculous to say that Grex lost the mail partition
> because of some great organizational fault.
I don't think that's ridiculous at all.
I think the mirroring scheme was set up by someone other than did the
repartitioning and the person doing the repartitioning didn't fully
understand the implications of the backup scheme, namely that unmounted
partitions aren't backed up.
Furthermore I think that there are organizational issues that led to
the mail disaster in other ways, too. I've refrained from commenting
because I haven't had a good idea how to separate criticism of the
upgrade from criticism of the people who performed the upgrade, but I
personally think it was a very bad idea to upgrade and restore in place.
If we had a spare SCSI disk (or perhaps set of disks) [which we should
have anyway, for disaster recovery] the entire upgrade could have been
performed without ever risking the data on the disk(s) the system had
been running on. As I understand it Grex has got a not excessive, but
still reasonable amount of money in the bank. Perhaps we should
invest in preventing exactly this sort of behavior the next time around.
And while I would never suggest that anyone jettisoned the mail on
purpose, I suspect a contributing factor in the mail loss is that
none of the people involved depend on the mail system here in any
way that's truly important to them. They shouldn't *have to* to
administer the system but it does tend to focus one's attention
when you've got something to lose.
|
tod
|
|
response 48 of 176:
|
Nov 29 18:42 UTC 2005 |
I agree to disagree with STeve about a full-time staff person. The Board can
at least make an attempt to find such person(s) with flexibility in
availability and accountability. A staff of several persons with dedicated
timeslots would be ideal but needs to happen by someone taking that task as
the lead. "We never did it before" and "isn't going to be" are empty excuses,
imo. Why is improving Grex uptime and maintenance so painful a concept?
|
steve
|
|
response 49 of 176:
|
Nov 29 18:46 UTC 2005 |
And how do we pay for this person?
|
steve
|
|
response 50 of 176:
|
Nov 29 18:50 UTC 2005 |
Well Mike, I would have *liked* to have had spare disks for the
upgrade. Here at work I keep entire spare machines such that an
upgrade is done on the next machine, with data transfers done onto
the new machine, and a switchover of IP addresses. I've had as
little as 4 seconds of downtime for such upgrades.
But I did not succeed in getting the board to move on getting
the PC Weasel because of costs. yes, I thought of asking for
money for at least one more 36G scsi disk, but I didn't want to
go through that, dealing about money again.
I lost mail of mine too, Mike, so I felt the pain as well...
|
mcnally
|
|
response 51 of 176:
|
Nov 29 19:19 UTC 2005 |
> But I did not succeed in getting the board to move on getting
> the PC Weasel because of costs. yes, I thought of asking for
> money for at least one more 36G scsi disk, but I didn't want to
> go through that, dealing about money again.
Right. Which supports my counter-argument against Jan's statement
and suggests that maybe the recent trouble points to some organizational
problems that we can remedy before the next time we reach a crisis.
|
tod
|
|
response 52 of 176:
|
Nov 29 19:23 UTC 2005 |
re #49
And how do we pay for this person?
We can pay them double what you're getting from Grex. ;)
|
steve
|
|
response 53 of 176:
|
Nov 29 19:24 UTC 2005 |
I'm not so sure. The management of Grex has always been prudent
about financial things, to keep the system healthy. Perhaps one
could say there is too much of that at times, but thats life. No
organization is perfect. Overall I think Grex does things pretty
well, and, in spite of my thoughts on things, I'd rather have
this organization than even a lot of "professional" organizations
in terms of how they work. This is not to say that we couldn't
stand improvement, just that we're less screwed up than most
business places, from my point of view.
|
steve
|
|
response 54 of 176:
|
Nov 29 20:12 UTC 2005 |
Re #40: It was *my* fault that we lost the mail partition, and mine
alone.
|
nharmon
|
|
response 55 of 176:
|
Nov 29 20:31 UTC 2005 |
Thank you for not displacing blame and taking responsibility Steve. I'm
sure we all know you didn't do it deliberately and will probably not
make the same mistake again. Thank you for being honest.
|
mcnally
|
|
response 56 of 176:
|
Nov 29 20:32 UTC 2005 |
re #54: Your error was the proximate cause, but that doesn't mean that
there weren't contributing issues that we should address in anticipation
of future mistakes -- anytime people are involved mistakes are inevitable
but through proper planning and procedures you can make a huge difference
in the outcomes..
I think you're seeing this as a discussion about blame, which is probably
not an unreasonable way to look at it from your standpoint, especially as
there are still a lot of people who want to discuss blame. I'd much rather
try to figure out how best to keep it from happening again, which requires
some degree of understanding what happened and why, but is a question to
which blame is pretty much irrelevant.
|
steve
|
|
response 57 of 176:
|
Nov 29 20:38 UTC 2005 |
No, I'm not looking at it from a blame stanpoint, just the truth.
I do however fully agree with you that we need to be able to look at
things and do some things better. Thats a good thing to do.
|
naftee
|
|
response 58 of 176:
|
Nov 29 22:04 UTC 2005 |
hey tod. how do you say 'fiduciary' in Romanian ?
|
bhoward
|
|
response 59 of 176:
|
Nov 29 22:28 UTC 2005 |
Re#50: It should be noted for the record, as of the last board
meeting, we have reconfirmed that the cost of a PC Weasel is being
covered by an anonymous donation.
Has the order been placed?
Once this and other post-mortem discussions have run their course,
I suggest we (staff) should take the points made, write up a summary
of how we'll go about the next one, making certain the process
described will address the shortcomings identified in the last one.
|
other
|
|
response 60 of 176:
|
Nov 29 23:05 UTC 2005 |
I have a suggestion.
Documentation tends to be written and then squirreled away in any of a
number of places where it may or may not ever be read or seen again.
I propose that Grex operational documentation be kept in a single file
or directory, and that the contents be tagged (XML, perhaps?) and the
relevant scripts/programs modified so that those scripts and programs,
when run, can access and echo to the screen of the calling user any
information which they should have in mind before any actions are
performed. Ideally, they would require an acknowledgement before
continuing.
The advantages are: easy updating of documentation (all in the same
location); improvement of documentation (since it would constantly be
appearing, chances are it would be written or rewritten to better
communicate important information); and less time wasted either writing
useless documentation or because of lack of documentation where and when
it was needed.
This is a fairly easy to implement suggestion, and will make it much
easier to have new staff trained in the vagaries of Grex whenever there
is new staff to train.
This may represent a significant allocation of time, but for those who
have already spent lots of time writing documentation, it shouldn't be
hard to see why this is necessary. It should be prioritized as highly
as any other staff responsibility including keeping the system running
and secure, because it will make both of those goals easier and faster.
Lastly, if anyone thinks they don't need to have stuff documented
because either "everyone knows it" or "I'm the only one who does this
and I know it," those are the persons who most need to be doing this.
|
other
|
|
response 61 of 176:
|
Nov 29 23:10 UTC 2005 |
By the way, this scheme easily allows for pointing to additional tools
and documentation to supplement the echoed information.
For that matter, both tools/scripts and documentation might be collected
in a keyword searchable database (using the same tagged source
documents) for anyone needing to know how to perform a certain function
on the system.
The more of this kind of thing that gets done, the less the system is
dependent on a few individuals with highly specialized knowledge to do
most of the things necessary to keep the system running properly.
|
tod
|
|
response 62 of 176:
|
Nov 29 23:28 UTC 2005 |
re #58
hey tod. how do you say 'fiduciary' in Romanian ?
demn de incredere
|
steve
|
|
response 63 of 176:
|
Nov 29 23:41 UTC 2005 |
We have a good start at documentation in the /grexdoc directory
and in the staff conference. Both need more work, but we do have
a good start for it.
|
aruba
|
|
response 64 of 176:
|
Nov 30 01:14 UTC 2005 |
Re #59: I haven't ordered the PC Weasel yet, but I will soon. Someone needs
to find out from Provide Net what it's going to cost us per month to have a
separate machine running, which is, as I understand it, what we will need in
order to make the PC Weasel work.
Tod said the board should be trying harder to get more staff. Well, I'm not
on the board right now, but I think I speak for them when I say, they're
open to suggestions.
|
steve
|
|
response 65 of 176:
|
Nov 30 01:39 UTC 2005 |
I'll send mail to John A again about the cost.
|