Agenda: Grex Board of Directors Meeting on Wed, Sept 14, 2005
1. arrivals 7:30p
2. Opening Gavel Tap 8:00p
3. Treasurer's Report
4. Staff Report
a. hardware needs
b. adding new staff
5. Old Business
- Open New User
7. Schedule Next Meeting
8. New Business
-consider hiring staff to repair grex during down times.
9. Closing Gavel Tap
The board meeting will be held upstairs at Zingerman's Next Door,
Detroit St, Ann Arbor. Anyone who needs directions can call me on my
cell phone (734)754-3773
33 responses total.
Are we really starting the meeting at 8:00? I thought it was 7:30. (Mary yawns)
I guess I couldnt remember. Let's start at 7:30
Hello: I love Grex, and what it offers became an indispensable set of tools that I produce here in the San Francisco Bay Area. Recent lengthy outages made me miss these tools that I rely so much on. I hope on Wednesday's meeting these outages would be discussed and ways to deal with them expeditiously found. Here in SF Bay Area, we are able to get new Linux-based computers as low as $179. I am thinking perhaps a backup computer can be built, and when the main computer goes down the backup computer put into service until the problem is fixed. If you give me the specs, I can also look around here to see if there are any free surplus used servers are available from the Silicon Valley firms. Ahmet Toprak
i <3 GreX
"consider hiring staff to repair grex during down times." how could grex possibly afford that? even if it was affordable, how many people are even available to be paid that know unix anymore?
Ann Arbor alone probably has dozens, if not hundreds of people with professional-level knowledge of Unix System Administration. However, it's not as easy as just hiring someone who knows Unix really well. As for how Grex can afford it -- what do you think Grex will be able to afford if it continues to go down without warning for a week at a time?
This response has been erased.
re #6 years back, in the olden days, grex used to go down routinely. If it was up for 24 hours straight it was cause for celebration. The down time never killed grex. When its up, people come back. Also grex's dues are classified as donations, so I don't think (not sure) that a member could request a partial refund of his dues because of excessive down time. Nor does it say anywhere in newuser, or it shouldn't, that grex in anyway guarantees the reliability of its service. If they could do that, if grex would feel compelled to honor such refund requests, then you might consider paying extra money to get grex up faster
Certainly, reliable uptimes should be a goal of the system administrators. But by no means should it be guranteed to the users.
That's right, Grex is not a fee-for-service organization. Still, we're going to lose a lot of users if we keep going down like this.
My thought was that if we get into a state where we are down and it looks like we'll be down for a while because staff is busy with paying jobs, that what's left is to at least be able to ask staff to help us out on a for pay basis. If I needed money and had to choose between spending time where I'd make money or instead spending that time in a volunteer position, I'd be working for the paycheck. That's sane. Grex can't expect our volunteer staff to do otherwise. Now, I'm not sure that even if we agreed to have this emergency option available that anyone on our staff would be able to bump Grex's problems up to a higher priority, or if we could afford the time they'd need to devote. But I do know one thing, that paying staff in an emergeny wouldn't stop anyone from donating any time that they otherwise would have been able to volunteer. It's not going to happen. Not with the people I know. This would be an emergency intervention. We couldn't afford to pay for what our volunteers do routinely. But, in my opinion, if we have to choose between Grex withering and dying offline, with a bank account, or being penniless, but online where we can ask for support, I'd go with the latter, and sweat the outcome.
The problem I see with hiring emergency expertise is figuring out a balance between several factors: 1) The amount of help Grex can afford Does Grex sign a blank check and then hope someone can pay the bill? At a couple of hundred dollars per hour, Grex's cash on hand would be quickly consumed. Then what? Grex could borrow, or run up a credit balance, but not honestly, given that it's income is declining steadly. 2) Making sure the help which is hired is actually helpful You kind of need a technical expert who knows Grex to be in charge. If such a person is available, chances are pretty good there's no need to hire an outsider. Other than that, you need someone who's dedicated and savvy and who isn't going to charge Grex for 3 hour coffee breaks. And who isn't going to take Grex's money, say they'll do X and Y and Z, then wander off to another project which they think is higher priority. 3) Determining when, and how, the emergency help is to be brought in Who presses the panic button, and when do they press it? How do they decide to do it? What if they're wrong? I don't see a way to hire an expert, unless we have a staffer or other insider for whom being paid would make a difference in whether they can help out Grex. Have we had a situation like that? If so, is there any harm in asking for specifics? (For a hypothetical example: janc could have been available September 6 if Grex could give him $500, but instead had to stick with his paying work.) Or more generally, what reason do we have to believe anyone could ever be available for Grex when Grex is encountering a critical situation?
I think it's mostly technology problems. Grex cut some corners and made some decisions that `made people happy' but were misguided at best. Now they're paying the price for it.
Could be Mr Cross. Those corners - can you ellaborate on them, please?
We cut out some jugs and some hooters. And some racks.
Well, ECC memory was billed as one of the biggest reasons *not* to get an Intel machine, even though you can get them with ECC memory. That didn't happen when we actually bought the machine, though. Not getting a SCSI RAID controller was also, I think, short sighted.
What can we do, Dan, to rectify these problems?
Oh, beats me. I'd say take some of the money and buy a new machine and install FreeBSD on it. I don't suspect I'm supported in that by anyone else, though.
Cross, I might support you if you could explain why you think FreeBSD is more secure or reliable than OpenBSD. I hode both in high regard.
I haven't seen any evidence that the problems grex are having are hardware-related. Am I blind?
There's a lot of custom software on top of the OS.
grex seems to be having a lot more problems since it moved to the co- lo. Was grex better off living by itself?
Not financially
When we moved to co-lo we also moved to a new machine. And you maybe have forgotten how incredibly slow the old machine was, and how it crashed all the time due to (what was thought) to be bad wiring at the pumpkin? Our biggest problem at the moment isn't hardware, or software, or colo. It's staff being available to help. Any problem requiring staff care and feeding would be bringing us to our knees.
Regarding #19; I think FreeBSD would be more reliable since (a) it has better hardware support, (b) it has more users, and thus a higher chance of getting bugs flushed out earlier on, and (c) has better people working on it (Kirk McKusick, for instance). In general, it seems more supported than OpenBSD ever will: more large sites (like Yahoo) run off of FreeBSD than OpenBSD. There's a reason for that. If OpenBSD was so much better, even with the attitudes in the industry, I think you'd see more precense. With regard to security, FreeBSD can't be much worse than OpenBSD (we've had some pretty major holes here on Grex; not all of which have been corrected). For instance, pick an unused tty, cat it, and wait for someone to telnet into it. As far as I can tell, OpenBSD 3.5 just shipped that way. Certainly, I've seen no proof to the contrary, and I can't find anywhere that anyone configured it to do that. But, FreeBSD also has its own source auditing project, and they follow OpenBSD's security fixes, and you'd be hard pressed to find many security bugs in FreeBSD that aren't also in OpenBSD or that don't affect Grex. Grex's problem isn't a lack of staff, it's that grex allowed a small minority of staffers to dictate what software it would run without really exploring the consequences of that, and with a firm belief that they were right without anything to *really* back that up. Those staffers more or less left and haven't been terribly active lately (except for one in one time of major crisis, who did some partial work and then left it unfinished for over a month, and another who wrote a helpful email the other day but otherwise hasn't done anything in, I think, over a year). In retrospect, OpenBSD wasn't a good choice. Things that should have worked in OpenBSD 3.5 just didn't (e.g., soft metadata updates in the filesystem; these work fine on FreeBSD). However, I don't think me or anyone else saying that is going to change anything: the perennial responses to pointing out problems with OpenBSD are either, ``well, we just didn't do that right...'' or, ``Wait until the next release! That's all fixed now!'' At what point, like wth the Bush administration, do epople just get fed up and say, ``no, it probably won't be fixed next release, so let's do something we have a pretty good idea will work?'' Another data point with respect to reliability.... Mnet gets much more usage than grex now, and runs an older version of FreeBSD on less `server-class' hardware, and seems to have far fewer reliability problems. The direction I'd like to see grex take is the following: Invest in a new machine with dual AMD processors, 2GB ECC memory, a *hardware* RAID controller going onto SATA disks (you really don't need SCSI) and a rack-mount case. Install FreeBSD on it, and go from there, including an audit and re-write of all the locally installed software. I don't see it happening, though.
On what platform is OpenBSD primarily developed on? Perhaps (assuming there are no grievous hardware faults) we could exchange current hardware for a platform which OpenBSD has better support for? Second-hand systems are two a penny on eBay.
It seems to be a fundamental step backwards to go from zippy, current, modern hardware that's not supported well by software we chose nearly abritrarily to flakey old hardware that may be marginally better supported. Instead, why not get the well-supported software on the newer hardware?
This response has been erased.
It might not be too expensive if Grex bought a rack-mountable ATX case they could transfer the present hardware into, and then perhaps use a seperate SATA drive array. The biggest costs will be buying a SCSI controller with hardware RAID, and the SATA drive array itself. I wonder what our ISPs costs are dependant on how many U's of rack space we use. Another option is to perform regular backups to DVD. This would just require the cost of a DVD-RW drive. A staffer could perform a full system backup bimonthly to DVD, and then put in a CD or DVD for differential backups each week. This would also be an effective disaster recovery measure in case our ISP is involved in a disaster.
Second-hand doesn't necessarily mean old. Current Grex hardware was already one or two years old by the time NewGREX was up.
True.
nharmon #29 .. prophetic. redundant, but prophetic. the echo is amazing and ignored just the same. to this day.
TROGG IS DAVID BLAINE
You have several choices: