Grex Board Meeting Minutes - June, 24, 2007
Present: S. Lynne Fremont (slynne) - Board Member, President
Jan Wolter (janc) - Board Member, Secretary - Staff
Mark Conger (aruba) - Board Member, Treasurer
Colleen McGee (cmcgee) - Board Member
Larry Kestenbaum (polygon) - Board Member
Telepresent:
Dan Cross (cross) - Board Member
Absent:
Bruce Howard (bhoward) - Board Member - Staff
Opening Gavel Tap: 7:31 pm
Some discussion of the beauty of the view, and of the walnut do-daddies.
Treasurer's Report:
Grex took in $67.21 in May and spent 152.18. To date, there has been
$198 income in June, and the usual expenses of about $150. Bank balances
is still high, near $7000.
Staff Report:
Grex had a series of crashes, three within a period of a week. Staff
has no clue why this is happening. There are no relevant error messages
appearing on the console or in the logs. The machine is not actually
crashing (which would be nice because then it would auto-reboot) but is
instead just freezing up, and going mostly unresponsive.
Mark will prod STeve Andre to see if he has any more ideas on this than
the rest of staff.
Grex is due to be upgraded to a newer version of OpenBSD, probably 4.1
stable. Jan reluctantly agreed to do the initial install. This will
probably entail several days of downtime for Grex. Mark will assist.
We hope to do this in the next few weeks.
Mark had a call from a somewhat irate user whose account had been
deleted although it hadn't been idle for 90 days. The account was
deleted in the course of a reap that Dan had done recently. He
thinks he may have messed up with some modifications he made to the
reapcheck program (the program that is used to identify candidates for
reaping). Staff and board apologize profusely.
Old Business:
Colleen has entered some items in the coop conference to try to move along
some of the topics raised in previous meetings.
New Business:
Jan is currently working on a blog interface for Backtalk. He described
how this would work. An item with similar information has since been
entered in coop.
Dan has installed the RT Helpdesk trouble ticket reporting system.
Schedule Next Meeting:
The next board meeting will be on August 5, 2007 at Mark Conger's house.
Gather at 7:00pm, gavel at 7:30pm.
Closing Gavel Tap: 8:32 pm
21 responses total.
Whenever I type up these minutes, I always think of some things I wish I'd said at the meeting. In this case, before we can do the upgrade to OpenBSD 4.1, we need to do a backup to the system. To do a backup, it would be nice if we had the DVD R/W drive whose purchase the board authorized some time ago. I think we were waiting for someone or another to recommend a model to buy, there being some sentiment that I don't understand to the effect that Grex needs a better than average DVD R/W drive. If anyone has any bright ideas about what we should buy, I think we should move forward quickly with this. I'd really like to have it installed before we do the upgrade.
Since the org has the money to do so and has considered replacing parts of the system anyway (system board, RAM (to get error checking capabilities), disc subsystem, DVD burner), and with computer hardware cheaper and more efficient than ever, would it be reasonable to build a new machine for the 4,1 load and bring it up and burn it in in parallel with the existing machine? This would minimize downtime, and would let us get the new machine that we have been yakking about for months. Just a thought.
Several days of downtime is *so* 1990s.
Re #2: sounds good to me. Re #3: Cool, I finally made it out of the 1980's !
Maybe a whole new computer would be a good idea. I'd love to see a specific proposal for what to buy. But we'd still need some way to back up the current Grex. I think buying a basic commodity DVD R/W drive and installing it in Grex would be no bad idea. I've also heard (a little) about ISPs that offer virtual machines, on which you can install whatever OS you like. That might be an interesting way for Grex to go. Get away from owning our own computer entirely. However I know next to nothing about such services and don't know if they could really meet our needs.
http://www.slicehost.com/ is an example of the sort of thing I was talking about. That particular one doesn't seem to support OpenBSD, and arranging dial in access would be a problem.
I like the idea of new hardware but I dont quite know enough about it to make a good recommendation
If you would like, I can start getting quotes from vendors. A spec that I would see as appropriate is: - Full tower or 4U rack chassis - Redundant power-supplies - 2 GBytes ECC RAM - Single Opteron or Xeon processor - 2 x 80 GByte SATA drive (RAID-mirror) for / - 2 x 500 GByte SATA drive (RAID-mirror) for /home and /var - 4-port OpenBSD-supported RAID board (prefer 3Ware Escalade series) - OpenBSD-supported Gigbit Ethernet NIC - IPMI or other BMC board (prefer one that gives remote console in addition to remote reboot and sensor data) - Minimum 2 COM ports for modems for our last few dial-in users Does this sound reasonable to people? Do we want hot-spares on our RAID sets? Do we need to split / off from the /home and /var slices onto a separate drive-pair? Thoughts? Suggestions? Also, do we have a timeline for when we would be thinking about doing this? Vendors like to know when they could expect an order if they were chosen.
Is there a way to get a rough estimate of the cost of such a system without having vendors go out of their way to provide the information. We could decide what we want and then contact vendors. As for the hardware you have suggested, cross...it seems ok to me. Would those hard drives be redundant or would the data be striped across them? I am assuming the former since you're talking 'mirror' in which case I am in favor of that configuration. if not, then we should add something for backups.
(I think i'd lean toward separate disks for /, /var and /home.)
I forgot, add to my description: - SATA DVD +/- RW drive or DLT Tape drive
re: resp #9 I am actually maus, not cross. Although we are both nerdy and both advocate for similar facilities for Grex-server, we are not in fact the same user.
Maus is right, we are not the same person. :-) Regarding #10; With RAID, things are spread across multiple physical drives, but aggregated to look like one logical disk. Redundancy provides resiliance to any single drive failure, though I see point in not just getting 4 x 500GB SATA drives. I agree whole heartedly with Maus's hardware recommendations (I recommend going with the rackmount case). I regret that I did not voice stronger opinions about hardware when we bought the current grex machine; I really wanted ECC memory (which, as it turns out, would have saved us a lot of downtime), hardware RAID (which also would have saved us a lot of downtime) and a rackmount case (which would have saved us space in the colo at provide.net). I still feel strongly that these things will be beneficial to grex.
OK, I'd prefer to see /, /var and /home on separate RAIDs. I'd also prefer three (or more) disks in each RAID, so we _could_ use striping rather than mirroring. (Actually, a three-disk RAID is probably overkill for / . IIRC, the root partition doesn't change much.)
Why on separate RAIDs, Joe? I'm curious what your rationale is.
I can see having /home and /var on a separate set from / so that if the system tanks, all of the data can be moved to a new box, rather than having to build a new RAID set and then recover from tapes (just bring the HBA/RAID-controller and the drives for the home+var over to a freshly built/formatted machine). Are we that performance-starved that we really need some form of striping and that we need to split /home and /var onto separate arrays? Joe, what are the performance problems that you are experiencing, and what are your expectations? What you have described suddenly means we need a much bigger chassis (possibly going from a tower to a pedestal) and 10 drives (since you suggest 3 per array, with 3 separate arrays, and add in a global hot-spare). There is a much greater incremental cost of going from 4 to 10 drives than there is from going from 2 to 4 -- not only would we need far more drives and a bigger chassis, but we would need a much larger and higher-class RAID-controller, would draw a lot more power, dissipate much more heat and noise and increase our statistical likelihood of a mechanical device failing. The extra space+power+heat+noise would probably piss off our hosting provider, and might make them think of raising our rates when it's time to renew our contract. If we need the additional performance of stripe-sets, separating /home from /var, then we probably need to consider the load we are using and profile the use of the machine. At that point, we would probably have a use profile better suited for multiple machines that do single tasks, rather than one huge multi-purpose machine. I may be biased toward it since it was my design, but I still like the way I specced it out -- nice and redundant, but not ridiculous.
As I said, a RAID may (OK, probably is) overkill for the root directory. I like having the root directory on a separate drive (physical AND logical) because it doesn't change much, and I don't want any changes that _do_ occur to be blocked by over-full home, tmp or log directories. Same for putting /var and /home on different (physical AND logical) drives, so one filling up doesn't affect the rest. I don't see the advantage of using a RAID just to mirror a disk.
re #17: > I like having the root directory on a separate drive (physical AND logical) > because it doesn't change much, and I don't want any changes that _do_ occur > to be blocked by over-full home, tmp or log directories. Same for putting > /var and /home on different (physical AND logical) drives, so one filling up > doesn't affect the rest. I don't understand why that requires separate drives, as opposed to just separate partitions. > I don't see the advantage of using a RAID just to mirror a disk. If a drive dies, in many cases you can continue running.
Putting /home and /var on separate filesystems will keep the filling of one from knocking over the other. Most filesystems begin life pre-sized, and cannot grow beyond that size without intervention (and in the case of UFS, I believe not at all, since it does not have an underlying volume management layer) which keeps one from stealing space from another -- it can only steal space from its own allocation. To keep /tmp from being a problem, I like to make it a presized file and use mount_vnd to mount it. This allows it to be mounted with more restrictive permissions (like noexec,nosuid) and it can be resized by creating a new, larger file, formatting that file and moving the vnode information with vnconfig and remounting. I don't see the advantage of using a RAID just to mirror a disk. I am not sure what you mean by this. Besides RAID, what other mechanisms are in place for mirroring drives?
I guess I'm not making any sense. I'll shut up now.
Ok, any action on this since the end of June?
You have several choices: