Grex Board Meeting Minutes - June, 24, 2007

  Present:  S. Lynne Fremont (slynne) - Board Member, President
            Jan Wolter (janc) - Board Member, Secretary - Staff
            Mark Conger (aruba) - Board Member, Treasurer
            Colleen McGee (cmcgee) - Board Member
            Larry Kestenbaum (polygon) - Board Member
  Telepresent:
            Dan Cross (cross) - Board Member
  Absent:
            Bruce Howard (bhoward) - Board Member - Staff

Opening Gavel Tap:  7:31 pm

  Some discussion of the beauty of the view, and of the walnut do-daddies.

Treasurer's Report:

  Grex took in $67.21 in May and spent 152.18.  To date, there has been
  $198 income in June, and the usual expenses of about $150.  Bank balances
  is still high, near $7000.

Staff Report:

  Grex had a series of crashes, three within a period of a week.  Staff
  has no clue why this is happening.  There are no relevant error messages
  appearing on the console or in the logs.  The machine is not actually
  crashing (which would be nice because then it would auto-reboot) but is
  instead just freezing up, and going mostly unresponsive.

  Mark will prod STeve Andre to see if he has any more ideas on this than
  the rest of staff.

  Grex is due to be upgraded to a newer version of OpenBSD, probably 4.1
  stable.  Jan reluctantly agreed to do the initial install.  This will
  probably entail several days of downtime for Grex.  Mark will assist.
  We hope to do this in the next few weeks.

  Mark had a call from a somewhat irate user whose account had been
  deleted although it hadn't been idle for 90 days.  The account was
  deleted in the course of a reap that Dan had done recently.  He
  thinks he may have messed up with some modifications he made to the
  reapcheck program (the program that is used to identify candidates for
  reaping).  Staff and board apologize profusely.

Old Business:

  Colleen has entered some items in the coop conference to try to move along
  some of the topics raised in previous meetings.

New Business:

  Jan is currently working on a blog interface for Backtalk.  He described
  how this would work.  An item with similar information has since been
  entered in coop.

  Dan has installed the RT Helpdesk trouble ticket reporting system.

Schedule Next Meeting:

  The next board meeting will be on August 5, 2007 at Mark Conger's house.
  Gather at 7:00pm, gavel at 7:30pm.

Closing Gavel Tap: 8:32 pm

21 responses total.

Whenever I type up these minutes, I always think of some things I wish I'd
said at the meeting.

In this case, before we can do the upgrade to OpenBSD 4.1, we need to do a
backup to the system.  To do a backup, it would be nice if we had the DVD R/W
drive whose purchase the board authorized some time ago.  I think we were
waiting for someone or another to recommend a model to buy, there being some
sentiment that I don't understand to the effect that Grex needs a better than
average DVD R/W drive.  If anyone has any bright ideas about what we should
buy, I think we should move forward quickly with this.  I'd really like to
have it installed before we do the upgrade.

Since the org has the money to do so and has considered replacing parts
of the system anyway (system board, RAM (to get error checking
capabilities), disc subsystem, DVD burner), and with computer hardware
cheaper and more efficient than ever, would it be reasonable to build a
new machine for the 4,1 load and bring it up and burn it in in parallel
with the existing machine? This would minimize downtime, and would let
us get the new machine that we have been yakking about for months. 

Just a thought. 

Several days of downtime is *so* 1990s.

Re #2:  sounds good to me.

Re #3:  Cool, I finally made it out of the 1980's !

Maybe a whole new computer would be a good idea.  I'd love to see a
specific proposal for what to buy.

But we'd still need some way to back up the current Grex.  I think
buying a basic commodity DVD R/W drive and installing it in Grex would
be no bad idea.

I've also heard (a little) about ISPs that offer virtual machines, on
which you can install whatever OS you like.  That might be an
interesting way for Grex to go.  Get away from owning our own computer
entirely.  However I know next to nothing about such services and don't
know if they could really meet our needs.

http://www.slicehost.com/ is an example of the sort of thing I was
talking about.  That particular one doesn't seem to support OpenBSD, and
arranging dial in access would be a problem.

I like the idea of new hardware but I dont quite know enough about it to
make a good recommendation

If you would like, I can start getting quotes from vendors. A spec that
I would see as appropriate is: 

- Full tower or 4U rack chassis
- Redundant power-supplies
- 2 GBytes ECC RAM
- Single Opteron or Xeon processor
- 2 x 80 GByte SATA drive (RAID-mirror) for / 
- 2 x 500 GByte SATA drive (RAID-mirror) for /home and /var
- 4-port OpenBSD-supported RAID board (prefer 3Ware Escalade series)
- OpenBSD-supported Gigbit Ethernet NIC
- IPMI or other BMC board (prefer one that gives remote console in
addition to remote reboot and sensor data)
- Minimum 2 COM ports for modems for our last few dial-in users

Does this sound reasonable to people? Do we want hot-spares on our RAID
sets? Do we need to split / off from the /home and /var slices onto a
separate drive-pair? 

Thoughts? Suggestions? 

Also, do we have a timeline for when we would be thinking about doing
this? Vendors like to know when they could expect an order if they were
chosen. 

Is there a way to get a rough estimate of the cost of such a system
without having vendors go out of their way to provide the information.
We could decide what we want and then contact vendors. 

As for the hardware you have suggested, cross...it seems ok to me. Would
those hard drives be redundant or would the data be striped across them?
I am assuming the former since you're talking 'mirror' in which case I
am in favor of that configuration. if not, then we should add something
for backups. 

(I think i'd lean toward separate disks for /, /var and /home.)

I forgot, add to my description: 
 - SATA DVD +/- RW drive or DLT Tape drive

re: resp #9

I am actually maus, not cross. Although we are both nerdy and both
advocate for similar facilities for Grex-server, we are not in fact the
same user. 

Maus is right, we are not the same person.  :-)

Regarding #10; With RAID, things are spread across multiple physical drives,
but aggregated to look like one logical disk.  Redundancy provides
resiliance to any single drive failure, though I see point in not just
getting 4 x 500GB SATA drives.

I agree whole heartedly with Maus's hardware recommendations (I recommend
going with the rackmount case).  I regret that I did not voice stronger
opinions about hardware when we bought the current grex machine; I really
wanted ECC memory (which, as it turns out, would have saved us a lot of
downtime), hardware RAID (which also would have saved us a lot of downtime)
and a rackmount case (which would have saved us space in the colo at
provide.net).  I still feel strongly that these things will be beneficial
to grex.

OK, I'd prefer to see /, /var and /home on separate RAIDs.  I'd also prefer
three (or more) disks in each RAID, so we _could_ use striping rather than
mirroring.  (Actually, a three-disk RAID is probably overkill for / . IIRC,
the root partition doesn't change much.)

Why on separate RAIDs, Joe?  I'm curious what your rationale is.

I can see having /home and /var on a separate set from / so that if the
system tanks, all of the data can be moved to a new box, rather than
having to build a new RAID set and then recover from tapes (just bring
the HBA/RAID-controller and the drives for the home+var over to a
freshly built/formatted machine). Are we that performance-starved that
we really need some form of striping and that we need to split /home and
/var onto separate arrays? 

Joe, what are the performance problems that you are experiencing, and
what are your expectations? What you have described suddenly means we
need a much bigger chassis (possibly going from a tower to a pedestal)
and 10 drives (since you suggest 3 per array, with 3 separate arrays,
and add in a global hot-spare). There is a much greater incremental cost
of going from 4 to 10 drives than there is from going from 2 to 4 -- not
only would we need far more drives and a bigger chassis, but we would
need a much larger and higher-class RAID-controller, would draw a lot
more power, dissipate much more heat and noise and increase our
statistical likelihood of a mechanical device failing. The extra
space+power+heat+noise would probably piss off our hosting provider, and
might make them think of raising our rates when it's time to renew our
contract. 

If we need the additional performance of stripe-sets, separating /home
from /var, then we probably need to consider the load we are using and
profile the use of the machine. At that point, we would probably have a
use profile better suited for multiple machines that do single tasks,
rather than one huge multi-purpose machine. 

I may be biased toward it since it was my design, but I still like the
way I specced it out -- nice and redundant, but not ridiculous. 

As I said, a RAID may (OK, probably is) overkill for the root directory.
I like having the root directory on a separate drive (physical AND logical)
because it doesn't change much, and I don't want any changes that _do_ occur
to be blocked by over-full home, tmp or log directories.  Same for putting
/var and /home on different (physical AND logical) drives, so one filling up
doesn't affect the rest.

I don't see the advantage of using a RAID just to mirror a disk.

re #17:
> I like having the root directory on a separate drive (physical AND logical)
> because it doesn't change much, and I don't want any changes that _do_ occur
> to be blocked by over-full home, tmp or log directories.  Same for putting
> /var and /home on different (physical AND logical) drives, so one filling up
> doesn't affect the rest.

  I don't understand why that requires separate drives, as opposed to just
  separate partitions.

> I don't see the advantage of using a RAID just to mirror a disk.

  If a drive dies, in many cases you can continue running.

Putting /home and /var on separate filesystems will keep the filling of one from knocking over the other. Most filesystems begin life pre-sized, and cannot grow beyond that size without intervention (and in the case of UFS, I believe not at all, since it does not have an underlying volume management layer) which keeps one from stealing space from another -- it can only steal space from its own allocation. To keep /tmp from being a problem, I like to make it a presized file and use mount_vnd to mount it. This allows it to be mounted with more restrictive permissions (like noexec,nosuid) and it can be resized by creating a new, larger file, formatting that file and moving the vnode information with vnconfig and remounting.

I don't see the advantage of using a RAID just to mirror a disk.

I am not sure what you mean by this. Besides RAID, what other mechanisms are in place for mirroring drives?

I guess I'm not making any sense.  I'll shut up now.

Ok, any action on this since the end of June?

Response not possible - You must register and login before posting.

- Backtalk version 1.3.30 - Copyright 1996-2006, Jan Wolter and Steve Weiss