response 1 of 31:

Feb 16 02:27 UTC 2007

I don't remember the details of STeve's proposal.  Let's hope he (or glenda)
has a moment to stop by.  

response 3 of 31:

Feb 18 05:32 UTC 2007

I think I can agree with that, although I am not familiar with Arco, and
so cannot vouch for them. How many ports does the Arco SATA board have?
Could we do RAID 1 + 0 or RAID 1 + concatenating and have hot spares? I
still like the 3ware boards, and they are known to work natively with
OBSD (the drivers are in the default kernel and the entire LUN shows up
as a single SCSI drive). 

response 5 of 31:

Feb 18 15:15 UTC 2007

Whatever solution we go with, OpenBSD *must* support it natively in the
current, standard distribution.  I'd rather have something we know works
than something that potentially the best board ever that isn't supported by
our software; this mistake was made when we upgrade grex to the present
hardware (the Ethernet chip on the motherboard had support problems, and
there were rumors of problems with the SCSI controller) and I don't want to
see a repeat.

response 7 of 31:

Feb 19 01:38 UTC 2007

   Huh?  The ethernet card had zero problems; it always performed wonderfully.
The Broadcom ethernet chip was a little new at the time, but was more slow
than anything else.
   As far as support goes for raid cards, the hardware only solution is
perfect.  The arco card works great with IDE.  I have a SATA Arco card
which I'm setting up next week but know of a couple happily supporting
OpenBSD now.

response 9 of 31:

Feb 19 05:07 UTC 2007

It seems that this is becoming critical.

Steve, I can't find any information about support for Arco cards on the
OpenBSD/i386 hardware listing for OpenBSD 4.0.  However, the 3ware cards that
Maus mentioned before are listed as supported; what are the advantages you
see of the Arco cards over the 3ware gear?

response 11 of 31:

Feb 19 19:52 UTC 2007

   Nick Holland has written about them in misc@.  Thats how I first got
wind of them.  I know the pata IDE works great as I've been beating on
my new fileserver for a while now with no problems.  Once I make a new
web server I can take the main one offline and stuff the 500G Arco sata
array there, and test that.

   I thought the Arco stuff was talked about somewhere, perhaps the faq.
If it isn't, I should write something up and submit it, because the card
I've been using is great.

response 13 of 31:

Feb 19 20:23 UTC 2007

Perhaps a good idea would be to set out some requirements for what we need
here in a RAID storage solution.  I see the following as being inviolable
requirements:

0) RAID 1 (mirroring; RAID 1+0 or 0+1 would be nice, but not necessary)
1) Will fit a standard rackmount enclosure.
2) Supported by OpenBSD in the current, standard, stable distribution.
   -current support or partial support is not good enough.  As of right
   now, that means OpenBSD 4.0 supports it out of the box.
3) Support for hot swapable disks, hopefully via front-panel replacement.
4) Automated rebuilds of the mirror after disk replacement.

From what I read on the OpenBSD lists, the Arco gear doesn't support some
of these features.  That would make it a non-starter in my opinion.

http://monkey.org/openbsd/archive/misc/0309/msg01787.html

response 15 of 31:

Feb 19 20:56 UTC 2007

   Well, we're going to get into a philosophical discussion here, but
I see the absolute #1 thing for the card to do is mirror well, and
be reliable.  One of the reasons why I got the Acro card for work was
the fact that I have friends who deal with Windows that have been
using these cards without incident over a few years now.  Having 
hot swap abilities or auto rebuild kind of scare me.  You'll call me
a technoligcal luddite for my saying this, but I like the KISS
principal (keep it simple, stupid) that to the extent that I'll
forgo extras if the unit basically does what it needs to do, namely
deal with data duplication.  I know the history of several of them,
which is why I made that choice.  The rebuild rate on the pata ide
card is about 53G/hour, which is about as good as the accusys sata
version. How fast the sata Arco will be I can't yet say, but I don't
think it will be any slower.  The Arco card isn't able to keep up
at the disks 3Gb/sec rate, but is faster than pata ide by a large
amount.
   I'm willing to optimize for simple operation that is reliable.
This is not to say that the accusys cards aren't great--I just don't
know.  I do know the Acro units.

   As for your requirements, we agree on 0) since it does raid 1.
The sata Arco device I have next to me would fit into a 3.5in bay,
so it should fit into rackmounts.  2) is done, in that it is pure
hardware--so OpenBSD anything.anything should work.  This is part
of the reason why I like this kind of card--it has no idea of 
whats upstairs, doing thigs. It just thinks in terms of sectors.
3) I think you're talking more money there.   For how often I 
see changing disks out, I'm willing to delve into the system.
Yeah, it would be nice to pop a disk out but I'm also thinking
of costs.  4) Again, I see a rebuild as a rare enough thing that
I'm willing to take some downtime.  I've had really bad experiences
with raid systems that were supposed to be able to rebuild, and
did, mostly.  As I said, I might be a luddite on this, but I'm
pretty conservative about disk data.

   However, we're in a planning stage for this, so I'm all ears
for other ideas on what we could buy.

response 17 of 31:

Feb 19 22:18 UTC 2007

   So what would you propose?  And, think of OpenBSD 4.1--it will be
out in 10 weeks, officially.  I'll have a 4.1 src tree the day that
4.1-current comes out, so we could move to it even faster.

   So we're into the philosophical now--I can't regard anything that
does auto-rebuilding as "simple".  I fear that people might think
that staff time won't be needed as much with things like raid
technology, which I've found not to be true.

   Part of this touches upon new hardware in general.  I know you've
said that our getting the Antec box was a mistake.  I specifically
chose that after talking with many people about reliability, 
especially with the power supply.  Given the budget we had, and
heat problems that others had with cheaper rackmount units, I
felt that the case was the better option.  As well as paying 
attention to the kiss idea, I really really like lots of air 
blowing on stuff and we have four fans.  The inside of that
case is pretty cool.  I don't think we need to change boxes
currently.  It isn't old, and the part that Grex has killed
many times over (disk) is the part that we're talking of
changing.

response 19 of 31:

Feb 20 16:43 UTC 2007

Auto-rebuild, hot-spares and hot-pluggable discs make life much easier.
Labour time has value, and if we are wasting the labour time of very
knowledgeable, experienced people, then we are effectively wasting
money. If we neglect to spend either the money up front or the time in a
crisis, we are not serving our users (and maybe being fiduciary fuckups
for our paying members). 

Think of it this way, what would be easier: 
 0) Getting a page that says a drive failed, pulling the drive while the
machine is running, slapping a new one (already in a hot-plug tray) into
the cage and walking away
 1) Getting a page that says a drive failed, posting to the HVCN
webpage, shutting down the system nicely, un-racking and opening the
system's chassis, unscrewing the six tiny screws that hold a drive in
place, replace the drive, screwing the new drive in, closing and
re-racking the system chassis, booting to the firmware of the RAID
controller, starting a rebuild by hand, booting to single-user
maintenance mode, running fsck and then booting to multi-user mode

Which of those could you squeeze in on your lunch break or on the way
home from work? I know which one I prefer, but what would a small rodent
know besides chewing cables and eating cheese? 

response 21 of 31:

Feb 20 17:31 UTC 2007

I think we should focus our research into SATA RAID controllers to those
on this list:

http://www.openbsd.org/i386.html#hardware

I was about to recommend an Adaptec controller based on the tremendous
experience we've had with them. But they aren't very well supported in
OpenBSD it would seem.

Also, I don't see any Arco/Dupldisk cards on that list.

response 23 of 31:

Feb 20 23:31 UTC 2007

I'm not willing to be cavalier about it either, Steve.

But as a board member, I'm also not willing to bet our collective farm on
staff's ability to drop whatever they're doing in their non-grex lives on a
dime and go take care of a sick disk, either.  I'm also not willing to say,
``well, then grex might be down for a week or two.''  There's just no reason
to do that.  We've had downtimes around a week due to dead disks.  Actually,
scratch that: because a disk died and no one was available to go swap it out
and rebuild it.  If we can automate that, then let's do it.

There really truly are RAID systems in our price range that can handle
automatic rebuilds just fine; we don't need a $100,000 EMC disk array.  I
think we need to move to a more automated solution.  Otherwise, we might as
well just buy another SCSI disk and stick with what we have, since the
downtime and effects would be comparable (except, perhaps, for lost data).

Hot spares, hot swapping, and online rebuilds.  These things aren't rocket
science, or even state of the art anymore; they're also affordable.