response 335 of 547:

May 21 23:48 UTC 2003

Are softupdates turned on on the raid filesystem?

response 337 of 547:

May 22 03:29 UTC 2003

Every write and every read is also a metadata update (mtime and atime).
Soft updates are definitely stable at this point; they're enabled by
default in FreeBSD.  OpenBSD tends to be somewhat more conservative,
though.

Gads; security be damned.  Grex would've been better off with FreeBSD.

response 339 of 547:

May 22 06:30 UTC 2003

For the heck of it, I ran eight copies of the Bonnie benchmark simultaneously
on the RAID 5 partition.  Below, A through H were started simultaneously.
The last line is just one benchmark process running

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
A        2047  8087  5.9  5867  1.2   331  0.1  1647  2.0  1775  0.4  22.5 0.1
B        2047  1889  1.4  7770  1.4   545  0.2   890  0.9  1646  0.3  14.2 0.1
C        2047  1020  0.8  7038  1.2   417  0.1  1929  2.7  1578  0.3  19.9 0.2
D        2047  8647  6.3  7474  1.3   253  0.1  1905  2.4  4597  1.1  89.4 0.8
E        2047  3997  2.9  6946  1.2   215  0.1 23458 27.9 29250  6.4 155.5 1.4
F        2047  8314  6.2  7149  1.3   369  0.1  1333  1.6  1707  0.3  21.2 0.1
G        2047  8926  6.3  7899  1.4   512  0.2   865  1.1  1132  0.3  15.0 0.1
H        2047  4280  3.2  7861  1.3   458  0.1   954  1.2  1649  0.4  19.1 0.1

raid 5   2000  9520  6.8  7974  1.3  5706  2.0 50932 62.5 63815 13.0 147.9 1.6

They didn't stay well synchronized - you can tell that process E continued
running long after the others had finished (process scheduling doesn't seem
to be very fair).  The write speeds didn't suffer too badly from the
competition, but the read times took a terrific beating - they are mostly
around 1/25 of the speed of one process.  Note that there were probably some
write processes still running while the read processes were going.

Here's a more sensible test, a comparison against the SCSI and IDE drives,
in non-RAID configuration, with just one process running:

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
scsi     2000 53754 43.4 54106 14.1 10090  2.6 60326 70.9 61067 11.5 201.2 0.8
ide      2000 27188 21.7 27038  6.9  9634  2.6 24889 29.9 25640  5.2  99.0 0.8

Seems the SCSI is about twice as fast on most benchmarks, and about the same
on the Rewrite test.

response 341 of 547:

May 22 13:10 UTC 2003

RAID would be nice, and if we're making such a huge jump in processing power
then I don't think the performance penalty (assuming it's only 2-1 or
something less) isn't an issue.

response 343 of 547:

May 22 13:59 UTC 2003

Jan - can you fool the OS into thinking Grex has less memory than it really
does?  Or tell it not to cache disk reads?

response 345 of 547:

May 22 14:33 UTC 2003

Re 343:  probably - but I'm not sure how.  I thought of just creating a
RAMDISK and letting that eat up much of the memory (I could also run the
benchmark on a ramdisk, which might be interesting), but it looks like
you need to do a lot of kernal work to bring up a ramdisk, and I'm
insufficiently motivated.

response 347 of 547:

May 22 14:48 UTC 2003

Hmmm...the faq (http://www.openbsd.org/faq/faq11.html) talks about the
BUFCACHEPERCENT kernel value.  It says the default is 5%.  I haven't touched
it, so if I'm reading this right, there should be 75M or less of disk cache.
Hmmm...Linux uses all free memory as disk cache.  A much nicer setup.

Well, if that's the case then I'm not sure what make those benchmark numbers
so goofy.

response 349 of 547:

May 22 18:20 UTC 2003

So am I; it seems unreasonably slow, and it looks vaguely like the
numbers start to converge when you have many processes working at
once, which is the normal mode of operation.  I'd be interested in
seeing what a test simulating a timesharing load would be like.

response 351 of 547:

May 23 13:49 UTC 2003

I'm reluctant to take the machine appart for such purposes.  Anyway, I'm a
software guy.

I certainly agree that we need better benchmarks, but I'm not sure how to
obtain them.  Anyone with better ideas is welcome to suggest them.  Those of
you with accounts on the system can probably run them yourselves, as the
relevant disk partitions are permitted 777.  We really want to get some sense
of how RAID would effect a realistic multi-user load.

I tried running a benchmark with a really small file, one where you should be
getting lots of use from cache.  Here's the 50 MB and 2000 MB results.  Explain
this, if you will:

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
raid 5     50 24882 20.9 22641  3.5  3731  1.2  7555  9.6 64346 14.7 511.7 3.1
raid 5   2000  9520  6.8  7974  1.3  5706  2.0 50932 62.5 63815 13.0 147.9 1.6

The small run has much faster output, and significantly faster seek times.
The block read is about as fast as the large file (suggesting that it is
mostly reading from buffer).  But what's going on with the per char read?

Note that the sequence of the tests is:

    Per Char Output
    Rewrite Output
    Block Output
    Per Char Input
    Block Input
    Seek

So it may be that the Per Char read was from disk, but left the entire file
in cache, so the block read was then very fast.  But why wouldn't it already
be in cache after the block output?  And why would the same speed be
achieved on the Block Read with the 2M file, which can't have all been
in cache.

I don't think I know enough about how buffering and disk I/O works in openBSD
to really interpret this stuff.

response 353 of 547:

May 23 15:00 UTC 2003

Maybe to help more people figure out what is being discussed here,
I should give a brief over view of RAID.

RAID stands for "Redundant Array of Inexpensive Disks" (the I-word
varies).  Some wrote a paper once upon a time survey various options for
putting a lot of small disks together, and named the variations RAID 1,
RAID 2, RAID 3, RAID 4, and RAID 5.  The RAID 0 name was coined later and
isn't really RAID.  The interesting ones are RAID 0, RAID 1 and RAID 5.
I'll also discuss RAID 4 because understanding it makes RAID 5 easier
to understand.

Suppose you needed a 100 Gig disk, and all you had was ten 10 Gig disks.
Well, you could put them all together in a box, and write a little
controller that would write the first 10 Gig to disk one, the next 10
Gig to disk two and so on.  To the computer, your box would look like
a single disk.

The performance of this disk array wouldn't be so hot though.  Most
programs access file sequentially, so as the 100 Gig file was read,
we'd first have disk one very busy, while the other nine sit idle,then
disk two would be busy, and so forth.  It'd be nice to balance the load
among the disks.

Which brings us to RAID 0 - also known as striping.  We slice the disks
into 32K chunks.  As you write a big file to the disk, the first 32K
goes to disk one, the second 32K to disk two, on through the tenth
32K chunk going to disk ten.  That completes a stripe.  The eleventh
32K chunk goes to disk one again.  This balances the load over all ten
disks, so you get better performance.  You can vary the chunk size for
different applications.

So RAID 0 gets you a large virtual disk and balances load over your
drives.  It doesn't give you any increase in reliability.  Quite the
contrary.  If a drive dies, than instead of losing a 10Gig hunk of data,
you lose lots of 32K hunks of data scattered through all your data.
This is probably harder to restore.

Load balancing over multiple spindles would be nice for Grex, but not
vital.  We don't have just a single process reading the disk sequentially.
Increasing the difficulty of reconstructing the file system after a
disk crash is too high a cost to pay for slightly better load balancing.
I think we can rule RAID 0 out as an option.

There is no Reduncancy in RAID 0 (so it should be called "AID 0").
Real RAID starts with RAID 1 - also called "mirroring".  We are still
trying to make a virtual disk out of many real disks.  This time we'll
group our ten 10Gig disks into five pairs, disk 1A, 1B, 2A, 2B, etc.
Whenever we write data to disk 1A, we also write a copy of the same data
to the corresponding location on disk 1B.  The first obvious effect is
that our virtual disk only contains 50 Gig instead of 100 Gig.  But now,
if disk 1B dies, we have an up-to-the-nano-second backup copy.  We can
replace the disk 1B with a new disk, copy the contents of disk 1A onto
it, and be back up and running with no loss of data.

Ideally, in RAID 1, we'd do the writes to the two disks simultaneously,
so writing is no slower than reading.  (In software implementations of
RAID 1, this may not entirely work.)  On reads, we don't have to read
from both disk.  We just select the one that is less busy at the moment
and read from that.  So, we get decent performance and the capability
to survive a single drive failure, but at the cost of half our disk space.

I've heard of RAID 0+1, but not read much about it.  I assume it's just 
striping over the 5 pairs of mirrored disks in the example above.

RAID 4 is an attempt to get the same benefits as RAID 1, but with less
loss of disk space.  This time we call 9 of our disks "data disks" and
the other one a "parity disk".  Parity just means "even" or "odd".  The
129th bit stored on the parity disk depends on the values of the 129th
bit stored on the other nine drives.  If an odd number of those nine bits
are 1's, then a 1 is stored at that location on the parity disk.  If an
even number of them ar 1's then a 0 is stored at that location on the parity
disk.  In geek terms, the content of the parity disk is just a bit-wise
exclusive-OR of the contents of all the other drives.

Suppose a drive dies.  If it was the parity drive, we can just recompute its
value from the other drives.  But what if a data drive dies?  Well, we have
all the other drives and the parity drives.  So for each bit we have something
like:

   data1 data2 data3 data4 data5 data6 data7 data8 data9 parity
     1    0     1     X     0     1     0     0     1      1

The parity bit is 1, so we originally had an odd number of 1's on the
data disk.  There are 4 ones on the surviving drives, so the bit on the
dead drive must have been 1.  (In fact the dead drives contents are just
the bit-wise exclusive-OR of all the surviving data and parity drives, so
the reconstruction process for a dead data drive is identical to the 
reconstruction process for a dead parity drive).

So, this is cool.  We now have a virtual drive holding 90Gig of data, so
we've lost only 10% of our storage, and we can still reconstruct all the
data on any lost drive.

There are some additional performance costs though.  The first problem is
the parity drive.  Every time you write data to a drive, you have to update
the data on the parity drive.  So though data writing is split over nine
drives, parity writing is all on one drive, so that drive is nine times as
busy as the other drives.  It becomes a performance bottle neck.

The solution to this problem is RAID 5 - stripe the parity data over all the
drives.  For example, the parity data for the first 32K of all the drives
would be on drive 1, the parity for the second 32K of all the drives would
be on drive 2, and so on.  So there is no one parity drive and parity is
spread over all disks.  (Note that disk reconstruction doesn't change -
you still just exclusive-OR all the other drives to reconstruct get the
lost drive.)

There is a second performance hit in RAID 4 and 5 though.  Like RAID 1, every
write is to two drives - data to one drive and parity to another.  However,
before we can write the parity, we have to compute the parity, and that means
we need to read the corresponding data from the other eight data drives.  So
a simple write turns into 8 reads and 2 writes.

Also, in RAID 1, we were able to improve read performance by always reading
the data from the less busy drive of the two that had the data.  In RAID 4
and 5, the data is only one one drive, so we can only read it from that drive.
However, we like to assume the striping in RAID 5 will balance the load among
the drives pretty well anyway.

There are lots of hardware RAID devices that optimize this kind of thing, but
we can't afford them.  The option we are considering is software RAID, which
is implemented in the OpenBSD kernel by a program called RAIDframe.  It's
pretty solid and rather nice.  You can set up a RAID array, possibly with
spare drives.  If a drive fails, and there is a spare on-line, it will
automatically bring the spare on line, reconstruct the lost data and proceed
without interuption of service.  If there are no spares, it'll run with a
drive short (in RAID 5, any read from the dead drive is simulated by reading
from all the others and exclusive-Oring them).  This is all terrific if you
need a server up 24x7, which Grex doesn't really.

Note that the redundancy in RAID gives you some protection against single
disk failures (it's assumed that you do something before the second disk
dies).  It does not replace a backup.  If you accidentally delete the wrong
file, or a vandal breaks in and alters all your files, the RAID will give
you nice redundant copies of the altered files, not the original ones.
So RAID is not a subtitute for backups.  It's protection against hardware
failure and that's all.

RAID 0 can give you some performance enhancements by load balancing.  The
other versions of RAID are all likely to be slower than a non-RAID setup,
especially if implemented in software.  RAID 0 doesn't cost you any disk
space.  The other versions are going to eat up some of your disk space.
In our case, since we have 3 drives, RAID 1 doesn't quite work and RAID 5
would eat up 1/3 of our disk space.

response 355 of 547:

May 23 15:43 UTC 2003

Where I work, we use rsync to keep a mirror of about 50 gigs worth of
data.  We're doing it across the Internet, via a T1, as well.  It does
cause a fair amount of disk thrashing on both ends when it figures out
what files need to be transferred (very much like doing a 'find' across
the filesystem) but overall it seems very efficient.  It's worked well
for us.  My guess is the "expense" of doing an rsync to another local
disk a couple times a day is going to be pretty low, especially since
you're not transferring over a network and so won't need to involve ssh
or compression.

response 357 of 547:

May 23 16:02 UTC 2003

Right.  I installed rsync from the ports tree (I like the ports tree).

I then went to /sd0 (the test partition on the first scsi disk) and did
  time rsync -ax /usr .
This should copy the whole /usr partition from the IDE disk to the SCSI
disk (which is backwards from the direction we would be going) and give
me some statistics.  The /usr partition contains 664,632K of data.  The
result from 'time' was;

   12.0u 24.5s 4:28.34 13.6% 0+0k 161046+660947io 36pf+0w

So it took 4.5 minutes elapsed time, eating 13.6% of an otherwise idle CPU.
I than reran it.  In this case it should be checking the two copies against
each other, and copying over only what changed (little or nothing).  The
time result was:

  3.8u 3.5s 0:46.13 16.1% 0+0k 47000+1454io 1pf+0w

This took 45 seconds.

In real life we'd want the --delete option on the command, so files that
don't exist on the source are removed from the copy, but I didn't do it in
the test because I was paranoid about getting the arguments backward.  Even
so, we'd want our target partitions rather larger than the source partitions.
Maybe just one big target partition instead of separate ones corresponding
to the different source partitions, the whole thing readable only by root
and possibly unmounted when it isn't being updated.

Doing this a couple times a day seems a much lower impact way to data
reduncancy than RAID.

It'd be tempting to keep two copies of some partitions, and update them on
alternate days.  Dunno if that's necessary.

This is not a substitute for real backups to tape, of course.