response 108 of 124:

Jan 6 08:22 UTC 2007

Hi.  I'm doing my very occasional look through Grex, and saw this.

I'm a network person, not a systems person, so I'm way over my head when
talking about specific systems components.  That said, I do manage a lot
of "critical infrastructure" type services on a low budget with a small
staff, so I spend a lot of time thinking about how to keep services
reliable while also cheap and easy to operate.  And, for those of you
who are new in the last few years and don't know me, I'm a former Grex
staff and board member.

I'm a big fan of systems where the answer to a problem is to turn off
the malfunctioning component, and the users don't notice.  For that
reason, I like that hardware RAID systems and the like are being
discussed.  For partitions with lots of dynamic data that needs to stay
up to date, like the conferences, RAID or some equivalent is absolutely
the right way to go.

I also like that this is being thought about now, at a time when I
gather Grex has been relatively stable.  "If it ain't broke, don't fix
it" and "the number one cause of network outages is network engineers"
are both appropriate rules to keep in mind, but if you've got something
that looks really likely to fall apart it is easier to fix it before it
becomes an emergency.

However, there are a few other things about this discussion that worry
me.  Doing piecemeal upgrades to several year old hardware seems like a
good way to run into unexpected incompatibilities.  Using internal RAID
enclosures with the idea of moving them to as yet unspecified new
hardware seems like a big loss of flexibility.

If I were specing this out, and if it could fit within the budget,
here's what I would probably do:

Get a networked disk array, such as the one maus talks about in #106
(rack mountable, consuming as few rack units as possible), and put all
the dynamic data on it.  Then get a couple of 1U servers, standard and
self contained, with serial consoles and ideally some sort of "lights
out manager" thing.  Put the static non-changing stuff on the internal
hard drives, and set them up as clones of each other.  Add in a cheap
Ebay-purchased console server to manage it.  If the applications support
it, run the two 1U servers side by side, accessing the same data off the
RAID array and sharing the load; otherwise, keep one as a hot spare.

If one of the disks in the RAID array fails, pull and replace it.  If
one of the servers fails, turn it off and run on the other one.

I suspect you could fit this all in six U or so of rack space, which is
still not huge.

That said, I'd also question somewhat whether Grex should still be in
the hardware business.  It might be worth looking at some of the
"dedicated server hosting" companies, and see how what they're charging
to rent a server that includes colo space, network connectivity, and
hands on hardware support, compares to what you're paying ProvideNet.

response 110 of 124:

Jan 6 13:08 UTC 2007

It has been a while since I have priced it out but if I remember
correctly, ISP-provided servers are either very expensive, or
virtualized. So I'm not sure if it would be workable.

response 112 of 124:

Jan 7 00:44 UTC 2007

resp:108

scg,

Just curious how you handle high-availability. I have looked over a few
solutions, though i have only ever implemented them in a lab setting, so
I am not sure how they would behave in the wild. I particularly like: 

 - HSRP/CARP with synchronization data sent over a dedicated seperate
 LAN - Round-robin NAT with IP-takeover (and using IPMI/LOM to "shoot
 the
other guy in the head")

I am sure there are much better solutions, and would love to learn them.

response 114 of 124:

Jan 7 02:17 UTC 2007

I need to amend my previous statement. I had forgotten about RackSpace;
they have a good rep. I don't know ServPath or Affinity. 

response 116 of 124:

Feb 14 04:21 UTC 2007

While sitting in a hollowed out baseboard and chewing on some wires, I
found an Ultra320 SCSI RAID board in what appears to be new condition.
Do we want me to send it in so that when we reinstall, we can mirror the
root volume on high-speed SCSI drives? 

response 118 of 124:

Feb 18 05:26 UTC 2007

It is pure hardware RAID. I would have to check the make and model
(documentation should be on the mfc's webpage). 

response 120 of 124:

Feb 18 07:28 UTC 2007

   Hardware raid is definitely what we want.  I will look at this.

response 122 of 124:

Mar 30 11:20 UTC 2007

Yes, 10k RPM drives have higher I/O performance than slower spinning
drives. They also tend to have a lower capacity and are more expensive.
The rule of thumb I usually use to calculate I/O performance is:

                         RPM/100 = iops

That is, RPMs divided by 100 gives you I/Os per second. Of course, I
mainly deal with fiber channel drives so this may be way off. Your
arrangement is as important as your individual disk performance too. A
RAID 10 array is much faster than a RAID 5 array, but sacrifices a lot
of storage space.

response 124 of 124:

May 5 03:51 UTC 2007

I've heard that these "perpendicular" drives at 7200 RPM are actually the
fastest for most situations.