response 108 of 124:

Jan 6 08:22 UTC 2007

Hi.  I'm doing my very occasional look through Grex, and saw this.

I'm a network person, not a systems person, so I'm way over my head when
talking about specific systems components.  That said, I do manage a lot
of "critical infrastructure" type services on a low budget with a small
staff, so I spend a lot of time thinking about how to keep services
reliable while also cheap and easy to operate.  And, for those of you
who are new in the last few years and don't know me, I'm a former Grex
staff and board member.

I'm a big fan of systems where the answer to a problem is to turn off
the malfunctioning component, and the users don't notice.  For that
reason, I like that hardware RAID systems and the like are being
discussed.  For partitions with lots of dynamic data that needs to stay
up to date, like the conferences, RAID or some equivalent is absolutely
the right way to go.

I also like that this is being thought about now, at a time when I
gather Grex has been relatively stable.  "If it ain't broke, don't fix
it" and "the number one cause of network outages is network engineers"
are both appropriate rules to keep in mind, but if you've got something
that looks really likely to fall apart it is easier to fix it before it
becomes an emergency.

However, there are a few other things about this discussion that worry
me.  Doing piecemeal upgrades to several year old hardware seems like a
good way to run into unexpected incompatibilities.  Using internal RAID
enclosures with the idea of moving them to as yet unspecified new
hardware seems like a big loss of flexibility.

If I were specing this out, and if it could fit within the budget,
here's what I would probably do:

Get a networked disk array, such as the one maus talks about in #106
(rack mountable, consuming as few rack units as possible), and put all
the dynamic data on it.  Then get a couple of 1U servers, standard and
self contained, with serial consoles and ideally some sort of "lights
out manager" thing.  Put the static non-changing stuff on the internal
hard drives, and set them up as clones of each other.  Add in a cheap
Ebay-purchased console server to manage it.  If the applications support
it, run the two 1U servers side by side, accessing the same data off the
RAID array and sharing the load; otherwise, keep one as a hot spare.

If one of the disks in the RAID array fails, pull and replace it.  If
one of the servers fails, turn it off and run on the other one.

I suspect you could fit this all in six U or so of rack space, which is
still not huge.

That said, I'd also question somewhat whether Grex should still be in
the hardware business.  It might be worth looking at some of the
"dedicated server hosting" companies, and see how what they're charging
to rent a server that includes colo space, network connectivity, and
hands on hardware support, compares to what you're paying ProvideNet.