Running Grex as a VPS

Dec 2 13:05 UTC 2008

Virtualization technology seems to be pretty mature.  With software
tools such VMware or Parallels, you can create your own virtual
machine running as a "guest" on some other OS.  For example, you
could run a Linux guest on Windows Vista or a FreeBSD guest under
OS X.  On a more global scale, Amazon offers its EC2 (Elastic Compute
Cloud) service, where you can create and control your own machine
image running on Amazon's hardware.

A couple of months ago I decided to educate myself on the VPS
(Virtual Private Server) offerings that are out there, partly for
my own edification and possible personal use, and partly to see if
VPS might be a viable alternative to Grex's owning and operating
its own hardware.

I found a company called RootBSD (http://www.rootbsd.net) that
offers FreeBSD VPS's using Xen virtualization technology.  Their
virtual machine packages come in various sizes and they charge by
the month - $20/month at the low end up to $120/month at the high
end.  The high-end VPS has 2gb of memory, 120gb of disk (RAID), and
a cap of 400gb/month of network bandwidth.  That seems roughly
comparable to the resources Grex has now, at a lower price than we
pay our current provider.  You get a static IP and full root access.

To see how the service works in practice, I filled out their web
form and took out a subscription to their low-end package.  Within
an hour they had set me up with a VPS having 256mb RAM, 10gb of
disk, running a basic installation of FreeBSD 7.0, and with a
dedicated static IP address.  The normal way to access the machine
is via SSH, but they also provide a web-based control panel that
includes a virtual console that allows you to do such things as
take the machine off the network and run it in single-user mode.

I put the machine through its paces to see if there were any
limitations to what I could do software-wise as an administrator.
So far I haven't found any limitations.  One of the first things I
did was to upgrade the OS to FreeBSD-stable, which involves compiling
and installing the userland and kernel source from scratch.  This
went without a hitch.  Speed seemed pretty good for a machine with
only 256mb of memory - the "buildworld" step (compiling the userland
software) completed in under two hours with the machine running in
multiuser mode, if I recall correctly.

I've installed several dozen third-party software packages using
FreeBSD's "ports" facility, including Apache, Mysql, and Postfix.
These three services have all been running for some time now without
problems.

Other packages I've installed under FreeBSD (on other machines)
include Drupal, Mediawiki, and the Grex-specific Party and Write
programs.  Haven't tackled Backtalk/Fronttalk yet, but I doubt
they'd pose problems.

At one point I managed to break ssh access to the machine by
accidentally deleting the ssh binary.  No problem - I logged into
the RootBSD control panel, booted into single user mode, and
recompiled and installed ssh from source via the virtual console.
Sure beats driving over to ProvideNet.

Note that running Grex as a VPS would solve a couple of recurrent
problems with our current setup that has caused Grex to be down
for days on several occasions:  Dependence on physical access and
availability of local staff.  With VPS, any staff member with web
access could log into the control panel and perform operations that
currently require access to the hardware.  A remote console device 
such as PC Weasel has been considered as a way of addressing this
problem, but a VPS with a web-accessible control panel would provide
the same functionality at less expense.

My conclusion:  Grex should seriously explore the VPS route as an
alternative to owning and maintaining its own hardware.  Obviously,
there are various issues that I haven't addressed here that would
need to be looked at before taking such a step.  But from a purely
technical standpoint, it seems feasible based on my experience.

Thoughts?

response 2 of 56:

Dec 2 17:20 UTC 2008

I agree. 

response 4 of 56:

Dec 2 20:59 UTC 2008

Actually, I think the "high end" package is slightly *more* than
what we currently pay at Provide.net (we pay $100/mo for colo).

Going virtual would mean porting the OpenBSD specific parts of
Grex's software investment to FreeBSD.  I'm fine with that (I think
we should have gone with FreeBSD instead of OpenBSD to begin with,
but lost that battle), but some other folks might not be.  Steve,
for instance, is a really big fan of OpenBSD and might not want to
switch.  It would not be *nearly* as big a deal as the move from
SunOS to OpenBSD was, but it is something to bear in mind.  It also
likely means giving up our dialin lines.  Again, I'm fine with that,
but I think it's likely going to meet stiff opposition from a few
diehard users of that service.

On the other hand, these things aren't insurmountable; after all,
most of the software we use is open source, meaning we can move it
around very easily (the only exception is Picospan, which we're
going to have to bite the bullet and say goodbye to sometime over
the next year anyway).  And it may be possible to rig up something
for a dialin solution (maybe a terminal server in somebody's house
configured to host modems and automagically SSH into Grex or
something.  Then you have to factor in the risk that this hosting
company might go out of business or change its pricing structure
or something of the like.  In these economic times, that's a
non-trivial amount of risk to assume.  Not a show-stopper, but
again, something to think about.  Finally, is what Grex does, with
providing access to a lot of services to nearly complete strangers,
really something that is going to be inline with this hosting
company's AUP?  That needs to be researched and settled; we've had
some problems with unruly Grex users causing problems for Provide,
and I can imagine that the problem would be worse in a virtual
context where our workload is competing for resources with those
of other paying customers.

The reason I'm guessing that Remmers believes this would be cheaper
is because, as I understand it, our *total* outlay per month is
about $150.  That includes $100 for colocation and another $50 or
so for the two phone lines.  If we went with this hosted solution,
and kept dialin access, costs would increase to around $170.00 plus
electricity and hosting for the modems and a terminal server.  I'm
guessing we'd end up spending around $200/mo, when you factor in
the cost of getting and maintaining a terminal server locally.

There are certain advantages with going with a virtualized solution,
but I'm not sure the problems we've had with Grex in the past are
among them.  The hardware problems we've had have largely worked
themselves out, and now that we've figured out the dance with calling
provide and getting them to reboot the machine if it crashes,
physical access is much less of a problem.  Before, we were much
more hesitant to do that since we thought we might be able to get
good information from the console about *why* Grex was down.  Almost
universally, we didn't do that, or what information we *did* get
was just the soft of thing that enabled us to say, "yup, there's a
bug in OpenBSD.  It's known and they're working on it."  It wasn't
very *useful* information, so now we've more or less given up on
that and just say, "screw it; reboot it and if there's anything
useful it'll be in the logs or the crash dump anyway."  Further,
recent security problems with virtualization servers make me wary
of wanting to host a service like Grex somewhere where other
organizations can have root on the same *physical* hardware.

On the other hand, it's getting about time that we got a new machine.
That's going to cost about $2500 or so.  That's equivalent to the
extra money we'd spend over a little over four years of virtual
hosting, and that's about how long we could expect to have that
machine before it was time to upgrade again.  So I'd guess that
financially, we'd about break even.

I do think there's a place for virtualization on Grex, but it has
a lot more to do with partitioning services and seemless upgrades
than cutting cost or boosting reliability.  For instance, we may
want to push email *processing* to a different "host", and it would
make upgrades really, really nice if we could "clone" Grex, upgrade
the clone, *test it*, and then "shift" the references to user data
directories in a virtual machine and bring up a new, fully tested,
upgraded Grex with a single reboot.  But that is *not* what I gather
this service is offering.

response 6 of 56:

Dec 3 18:11 UTC 2008

I wonder if it might be possible to get temporary access to a 2nd
virtual machine at a reduced cost for purposes of testing an upgrade? 
Maybe Grex could buy access to a 2nd virtual machine permanently,
perhaps with limited access and a reduced cost, so the staff would have
a sandbox for testing.  This surely is not a unique need for Grex.

I regard a move away from physical hardware to be a beneficial one for
Grex.  A virtual machine will never wear out or become outdated.  We're
a virtual community anyway and don't need physical hardware for any
purpose as far as the community is concerned.  Is a professionally
administered virtual machine really going to be less secure than what we
have now?

I wonder if we couldn't buy access to an inbound dial-up ISP account and
distribute it for use by anyone who wants to use it for less than we
currently spend on a phone line.  I suppose that would run us the risk
of someone making it their own permanently.  It seems to me there ought
to be some solution like that which is much cheaper for us than
maintaining a terminal server with Internet access and a couple of phone
lines running to it.

response 8 of 56:

Dec 3 18:27 UTC 2008

And "administer" Grex using cpanel... argh...

response 10 of 56:

Dec 3 19:45 UTC 2008

A "virtual machine" is _still_ a machine. 

I would be leery of sharing a processor. 

as cross indicated in resp:4 
>we've had some problems with unruly Grex users causing problems for 
>Provide, and I can imagine that the problem would be worse in a 
>virtual context where our workload is competing for resources with 
>those of other paying customers.

Likewise system bogging by our 'virtual neighbors' could bring GREX to 
a crawl in spite of all efforts to restrict foul play. All it takes is 
a badly written SQL query to lock up a shared CPU then somebody will 
still have to call somebody for a reboot.

Imagine the thrill of getting root on the mothership - the cracking of 
that 'virtual devide' - tell me that ain't going to be happening. ;-)

response 12 of 56:

Dec 4 03:45 UTC 2008

resp:5 Did you read resp:4?

resp:6 Maybe, but that needs to be investigated.  My suspicion is not; this
is the new version of what services like rackspace used to do; sell you a
machine in colo, the hardware maintained by them, but you run whatever you
want on it.  It's a specification of "the cloud."  And it's a nice idea, as
long as your provider doesn't go out of business.

I think the idea of getting ISP access for our users isn't a bad one.  That
would be pretty interesting, and might save us some dough.

resp:10 Part of the problem with virtualization is that bugs have recently
been discovered in both Xen and VMware (and presumably other virtualization
packages) that allow root on a virtual host to leverage its way to root on
the physical host.  From there, all the other virtual machines are toast. 
That's a lot of risk to assume.

resp:11 Darn skippy!  I doubt mostof these hosting services are ready for
what Grex does.