Virtualization technology seems to be pretty mature.  With software
tools such VMware or Parallels, you can create your own virtual
machine running as a "guest" on some other OS.  For example, you
could run a Linux guest on Windows Vista or a FreeBSD guest under
OS X.  On a more global scale, Amazon offers its EC2 (Elastic Compute
Cloud) service, where you can create and control your own machine
image running on Amazon's hardware.

A couple of months ago I decided to educate myself on the VPS
(Virtual Private Server) offerings that are out there, partly for
my own edification and possible personal use, and partly to see if
VPS might be a viable alternative to Grex's owning and operating
its own hardware.

I found a company called RootBSD (http://www.rootbsd.net) that
offers FreeBSD VPS's using Xen virtualization technology.  Their
virtual machine packages come in various sizes and they charge by
the month - $20/month at the low end up to $120/month at the high
end.  The high-end VPS has 2gb of memory, 120gb of disk (RAID), and
a cap of 400gb/month of network bandwidth.  That seems roughly
comparable to the resources Grex has now, at a lower price than we
pay our current provider.  You get a static IP and full root access.

To see how the service works in practice, I filled out their web
form and took out a subscription to their low-end package.  Within
an hour they had set me up with a VPS having 256mb RAM, 10gb of
disk, running a basic installation of FreeBSD 7.0, and with a
dedicated static IP address.  The normal way to access the machine
is via SSH, but they also provide a web-based control panel that
includes a virtual console that allows you to do such things as
take the machine off the network and run it in single-user mode.

I put the machine through its paces to see if there were any
limitations to what I could do software-wise as an administrator.
So far I haven't found any limitations.  One of the first things I
did was to upgrade the OS to FreeBSD-stable, which involves compiling
and installing the userland and kernel source from scratch.  This
went without a hitch.  Speed seemed pretty good for a machine with
only 256mb of memory - the "buildworld" step (compiling the userland
software) completed in under two hours with the machine running in
multiuser mode, if I recall correctly.

I've installed several dozen third-party software packages using
FreeBSD's "ports" facility, including Apache, Mysql, and Postfix.
These three services have all been running for some time now without
problems.

Other packages I've installed under FreeBSD (on other machines)
include Drupal, Mediawiki, and the Grex-specific Party and Write
programs.  Haven't tackled Backtalk/Fronttalk yet, but I doubt
they'd pose problems.

At one point I managed to break ssh access to the machine by
accidentally deleting the ssh binary.  No problem - I logged into
the RootBSD control panel, booted into single user mode, and
recompiled and installed ssh from source via the virtual console.
Sure beats driving over to ProvideNet.

Note that running Grex as a VPS would solve a couple of recurrent
problems with our current setup that has caused Grex to be down
for days on several occasions:  Dependence on physical access and
availability of local staff.  With VPS, any staff member with web
access could log into the control panel and perform operations that
currently require access to the hardware.  A remote console device 
such as PC Weasel has been considered as a way of addressing this
problem, but a VPS with a web-accessible control panel would provide
the same functionality at less expense.

My conclusion:  Grex should seriously explore the VPS route as an
alternative to owning and maintaining its own hardware.  Obviously,
there are various issues that I haven't addressed here that would
need to be looked at before taking such a step.  But from a purely
technical standpoint, it seems feasible based on my experience.

Thoughts?

56 responses total.

This seems like a real winner in terms of supportability for a staff
with limited resources and which is not centrally located in Ann Arbor
any more.

I agree. 

Very interesting stuff remmers.
 
The Ghost of Grex?
 
  I have always found the tales of woe and perils of hardware trouble a 
certain percentage of why I visit. Perhaps it is the hardware hound in 
me that enjoyed the thought of poking around the circuits housed in 
the 'pumpkin'. The move to provide.net has taken away much of that 
romance as it is. 

    The move to a virtual grex would mean that it is all about the 
content (for me - I'm saying.) While it would obviously be a boon to 
the staff, in my opinion (for what it's worth) is it would make 
cyberspace.org just another forum - and frankly not one of the more 
interesting ones. I would still poke around tho. Speaking as someone 
who has not contributed cash to the cause I agree the savings on 
overhead IS attractive. (If I ever get a buncha money I will be sure to 
give some to GREX in any case.)

IDK - it rings of doing ham radio via the Internet. I came to GREX when 
my local dialup BBS went 'Hollywood' my best advice would be not to 
tell anyone if you did move it to the 'cloud' and stage the drama. 

Sometimes a drive over to ProvideNet is just what the doctor ordered. 
Particularly to the grex die-hards - an excuse to get out and get some 
fresh air. 

Actually, I think the "high end" package is slightly *more* than
what we currently pay at Provide.net (we pay $100/mo for colo).

Going virtual would mean porting the OpenBSD specific parts of
Grex's software investment to FreeBSD.  I'm fine with that (I think
we should have gone with FreeBSD instead of OpenBSD to begin with,
but lost that battle), but some other folks might not be.  Steve,
for instance, is a really big fan of OpenBSD and might not want to
switch.  It would not be *nearly* as big a deal as the move from
SunOS to OpenBSD was, but it is something to bear in mind.  It also
likely means giving up our dialin lines.  Again, I'm fine with that,
but I think it's likely going to meet stiff opposition from a few
diehard users of that service.

On the other hand, these things aren't insurmountable; after all,
most of the software we use is open source, meaning we can move it
around very easily (the only exception is Picospan, which we're
going to have to bite the bullet and say goodbye to sometime over
the next year anyway).  And it may be possible to rig up something
for a dialin solution (maybe a terminal server in somebody's house
configured to host modems and automagically SSH into Grex or
something.  Then you have to factor in the risk that this hosting
company might go out of business or change its pricing structure
or something of the like.  In these economic times, that's a
non-trivial amount of risk to assume.  Not a show-stopper, but
again, something to think about.  Finally, is what Grex does, with
providing access to a lot of services to nearly complete strangers,
really something that is going to be inline with this hosting
company's AUP?  That needs to be researched and settled; we've had
some problems with unruly Grex users causing problems for Provide,
and I can imagine that the problem would be worse in a virtual
context where our workload is competing for resources with those
of other paying customers.

The reason I'm guessing that Remmers believes this would be cheaper
is because, as I understand it, our *total* outlay per month is
about $150.  That includes $100 for colocation and another $50 or
so for the two phone lines.  If we went with this hosted solution,
and kept dialin access, costs would increase to around $170.00 plus
electricity and hosting for the modems and a terminal server.  I'm
guessing we'd end up spending around $200/mo, when you factor in
the cost of getting and maintaining a terminal server locally.

There are certain advantages with going with a virtualized solution,
but I'm not sure the problems we've had with Grex in the past are
among them.  The hardware problems we've had have largely worked
themselves out, and now that we've figured out the dance with calling
provide and getting them to reboot the machine if it crashes,
physical access is much less of a problem.  Before, we were much
more hesitant to do that since we thought we might be able to get
good information from the console about *why* Grex was down.  Almost
universally, we didn't do that, or what information we *did* get
was just the soft of thing that enabled us to say, "yup, there's a
bug in OpenBSD.  It's known and they're working on it."  It wasn't
very *useful* information, so now we've more or less given up on
that and just say, "screw it; reboot it and if there's anything
useful it'll be in the logs or the crash dump anyway."  Further,
recent security problems with virtualization servers make me wary
of wanting to host a service like Grex somewhere where other
organizations can have root on the same *physical* hardware.

On the other hand, it's getting about time that we got a new machine.
That's going to cost about $2500 or so.  That's equivalent to the
extra money we'd spend over a little over four years of virtual
hosting, and that's about how long we could expect to have that
machine before it was time to upgrade again.  So I'd guess that
financially, we'd about break even.

I do think there's a place for virtualization on Grex, but it has
a lot more to do with partitioning services and seemless upgrades
than cutting cost or boosting reliability.  For instance, we may
want to push email *processing* to a different "host", and it would
make upgrades really, really nice if we could "clone" Grex, upgrade
the clone, *test it*, and then "shift" the references to user data
directories in a virtual machine and bring up a new, fully tested,
upgraded Grex with a single reboot.  But that is *not* what I gather
this service is offering.

 I recommended something similar a while back and still think it's
 a good idea.  This would solve one of the big issues that has lead
 to several multi-day Grex outages -- limited staff access to the
 Grex hardware and the complications that causes for crash recovery.

I wonder if it might be possible to get temporary access to a 2nd
virtual machine at a reduced cost for purposes of testing an upgrade? 
Maybe Grex could buy access to a 2nd virtual machine permanently,
perhaps with limited access and a reduced cost, so the staff would have
a sandbox for testing.  This surely is not a unique need for Grex.

I regard a move away from physical hardware to be a beneficial one for
Grex.  A virtual machine will never wear out or become outdated.  We're
a virtual community anyway and don't need physical hardware for any
purpose as far as the community is concerned.  Is a professionally
administered virtual machine really going to be less secure than what we
have now?

I wonder if we couldn't buy access to an inbound dial-up ISP account and
distribute it for use by anyone who wants to use it for less than we
currently spend on a phone line.  I suppose that would run us the risk
of someone making it their own permanently.  It seems to me there ought
to be some solution like that which is much cheaper for us than
maintaining a terminal server with Internet access and a couple of phone
lines running to it.

Run the forums on phpBB... ick...

And "administer" Grex using cpanel... argh...

What becomes of Grex-damashii, then?

A "virtual machine" is _still_ a machine. 

I would be leery of sharing a processor. 

as cross indicated in resp:4 
>we've had some problems with unruly Grex users causing problems for 
>Provide, and I can imagine that the problem would be worse in a 
>virtual context where our workload is competing for resources with 
>those of other paying customers.

Likewise system bogging by our 'virtual neighbors' could bring GREX to 
a crawl in spite of all efforts to restrict foul play. All it takes is 
a badly written SQL query to lock up a shared CPU then somebody will 
still have to call somebody for a reboot.

Imagine the thrill of getting root on the mothership - the cracking of 
that 'virtual devide' - tell me that ain't going to be happening. ;-)

It seems to me that a more likely problem would be that vandals on Grex
screw up things for our neighbors on a virtual machine. It would suck to
get everything all set up only to have a provider decided we aren t
worth the bother and kick us off. 

resp:5 Did you read resp:4?

resp:6 Maybe, but that needs to be investigated.  My suspicion is not; this
is the new version of what services like rackspace used to do; sell you a
machine in colo, the hardware maintained by them, but you run whatever you
want on it.  It's a specification of "the cloud."  And it's a nice idea, as
long as your provider doesn't go out of business.

I think the idea of getting ISP access for our users isn't a bad one.  That
would be pretty interesting, and might save us some dough.

resp:10 Part of the problem with virtualization is that bugs have recently
been discovered in both Xen and VMware (and presumably other virtualization
packages) that allow root on a virtual host to leverage its way to root on
the physical host.  From there, all the other virtual machines are toast. 
That's a lot of risk to assume.

resp:11 Darn skippy!  I doubt mostof these hosting services are ready for
what Grex does.

hmmmm, interesersting remmzz ... how woudl they react to teh two
signarure factors of grex.
 
    oper newuser
  
    unix experimentation?
  
remmz... is it possile for your site to (ummm) 'test' the  newuwer
adn compilerer optons that we offer here? ... just fo e  kikx-n-grims?
  

It is nice to see that cloud offerings have matured enough to make this
viable. 
Is there any greater liability for the operation (such as oper newuser
and experimentation) when you are "embedded" as it were into their
hardware? 

Yes.

Re resp:14 - Hi Ken!  Long time no see!!

Re the OS - I'd be fine with switching from OpenBSD to FreeBSD.
It's something to consider, even if Grex continues to run on its
own hardware.  I have the impression that FreeBSD has a much larger
installed base and user community than OpenBSD, an important consider-
ation when it comes to support.  Porting Grex software would
undoubtedly be possible and would probably be pretty easy.  Users
wouldn't notice much difference.  Hey, M-Net runs on FreeBSD.  And I
suspect that the security advantages of OpenBSD are largely illusory,
at least for a service like Grex.  Historically, can't most of our
security issues be traced to the fact that we've been running an 
open newuser with immediate Unix shell access?

Re resp:7, resp:8, and resp:9 - Bellstar, I'm not advocating
anything of the sort.  The user interface under a FreeBSD VPS would be the same
 as it is now.

Regarding going the VPS route - It still seems to me that making the
hardware somebody else's problem, not our problem, would be a big win
for us.  I think Dan's right that the service I've been playing
around with (rootbsd.net) doesn't offer the kind of flexibility
we might want (e.g. cloning).  On the other hand, something like
Amazon's EC2 service does - you can create and delete machine images 
by typing a few commands, shift an IP address from one virtual
machine to another, etc.  

On the other other hand, EC2 doesn't offer FreeBSD yet, although I 
believe it's being worked on.

Security issues with Xen and other virtualization technologies - well,
I'll have to plead ignorance there.  That issue would have to be  
looked at carefully, of course.  Any references you can cite on
the recently discovered bugs that allow root breakins on the 
physical host (re resp:12)?

In looking around, I've found tons of VPS hosting services that offer
Linux, rather few that offer FreeBSD, none that offer OpenBSD.  The
rootbsd.net service that I've been using is the only one I've found
that offers FreeBSD at a price that Grex could afford, although this
situation may change in the future.

After playing around with FreeBSD for a few months I must say I
like it a lot as a server environment, so I'd be delighted to see
Grex run it too.  But given that Linux seems to more widely supported,
I'll raise another question:  Would there be advantages to running
Grex on some Linux flavor (e.g. Ubuntu or CentOS)?

I'm not opposed to changing, but it's a matter of having someone
do the work.  That and convincing all the interested parties that
it's worth doing.  As I said, neither of those are impossible tasks,
but they are *tasks*.  If we wanted to switch, they would have to
be done.  Of course, if we upgrade our current hardware, that would
have to be done as well.  So really, nothing's free.

My own personal opinion is that right now, the VPS route isn't worth
it.  We'd just about break even financially, lose some flexibility,
and hopefully gain some reliability.  It would require a non-trivial
amount of effort to port the software and switch operating systems,
migrate the data, etc.  Dialin would be an issue (and one which no
one has addressed yet).  Staff time is an issue (I just transitioned
off of active duty and am in the process of transitioning back to
my day job; my time is severely constricted.  I think most of staff
is in the same boat).  The reliability isn't an issue so much right
now, regardless.

The virtual hosting thing should be something that we keep a close
eye on moving forward, but my sense is that we need to wait at least
a couple of years for (a) dialin to finally be a non-issue, and (b)
for the offerings in the space to mature a bit.  My strong suspicion
is that we should do at least one more iteration of Grex on our own
hardware, this time not penny-pinching about things like hardware
RAID.

I argued strongly for FreeBSD on the current Grex hardware, but
lost that fight.  Marcus and Steve really, really wanted OpenBSD
and no one else much cared either way.  We can revisit it now, if
folks want to; there would be advantages to moving, and some
disadavantages (mainly involving the effort required to switch),
but is that a good use of limited resources at the moment?  I'm
asking here; I really don't know.  My sense is that there are more
pressing things (one of which is getting some more staff blood
involved with Grex).

Well, I've already ported party and Jan's write/tel to FreeBSD
(actually, the latter is an official FreeBSD port) and have started
looking into installing Backtalk/Fronttalk as well.  As mentioned in #0,
I've also installed and configured Apache, MySQL, PHP, and Postfix on my
FreeBSD system.  They all work pretty well too! :)

So I'm getting some familiarity with FreeBSD system administration and
would certainly be willing to pitch in and help should folks decide to
go the FreeBSD route.

I was under the impression you could just map the serial port for the modem
to the VM. 
Here is a link to using serial ports with VMware.. not sure if it is at all

helpful, but... i thought i would throw it in
http://www.vmware.com/support/ws55/doc/ws_devices_serial.html

resp:18 Are you saying you'd like to rejoin staff?  'cause we could use the
help.  But quite a bit has changed since you left.

resp:19 No, the issue here is where to physically put the modems.  In the
case of what remmers is proposing, the entire machine would be virtualized; we
wouldn't even really know where the hardware was.  And if that's the case, then
where do you put the modems?

ummmmmmm, can we users login with newuser?  url pse?

re #20 - thanks.
 
 What if we virtualized grex and m-net on the same hardware? like a two in
one type system? just kinda merge them together, but keeping them seperate?

Ok, i'll shut up now.

I'm not opposed to doing that.

oh, wow ... the word 'merge' is anathema in extremis.
  
not even teh det freepress & det newds is as bad a thought. 
  
but a thought nonetherlwess..   
  
and it;s not an hostility thing .. it's a2 cultural.
  

A JOA between M-Net and Grex?

There would be little reason at that point not to simply merge the two
systems.  Both have Boards which do little and need to do little; a
shortage of staff members; and nearly identical conferencing systems
with mostly the same users.  There is one staff member in common with
the two, who is cross.  In a lot of ways it is already hard to tell
which system you're on.  There could be a general and an agora
conference, with a few specialty conferences shared by both.

The difference between running the two organizations on one computer,
and running one organization, is minimal.

I do think we are heading in that direction.  I've thought so for
several years.  I think there are a few who would be vehemently opposed,
a few who would quit using whichever system they use now and go away
entirely, and the rest (a pretty small number) would go along with it.

Eh, diversity is good.  But virtualizing the hardware and running both systems
on the same physical box might not be a bad idea.

I dunno, when it's the same group of users, I don't see that there's
much more diversity from two systems than from one.  The world has a lot
of diversity in discussion forums, anyway.  Grex and M-Net have very
little influence on that, whether each or both continue to exist.

Re resp:25 - "The difference between running the two organizations on
one computer, and running one organization, is minimal."

Wrong, if each runs in its own VPS.

But the question of a Grex/Arbornet merger is material for a different
item.  For now I'll just say that I agree with the first sentence of
Dan's resp:26.

Re an earlier question:  No, not interested in returning to staff at
this point, at least not the day-to-day-keeping-things-running variety
of staff work.  Did it for 15 years; that's enough for the forseeable
future, for me.  I'd consider lending a hand for projects of limited
duration where I feel I could be of help, such as migrating Grex to
FreeBSD, if folks should decide they want to go in that direction.

Fair enough.

you can forget about any grex/arbornet merger

I wasn't proposing one.  I was forecasting that that could happen someday.

having the wequivant of 'off site backup'  grex<==>m-b0x creates a
more reliable 'place to go' - not too likely tat both b0xes, in different
placers, would be broken at the same time.

are the modems really needed anymore?

Too cool.  Jared stopped by.

Welcome to Grex!

resp:33 Needed or wanted....

Getting back to the idea of running Grex as a VPS...  It could have
avoided the recent multi-day outage.

Could it have?  What was the recent multi-day outage all about?

- Backtalk version 1.3.30 - Copyright 1996-2006, Jan Wolter and Steve Weiss