Grex: It's time to switch operating systems.

Nov 29 20:48 UTC 2010

So after fighting the OpenBSD package system for the last week or so, 
I'm about to give up.  The stuff just doesn't work.  OpenBSD sucks.

Background: When Grex was founded, and for the first few years of its 
existence, it ran on Sun hardware using Sun's operating system, 
SunOS.  At the time, SunOS was a derivitive of the Berkeley 
distribution of Unix (Sun later switched to AT&T's System V Release 4, 
which is more commonly known as "Solaris" or "Solaris 2", but formally 
known as SunOS version 5 [SunOS 1 through SunOS 4.x were the BSD 
derived versions; actually, the earliest might have been 7th Edition 
derived, but that's neither here nor there and really getting into the 
weeds on something that's only tagentially related].  Confusingly, as 
a marketing tactic, they later retroactively named the older, BSD-
derived SunOS Solaris 1).  The hardware originally used Motorola's 
68000-derived family of processors, but eventually switched to the 
RISC SPARC processor.

Around 2002 or 2003, Grex's hardware was starting to break down, the 
BSD-derived SunOS had been EOL'd by Sun, and the price/performance 
point for Sun hardware was really being eclipsed by x86 hardware.  
Further, there were several free, very stable and mature operating 
systems available for x86 hardware.  It became clear that it was time 
to at least upgrade; after a lively round of debate, it was decided to 
use an x86-compatible AMD processor, new SCSI disks, etc.  Then the 
debate around operating systems started in earnest, with the field 
eventually narrowed to two reasonable candidates: OpenBSD and 
FreeBSD.  Of these, FreeBSD has a much larger userbase, much better 
third party software support, and is generally more advanced.  OpenBSD 
focuses more on software security and correctness, with many 
developers actively auditing the source base looking for (and often 
finding and fixing) security problems before they become real 
problems.  Of note, and relevant to this discussion, is that FreeBSD 
does the same thing; also, something that's come up recently but that 
was not, and is not now a consideration, is that M-Net runs FreeBSD.

Marcus and Steve both argued passionately for OpenBSD on security 
grounds, I argued for FreeBSD on technical grounds, everyone else was 
ambivalent.  Ultimately, the decision was made to go with OpenBSD.  
Hardware was purchased, and then ... nothing happened for a year.  The 
hardware sat in somebody's house with no one really looking at it.  
Finally, Jan Wolter took the initiative to get Grex up and running on 
the new system; he developed a less than favorable impression of 
OpenBSD along the way (see some of the old archives in the garage 
conference).  Joe Gelinas and I did some additional work to get things 
going, someone (probably Jan) moved the data over from the Sun and put 
the new machine into production in December of 2004; it's been running 
ever since, modulo some new hardware to replace failed components 
(particularly hard discs) and upgrades of the operating system.

Since sometime in 2007, I've been what you might call the primary 
staff guy, in the sense that I've done most of the upgrades and a lot 
of work on the software, the web site, etc.  In that time, I've formed 
the opinion that OpenBSD was a mistake, and that we really, truly 
would be better off with FreeBSD.  I want to bring this back up for 
discussion now.

Some basic technical points:

a) Grex's current hardware is getting long in the tooth and needs to 
be replaced at some point.  Modern computers are based around 
multicore processors; OpenBSD doesn't support these particularly well; 
certainly not as well as FreeBSD.

b) I do not believe that OpenBSD is any more secure than FreeBSD.  In 
fact, we've seen instances of OpenBSD security holes *on Grex* that 
didn't appear in FreeBSD.  Many of the flaws that Chad and Mickeyd 
used to crash Grex continuously don't appear in FreeBSD: despite 
trying to do the same things on M-Net, that system remained stable.

c) Many virtual hosting providers support FreeBSD, but relatively few 
support OpenBSD.  Grex has probably got one more round on its own 
hardware before going virtual and living in the cloud.  It would 
behoove us to transition to a software stack that's going to make that 
transition as painless as possible.

d) FreeBSD has a LOT more third party software support.  Things that 
could be a big draw for potential users exist in the FreeBSD ports 
tree, but not OpenBSD; sure, those things *may* eventually get ported 
to OpenBSD, but why wait?

I can go into more depth, but I think it's time to get serious about 
switching.

response 2 of 29:

Nov 29 23:08 UTC 2010

I think most of us are going to make, or go along with, any decisions on
the basis of non-technical information.  I don't know enough about the
differences between OpenBSD and FreeBSD to make any difference.  Neither
do most of the users, or most of the Board.

I would be willing to bet there are two or three people involved in
running Grex who should decide such things, and the rest of us will go
along.  I'm interested in details like whether the slime who hack Grex
and bring it down can be stopped, not how they can be stopped.  If there
are cool capabilities to be gained, I would be interested in what those
are.  Can Grex be opened up again so people can log in and use it right
away, the way I did when it first opened to the public?  I want it open
again, and I want it as soon as possible.

How long is it going to take to transition to FreeBSD, once the decision
has been made to make that move?  I mean in weeks on my calendar, not
hours of staff time.  If you can do it in a week, great. Start this
Sunday and we'll see you in a week.  If not... then how long?  Are you
going to do it yourself, Dan?  If not, do you have the help you need?

How long until Backtalk will be available?  It's what I use here, so
it's what I care about.

Also, is there any cost?  How much?  Are there users who will pay for
it?  As I said elsewhere, I can chip in some.

Tell me things like that.

response 4 of 29:

Nov 30 15:08 UTC 2010

resp:2 Backtalk, etc, would be available immediately (that is, FreeBSD 
wouldn't be put into production until the software had been ported and 
tested).  I don't know how long it would take, but total down time 
would be minimal (like, a day or two max).  That is, most of the work 
would proceed in parallel with keeping the current Grex running.  
There would be no cost.

response 6 of 29:

Nov 30 15:23 UTC 2010

ZFS is another great point.

response 8 of 29:

Dec 1 19:59 UTC 2010

  
also, from  sys.cf ther is this:
  
Item 100: Linus Torvalds on OpenBSD
Entered by John H. Remmers (remmers) on Thu, Jul 17, 2008 (10:18):


 Ran across this today at
 http://article.gmane.org/gmane.linux.kernel/706950
  

response 10 of 29:

Dec 2 05:46 UTC 2010

  
re 9 ... greate about findeirng the prob & fix!! now, it shold never appear
again ????  pkg_add was too new comparred to taht old depenendency for it to
have been considered ?? just asking.  
  

response 12 of 29:

Dec 2 09:34 UTC 2010

Here's another interesting point about OpenBSD: they will categorize
security problems that affect their system, but weren't discovered
by them, as "reliability" problems.  For instance, the recent OpenSSL
vulnerability (for which the FreeBSD project released a security
advisory) was listed as a reliability problem by OpenBSD.

Well, I guess their project can continue to claim that they've had
some artifically low number of security holes in "a heck of a long
time" if they just don't call security holes security holes.

At the last staff meeting back in March, Steve made mention of a
bug in FreeBSD's ftpd, citing that as a reason we should stick with
OpenBSD.  Unfortunately, that *exact same bug* affected OpenBSD and
existed on Grex.  Again, the OpenBSD project marked this as a
"reliability fix."  Despite their supposedly superior auditing,
they didn't catch either of these problems.

So this is another way of saying that I don't buy OpenBSD's security
claims.  I'm not saying they don't do a good job, but so does everyone
else.  In a lot of ways, it appears they do a better job, but that's
partly self-selection on their part: when your definition of what a
"security hole" is is so tightly focused, it's not that hard to make
it look like you're light years ahead of everybody else, but are you
really?  I say no.  And in that case, the main rationale for why we've
stayed on OpenBSD for so long is, I think, demonstrably false.

response 14 of 29:

Dec 3 14:19 UTC 2010

Solaris has stable zfs (from what i've seen at wrk but my exposure is 
minimal) Why don't we create a test partition and see..

I think we are looking at it the wrong way (Linux/ZFS/etc).. We have a 
bunch of volunteers right? A, B, C etc.. If a A wants to try something 
let him try it so long as he doesn't create unwanted work for B! OR in 
other words B pre-approves the task.. Eg: If cross wants ZFS he should 
get it - If Tsty approves off it before hand (because TS/or-someone-
else will have to go reset the box/re-install the b0x if ZFS b0rks).. 
so get it pre-approved by WHOEVER has to clean up the mess..

This won't create problems and heated debates about which is the 
"better" solution. The actual "better solution" is ultimately people 
using the box.. if you got 50ppl in a bullock-cart and 1 in a car.. the 
cart is better simply because it serves more ppl..

response 16 of 29:

Dec 4 20:48 UTC 2010

Interesting - you were the resister and quite opposed to a move to the
cloud when I suggested it a couple of years ago.  What's changed since then?

And how might a move to the cloud in the future affect our choice of OS
today?

response 18 of 29:

Dec 8 16:38 UTC 2010

In addition to getting a handle on the legal ramifications, we'd also
want a cloud hosting service whose TOS are compatible with the kind of
system Grex is and wants to continue to be:  Free speech in the forums,
and full access by users to the full range of Unix tools and programming
languages after minimal verification.  Offhand I'm not so sure how easy
that will be to come by.  

Currently, there's a kind of "Grex annex" that I've been donating and
that resides in the cloud.  It's a FreeBSD virtual machine hosted at
rootbsd.net.  I've been using it for a while to keep an offsite (as in,
somewhere in North Carolina) backup of the conferences and the
grexconfig repository -- these are sync'd several times a day using
rsync, so that in case some catastrophe befalls Provide, we can recover
some essential data.  More recently, Dan used it to test pnewuser before
installing it on Grex and has been doing some other work on it as well.
 Any other root staffers on Grex are welcome to access it should they
wish to.  The general idea is to provide a resource separate from Grex
that staff can use to test things of possible use on Grex.

That machine works fine for the purpose for which it's being used, but
if you look at the rootbsd.net terms of service, it's pretty clear that
it wouldn't be a satisfactory host for the real Grex.  Too many
restrictions on what kinds of activities are allowed.  Part of moving
Grex to the cloud would entail finding a hosting service that gives us
the same flexibility that we have currently.

In the meantime, let's get back to the question of what we should be
doing now to prepare for an eventual move to the cloud.  Running Grex in
a VM on a machine that we actually own, as Dan suggests, would be a
reasonable step to take and pretty easy to set up, once we have
sufficiently modern hardware to do it on.  I'm wondering if FreeBSD is
the best choice of OS though, as Linux and Solaris seem to have more
widespread "cloud support" at the moment.

Or maybe the OS choice doesn't matter that much.  What we should
probably be working toward is making Grex portable, in the sense that we
could drop it into any Unix-ish environment (FreeBSD, Linux, Solaris, OS
X, whatever) and be able to bring up a fully flexible Grex system in an
automated way.  (Dan and I have already discussed this a bit.)

response 20 of 29:

Dec 14 20:20 UTC 2010

Speaking of the cloud, Netflix (definitely a big-time enterprise)
recently moved most of its ever-expanding services to Amazon's AWS cloud
service, as opposed to beefing up their own data centers.  Here's an
interesting post on the rationale, from Netflix's official "tech" blog:
http://techblog.netflix.com/2010/12/four-reasons-we-choose-amazons-cloud-as
.html
 (http://tinyurl.com/3yqxs5s).

response 22 of 29:

Dec 15 17:04 UTC 2010

Cool!

response 24 of 29:

Dec 15 18:13 UTC 2010

Yikes!  That's a huge bummer....

Yeah, we really need to switch.