OpenBSD: Discussion of appropriateness for grex and technical merits.

Sep 4 23:25 UTC 2006

Here are some reasons I believe one may consider OpenBSD inappropriate
for grex and/or "not modern."  I divide this into two sections.
The first consists largely of reasons I don't think it's particularly
appropriate for grex.  The second are reasons I wouldn't necessarily
consider it "modern," and is fairly technical.

Note: OpenBSD on the i386 architecture is certainly better for Grex
than was SunOS 4 on sun4m.  Compared to SunOS 4, OpenBSD is a panacea
of modernity.  That doesn't mean it is on an absolute scale, nor
relative to some of its contemporary peers.  In any case, it was
and remains an improvement on what we had before.

OpenBSD was chosen for grex largely because of its reputation as a
"secure" operating system, and largely at the behest of two staff
members.  One of which is no longer active with grex; the other
remains involved and is well known for his care of and ability to
maintain grex.  Despite this, whether by a defect in the operating
system or through improper configuration, we have had security
problems with it.  For instance, the `cat'able tty' hole that existed
under OpenBSD 3.5 on grex.  One naturally questions whether we could
have done better with another operating system?

INAPPROPRIATE FOR GREX:

Most of the problems with OpenBSD result from its relative obscurity.
OpenBSD is far more obscure than some of its better-known relatives;
even those in the BSD family tree.  In particular, FreeBSD has a
much larger userbase and audience.  This leads to several problems:

1) Potential administrators are less familiar with it, and the
   more esoteric features that are used by grex present a learning
   curve for anyone interested in the staff side of grex.  Certainly
   extremely important components of the system differ in non-trivial
   ways from the defacto standards presented by better-used
   alternatives.  For instance, compare the structure of OpenBSD
   ports to FreeBSD ports.

2) There are significantly fewer ported applications on OpenBSD
   than on, say, FreeBSD.  Many useful and desired applications
   don't exist on OpenBSD, or aren't as well supported.

3) Related to #2, there is deficient Java support on OpenBSD.  It
   seems that a JDK port is fairly recent, and there are problems
   with it.

4) Lower overall usage leads to less testing and in a system that
   is, overall, less stable.  This is true both of the core operating
   system and ported applications.  We've had a number of stability
   problems with OpenBSD that have led to crashes (bugs in the
   filesystem code have wedged the machine).

5) The engineering done by the OpenBSD team is less polished than
   that of other projects, and their focus is more on experimentation
   and tinkering.  There seems to be very little in the way of
   process, when compared to say, FreeBSD.

NOT MODERN:

Unix is nearly 40.  Despite continuous evolution over its lifetime,
remarkably little about it has changed: the structure of the current
operating system largely solidified sometime in the mid to late
1970's and then stagnated.  Despite several valiant attempts to
push it in a new direction, not much has happened, and while the
system certainly retains great utility, it can hardly be considered
a pinnacle of the modern state of the art.  Or, rather, it can only
in so much as the "art" has come to mean Unix, which is the textbook
operating system only because the textbook was written about it.
Even in this context, OpenBSD falls flat in several ways:

1) LRU based page replacement policies.  The 4.4BSD Mach-based
   virtual memory subsystem used in OpenBSD was replaced several
   years ago by the "new" UVM virtual memory subsystem to address
   deficiencies in the former that had been idetified over time.
   However, upon closer examination, one seems that UVM isn't all
   that modern: it's based on the same Least Recently Used page
   replacement strategy that Unix has used since paging was introducted
   on the VAX in the late 1970's and early 1980's.  This is despite
   some clear benefits that could be gained by using other strategies,
   such as working set methods.  In fact, the working set model was
   implemented by Charles Forsyth at the University of York in the
   1980's for SunOS 4, demonstrating that method's workability under
   Unix.

2) The antiquated and overly complex VNODE filesystem virtualization
   layer, adopted from 4.4BSD, has yet to be replaced with something
   more flexible.

3) The threading model is weak, lacking good support for kernel-level
   thread scheduling.  Multi-level scheduler activations remain
   non-existant.

4) SMP support remains weak, and most of the kernel is still single
   threaded (relying on interupt priority levels for synchronization
   between processes).

5) Too much emphasis is placed on the "Unix model" and "Unix way
   of doing things."  Hence, despite the widespread availability
   of bitmapped display hardware and networks, OpenBSD remains
   entrenched in the TTY based world for interaction.  The
   pseudo-terminal approach makes most programs think that they're
   talking to a VT100 terminal via a 9600 baud serial line, when
   in reality, they're interacting with a high resolution display
   over a high-speed local area network.  Things like X11 are poorly
   wedged onto this platform.

6) Evolution has led to a messy hodge-podge of interfaces for
   manipulating the system.  Network devices don't look like files
   but TTY's do; shared memory mechanisms are crude and lack higher
   level abstractions (e.g., we still do co-routine style concurrency
   management between processes that share memory, with explicit
   sychronization mechanisms like semaphores and mutexes, instead
   of CSP); we still use ioctl().  There is no push to clean up the
   mess of interfaces and approaches, despite prior examples of
   systems that do so but provide explicit compatability interfaces
   for legacy programs.

These are just a few.  I could come up with more, but I'm tired.

Comments?

response 2 of 80:

Sep 5 01:00 UTC 2006

I'll take these points on in the order presented by Cross.

INAPPROPRIATE FOR GREX

1. I know little about BSD admin in general, and even less about OpenBSD in
particular, but my impression from moving amonst Linuces is that there really
aren't that many show-stopping differences between them. A decent sysadmin
(and whilst I might not otherwise describe myself as one, let's do that for
the purposes of this post) can easily keep the differences between SuSE and
Gentoo in his head, for example.

2. Lack of wide software support is, surely, an issue if we use a wide variety
of software. I've not seen what I would consider as I wide variety either in
use, or being requested.

3. How much of a problem, realistically, is "lack of decent Java support"?

4. It's true that lack of a user/tester base can make a system less stable,
but I'm sure (given his comments in the past) Cross isn't recommending a move
to Linux, which definitely has the widest userbase of all free Unices and
(arguably) of all Unix systems in general. Even if I were to advocate a move
to Linux, it's clear to me that whilst the system itself tends to be rock
solid, applications such as Firefox (which, admittedly, we wouldn't be running
anyway) and even semi-critical OS components like NFS can be an unstable PITA.

5. I can't vouch for this either way, but I reckon Theo would have words to
say about it; granted, he seems to have words to say about a lot of things.

NOT MODERN:

OK, so OpenBSD isn't modern. FreeBSD (which seems to be the way a lot of
Cross's arguments are heading) might be argued to be more modern, but it's
actually older than OpenBSD itself, and both go back to a 30-year old operating
system (counting from the genesis of BSD specifically). I'm not being facetious
when I say that wheels aren't modern, either, and people still use those :-)

"State of the art" is overrated. Microkernels are arguably more
state-of-the-art than monolithic kernels, but the problem is that they tend
either not to work very well, be even more obscure than OpenBSD, or both. (I
don't consider Darwin, OS X or even mkLinux to be examples of successful
microkernels, since they essentially complicate a microkernel by building a
monolithic kernel over them; this might aid portability, but fulfills none of
the other "requirements" of a "successful" microkernel.)

Unix has the advantage that nobody else bothered to release a capable,
general-purpose, extensible, widely-supported, cheap OS *with source code*
just when operating systems research was arguably at its zenith. Windows is
widely available, but come on, it's not even under discussion for our
purposes. Plan 9 isn't the kind of improvement Cross seems to be after,
either, as all it seems to do is correct several flaws in the implementation
of the Unix networking stack (namely, that it disrupts the principle of
"everything is a file", and that distributed OSes built on Unix are untidy,
at best). It also has several disadvantages which coincide which those Cross
cites as being OpenBSD's (lack of support, lack of admins with sufficient
familiarity, etc).

1. I don't know of any OS that doesn't use LRU scheduling (though I certainly
don't claim to be an authority on the matter). OpenSolaris, maybe? I suspect
that the OpenSolaris community may be dwarfed by even the OpenBSD community.
All other OSes that might qualify are either even more marginal or unfit for
our purposes.

2. & 3. Which OSes with credible replacements for VNODEs and the OpenBSD
threading model are fit for our purposes?

4. IIUC, DragonFlyBSD solves some of the problems of SMP support, but it's
still far from being general purpose. I also understand that many of the
FreeBSD people themselves are unhappy with symmetric multi-threading on that
platform. SMP support on general purpose computers is either marginal (try
getting your hands on a Sequent) or experimental/new (just about everything
else).

5. Many of the problems Cross cites in his 5. could be solved by Plan 9;
unfortunately, even if I could see all Grexers moving to Plan 9 in a dual boot
or spare-machine configuration, current network latencies preclude this from
being a practical proposition. Either everyone has a T1 line (which rules out
most if not all international grexers, and I dare say a fair few domestic (US)
ones, or we put up with a system which *might, on occasion, have about 75%
of its heavy-use nodes online*.

X11 is fit for purpose; if we ever get to the stage where a substantial
proportion of grexers want to run graphical programs, there are X servers for
Windows and a bucket load of other OSes beside. On an aesthetic note, 81/2
and rio are ucking fugly.

6. Very few systems (if any) in as wide use (or greater) than Unix do NOT
suffer from a hodgepodge of interfaces; a great many, great interfaces of
yesteryear have been doomed by making them proprietary, leaving us only with
Beastie and The Beast (and the Penguin).

In short, Cross, you highlight many problems with OpenBSD and Unix, and I
agree with most if not all of them. The problem is, what are the reasonable
alternatives?

response 4 of 80:

Sep 5 03:43 UTC 2006

Regarding what Jeff wrote:

INAPPROPRIATE FOR GREX:

1) It's true that many of the differences are trivial, but some
aren't.  For instance, the whole authentication system is a bit
non-trivial.  Also, as Jan noticed when he put nextgrex together,
many of these things are poorly documented and in some cases just
aren't documented or, even worse, the documentation is wrong.  What
then?  Almost any Unix system is similar to any other, but that
doesn't mean you shouldn't know how one works, does it?

At some level systems are divergent just because they are, with no
better rationale.  Here's a difference between OpenBSD and FreeBSD
that was annoying around the time of the transition from SunOS: The
FreeBSD kernel caches a pointer to the current process's group
credentials in the networking code, which means that the FreeBSD
firewall can check that a user is a member of a specific group,
regardless of whether the saved, effective, or real group ID of the
process talking to the network is in that group.  OpenBSD does not
do this (it only saves the latter).  As a result, to implement the
packet filters previously handled by a custom hack to the (binary)
kernel, it was necessary to write a setgid wrapper and link lots
of things to it.  The wrapper would then look at how it was invoked,
set up the group permissions appropriately, and exec the proper
program.  This is a lot less flexible for our users, who may or may
not understand why the network program they wrote won't work.  It
was also a lot easier for junior sysadmins to install a network
program: under OpenBSD, install the program and THEN find and modify
the wrapper and reinstall that.  Under FreeBSD, they could just
install the program.

2) We might find we were able to provide more things that piqued
the interests of our users if we had the option of installing more,
varied, software.

3) Honestly?  I don't know.  But it *has* been requested, several
times.

4) No, I'm not recommending Linux, because frankly, the rate of
change for Linux is just too high and it's too insecure.  You're
right, it's not just testing, it's a balance of many different
things.  Testing must be compared to security, maintainability,
popularity, etc, etc, etc.  I don't think that Linux would really
win one of those competitions, but I certainly do think that FreeBSD
would beat OpenBSD on a level playing field.

5) I'm not sure Theo would be the one to ask.  He's too volatile.
But this goes along with (4), which really should be about the
stability of the system as a whole.  Testing is one measure of that,
but not the only one.  Also at play are the engineering practices
of the group that produces the OS.  FreeBSD does a bit of a better
job of this than OpenBSD, which seems to be more focused on doing
stuff that's "cool" and just sort of hacking it until it works than
on following the Unix model in the context of a well-defined systems
engineering process.

MODERNITY ISSUE:

I should have been clearer: I'm referring to OpenBSD being not
*modern* in an absolute sense, not with respect to whether or not
it's suitable for grex.  Certainly, a "modern" operating system
wouldn't include an abstraction for emulating a typewriter-based
TTY interface, but that model clearly works well for grex (well,
the next generation of glass text TTY's connected via serial lines
does, at any rate).  And yes, of course people still do use the
wheel.  But we *should* be able to separate what grex does from
what is "modern," and in that sense, very little of what grex does
is particularly modern at all.

As a note, Jeff knows full well that I'm a frequent Plan 9 user,
and that I do consider it quite an improvement over Unix.  It
certainly addresses many of the problems I raised (though it suffers
from others, too).  But, that's just a digression....  Anyway, to
address some of Jeff's specific points:

1) LRU scheduling or page replacement scheduling?  Well, VMS uses
the working set model, as did the EMAS system.  Supposedly Windows
does too, but I've never looked at Windows internals.  I suspect
that might have been part of the original, nice, clean abstract
Windows that didn't make it very far off the drawing board before
it was perverted.  I doubt Solaris does working set, either.

But that's only one problem with the OpenBSD VM system: do they
have a merged filesystem/VM buffer cache, yet?  That makes things
like memory mapped files inconvenient at best (what happens if I
mmap a file, and read from it, and then write to a page that I've
mapped through the filesystem?  Which gets update?  Are the pages
copied from one to the other?  Which way?  Hmm....  FreeBSD solved
this long ago).

2, 3) I'm not sure that I'd consider any credible replacements for
OpenBSD on grex, but Plan 9's model of a central *protocol* for
accessing file (and file-like) resources with the kernel being
something of a protocol multiplexor, is quite nice and much cleaner
than the Berkeley/Sun derived VNODE interface.  8th Edition Unix
had something called the "File System Switch" that consisted of an
array of jump-vectors for filesystem operations, with each "inode"
containing an offset into the array for its filesystem type, as
appropriate.  The VNODE interface tries to be too general, and
that's where it falls down.

4) Dragonfly does a lot of neat things, like fixing the trap problem,
but yes, it is far from perfect.

5) Perhaps.  For grex, it's largely irrelevant since that's the
prevailing and expected mode of operation from the users' stand
point.  If TTY's suddenly disappared, I can think of at least one
grex user who would have a cow.

6) Plan 9.

Regarding #3; VMS is nice, but unlikely to gain traction.  Grexsoft
doesn't matter, it's a dead effort anyway.

response 6 of 80:

Sep 5 14:10 UTC 2006

Oh, by the way, Plan 9, even more than OpenBSD, is an ideological
statement, not a production operating system.

response 8 of 80:

Sep 5 14:41 UTC 2006

Another sidenote: I wonder how many of the proponents of systems of yesteryear
spout steam when they hear Unix being held up as the canine's genitals of "a
well-defined systems engineering process"!

response 10 of 80:

Sep 5 17:33 UTC 2006

Not sure what in #5 you're referring to, Todd.

response 12 of 80:

Sep 5 18:26 UTC 2006

re #11
While its true that advisories and bugtraq reports are demonstrative of
testbeds and white hat hacking, I think its worth noting since most exploits
are roughly based on same.  For the same reasons, a person can brag about lack
of virus problems on OS X.

response 14 of 80:

Sep 5 23:13 UTC 2006

The ultimate objective with security as it relates to Grex should be to ensure
that resources are managed within an acceptable level of risk.  Some of the
common arguments here are whether maintenance, ease of use, and cost are
aligned with the choice of secure OS.  The question is what does the Board
of Directors deem "unacceptable" and how do we go from there to the
acceptable?  We know what the threats are.  The vulnerabilities are pretty
easy to get a grip on based on OS..but the risks themselves?  Those are the
possibilities.  To me anyway, I think most would agree that the risks are
less with OpenBSD than with most other thing except maybe NetBSD.  Some staff
are squealing that they want an OS with every tweak imaginable which supports
every application imaginable but that contradicts the maintenance and ease
of use argument.  Really, the BoD need to decide whether Grex is going to be
some sort of highly complex monster which requires a few dedicated (read as:
"single point of failure") admins or whether Grex is going to adopt a simpler
platform which offers more sustainability through recruitment and ease of use.

response 16 of 80:

Sep 6 19:26 UTC 2006

re #15
In theory, I agree with you that Grex should want the flexibility of an OS
but in reality its historically significant to point out that the staff is
run tighter than corn rows and the adaptability of its staff to diversity in
the OS is not openly accepted.  OpenBSD presents the easiest out of the box
solution in way of security for this cluster of nepotist marauders.  Nobody
likes to hear the truth but that's been my observation.  Grex can't even get
a grip on email let alone mischevious lil memory leak gremlins so opening the
system up to a flexible OS is just inviting all sorts of available exploits
and hacks to occur under the not-so watchful eyes of this overburdened staff.

response 18 of 80:

Sep 6 22:19 UTC 2006

re #17
 I strongly disagree - and, note well, I'm coming from both an inside
 staff perspective and security background.
Ok, I'll note your self proclaimed perpsective and background.  Note that I'm
also an oldschool Grexer and have an extensive security background.
(Why credibility is even being approached in your response seems weird.) 

Your claim that Valerie left due to personal attack redacts facts about her
actions which were out of place and unprofessional.  Also keep in mind that
Cross split because he was being micromanaged and treated unfairly by certain
longtime Grex authorities.  Read as: Nepotism ruled both situations 

I don't know where you originated your definition of security principles as
being "availability of service", either.  That may be a security strategy but
certainly not a principle.  Principles would be things like cost
effectiveness, accountability, change control/integration, etc (Mind you, I'm
citing NIST 800-14 off the top of my head as a DoD certified IAM.)

 The biggest, by far, reason why staff has gone missing in action in
 large numbers on Grex is this IGNORANT cowboy (e.g. OpenBSD) mentality.
And here I thought it had to due with the staff physically local to the Grex
machine micromanaging the true troubleshooting and decision making.  Silly
me.  OpenBSD is just a periphery discussion (unless you are close minded.)

Let's address what you really intended to prove:
 We went down much less
 often when we were on our last operating system - and were also down for
 less time on the odd ocassion we were down.....  OpenBSD is by no means
 necessary for locking down Grex, and is actually counter-productive --
 worse still is the attitude of those who think it is necessary for our
 perceived security needs!
1) Same hardware when you had less downtime?
2) SunOS specialist did major bugtracking in each OpenBSD server abend?  Do
you have the utmp and other logs available to show the true causes?
3) Is it "attitude" or "opinion" which shows preference? (Again, you're 
mixing your definitions.)  If you are truly getting upset then perhaps you
should not participate in debates about open systems with open minded people.

response 20 of 80:

Sep 7 00:13 UTC 2006

<stands naked behind HP2628A smart termial and waves>

response 22 of 80:

Sep 7 05:06 UTC 2006

 The first strategy of attack an adversary will do against you to screw
 your business is a denial-of-service attack against system availability.
I dunno...bots and local privilege exploits have historically been the
strategy most deployed on Grex.   I haven't seen anything about DDOS beyond
the fubar sendmail processes.  (I agree lack of staff interest is a big piece
of the puzzle and 'reactive' responses or sketchball installs have played a
part rather than proactive design.)  If there's some magic config out there
we don't know about it would've been great to see staff using it.  So far as
I know, OpenBSD is setup as such from the get-go thus quirky guru staff (which
haven't always been available/interested) wouldn't be such a single point of
failure.

response 24 of 80:

Sep 8 00:46 UTC 2006

The problem, and let's be perfectly frank here, is that security on grex is
important, but not the overarching goal.  OpenBSD was chosen by a couple of
the "quirky guru staff" members that Todd mentions because they felt it was
"more secure" than FreeBSD who acted as if security should be the *number one*
priority.  They strong-armed the decision and pushed it through, despite some
misgivings from other of the staff.  One of them is no longer involved with
grex.

And let's be even more perfectly frank: the choice was and is *only* between
OpenBSD and FreeBSD.  Whether this is technical or just plain bigotry is
irrelevant, but it is what it is.  Some people might pretend to make an
"objective assessment" but those are the only two systems that will ever be
considered (unless someone can figure out how to go back to SunOS 4).

But the reality of it, from what I can tell, is that both are about equally
secure, with OpenBSD being more pompous about their "high" level of security.
But in terms of actual levels of exploits, they're about on par, with perhaps
FreeBSD being a little better (and a bit more candid; that comes from being
less egotistical).  However, FreeBSD gives you many more choices in providing
interesting applications and actually building software, is better tested and
better engineered, and has more of a future ahead of it.  OpenBSD talks the
talk, but doesn't really walk the walk any better than anyone else when you
get down to the nuts and bolts.

On grex, there's an infuriating tendency to do things "just because" or based
on some emotional reaction ("I *feel* that this is more secure/better/
whatever...") versus based on technical merit.  There's also a tendency to
defer to the historical guru's on nearly all matters.  That's why OpenBSD was
chosen, not due to its superiority with respect to security or anything else.