response 57 of 181:

Oct 15 19:09 UTC 2001

I see a lot to like in the idea of buying new and commonly available 
hardware, for which components and software are widely and cheaply 
available, which runs at the speed of the latest computers, and for 
which an upgrade path will exist for the foreseeable future.

response 59 of 181:

Oct 15 22:20 UTC 2001

I sent mailt o Marcus and STeve to ask them to let us know if the proposed
date doesn't work for them.

response 61 of 181:

Oct 16 00:47 UTC 2001

As far as my calendar goes STeve has nothing going for Nov 3.

response 63 of 181:

Oct 17 02:15 UTC 2001

A declining user base and risk of death are two entirely different things,
though they often coincide.  Sound weird?  Well, it is, sort of, but all grex
needs to function properly is enough users willing to pay to maintain it. 

response 65 of 181:

Oct 17 14:45 UTC 2001

Re #62: I think you're right about buying a pre-assembled rack mount 
server, unless we turn out to need something really unusual.  Some even 
have RAID on the motherboard, now.

Dell's rack mount servers, while *easier* to mount in their own racks, 
*can* be mounted in standard 2- or 4-post racks.  Similarly, to mount 
an ordinary rack-mount device in a Dell rack only requires a set of 
snap-in nuts to put in the square rack rail holes.  Dell does have 
their own peculiar hot-swap hard disk caddies for their rack-mount 
boxes, though, so if you buy one from them you generally have to get 
any internal disks from them as well.

NetBSD does do a really good job maintaining code bases for older 
hardware.  I was thinking more of Linux.  In some cases, for example, 
updates to SCSI drivers caused older cards to fail, and due to lack of 
interest this was never corrected.  A friend of mine has a couple 
Adaptec cards that are useless under Linux because of this.

response 67 of 181:

Oct 17 18:05 UTC 2001

Interesting discussion.  Some time ago I promised grex some hardware, but
have been remise in sending it to Michigan.  This is due to a lot of
different factors (the stuff is in another borough, I need to get it while
someone is around and then take it to UPS or FedEx or whatever, and that's
difficult during the day, the attacks and dealing with the aftermath had
all but erased it from my mind), but grex is still welcome to the machines
if they'd like them.  However, it seems that concensus is leaning towards
Intel machines, and these are Suns (slower Suns, too).  Should I hold off
on sending until the outcome of this meeting?

My own 2c: I recommend running etiher OpenBSD or FreeBSD on Intel machines,
for reasons I've already outlined elsewhere.

response 69 of 181:

Oct 18 01:04 UTC 2001

I think it's very likely we'll end up distributing at least some
functionality, so we can definitely use more machines.

response 71 of 181:

Oct 18 03:39 UTC 2001

Hmmm...if we start running multiple machines is one operating system or many
the better idea?

response 73 of 181:

Oct 18 03:43 UTC 2001

The advantage of running several different operating systems is the
confusion factor:  Breaking one doesn't make breaking the others any easier.

The disadvantage of running several different operating systems is the
confusion factor:  Maintaining one doesn't make maintaining the others
any easier.

response 75 of 181:

Oct 18 10:23 UTC 2001

Another advantage of running several different systems is we can take
advantages of the differences and strengths of each.  For instance, we
could more easily run mysql if we had a separate machine that ran linux
or solaris, because mysql likes to make heavy use of threads.  We might
not want to let random users lose on the same hardware, because of
various assorted security issues.

The worst case update scenario is when running a general purpose machine
that is fully exposed to potentially hostile users--ie, grex itself.  A
dedicated machine that is stripped down to provide just one purpose may
be much easier to secure, and it may be much less important to update it
regularly or to compile everything under the sun for it.  A kerberos
KDC, for instance, should not need a web server on it or X software on
it.

response 77 of 181:

Oct 18 15:10 UTC 2001

How do Sun and i386 machines of the class we're talking about
compare in cost?

response 79 of 181:

Oct 18 18:58 UTC 2001

Regarding #76; yes, but if 64-bit SPARC is what you're talking about in
terms of SPARC hardware (sun4m is probably too close to the end of it's
useful lifecycle to be anything other than a satellite server of some
kind), and OpenBSD is the only software platform you feel realistic, then
what to do about not having a sun4u version of OpenBSD?  I'd say FreeBSD
or OpenBSD running on i386 hardware would be better, even without the
hardware context switching stuff (does OpenBSD even know how to deal with
that?  BSD Unix on the VAX didn't use the VAX's hardware context switch
instructions, for instance, though VMS did).  But, bearing in mind that
grex could get something substantially faster than a 350MHz Pentium, I
wouldn't worry too much about context switch overhead.  Another example,
soda.cs.berkeley.edu is a mutli-processor PC running FreeBSD, and supports
hundreds of interactive users at any given time.  Also, ftp.freesoftware.org
is a FreeBSD machine running on Intel hardware that supports upwards of
5000 active FTP sessions at any given time, and moves over a terabyte of
data a day.  That's an impressive concurrency rate.  However, those
systems are turned for interactive, highly concurrent workloads, while
you're average out of the box system isn't.  Such tuning (or lack thereof)
could easily account for the performance difference you noted.

response 81 of 181:

Oct 19 01:27 UTC 2001

I disagree with Steve on several points.

First, I don't think security stands head and shoulder above all other
considerations.  Stability is right up there with it.  With all our
machines before the 4/670 we had the joy of regularly having he machine
crash, and having to reboot things.  No individual crash was quite as
annoying as a root break-in, but having a machine that constantly breaks
by itself is at least as bad as having one that occasionally gets broken
by someone else.  For this reason, I'm have little confidence in any
hardware/software combination that I do not see used by large numbers of
sites in intensive applications.

I think the three serious contenders are OpenBSD/x86, FreeBSD/x86, and
Solaris/sparc.  I've seen lots of reports of very heavy duty applications
of OpenBSD/x86 and FreeBSD/x86, but not of either on a sparc server.  There
is a big difference between running reliably on someone's desktop, and
running reliably as a busy server.  If you want to sell me on OpenBSD/sparc
then you'll have to provide some documentation of several such configurations
running reliably as heavy duty servers.

My other objection to OpenBSD/sparc is that many of the advantages of the
sparc architecture are probably going to be negated by running OpenBSD
instead of Solaris on it.  Solaris is, I like to imagine, extremely good
at making optimal use of the special features of Sun hardware.  Does OpenBSD
do as well?  I doubt it.  I think if you want OpenBSD then you should run
it on an x86 machine, where it gets most of its usage and development.  If
you want a sparc machine you should run Solaris, which is best optimized to
take advantage of the system.  Trying to mix and match is just sitting between
two chairs.

Second, I don't see OpenBSD as the only OS alternative.  I think good cases
can be made for each of the three architectures described above.  In fact,
I think OpenBSD would be my third choice.  I agree that OpenBSD has a security
advantage over FreeBSD, but I don't think it's as big as it appears at first
glance.  An awful lot of their idea of security is achieved by running a
minimum of remote servers and securing those very well.  Their security
releases tend to say things like:

  July 5, 2000: Just like pretty much all the other unix ftp daemons on the
   planet, ftpd had a remote root hole in it.  Luckily, ftpd was not enabled
   by default. The problem exists if anonymous ftp is enabled. (patch
   included) 

Of course, on our system ftpd *would* have been enabled, as it would be on
many OpenBSD sites.  The only thing that is "lucky" here is that they were
able to keep their claim of "no remote exploits in the default install" in
spite of the fact that they were no better than anyone else at spotting this
bug early.  I sense in this an orientation toward castle-building.  Strong
outer walls, narrow points of entry, carefully guarded.  Fine for lots of
people, but utterly useless for Grex.  Once you throw away most of the outer
defenses, as Grex certainly will do, then the case for the security difference
between OpenBSD and FreeBSD is less compelling.  FreeBSD actually has some
features (like being able to set up binaries on /suidbin so even root can't
edit them without rebooting the machine to a lower security level or setting
up log files so they can only be appended to unless you do the same thing)
that might be noticably useful to Grex.  FreeBSD issues security advisories
for most of the common programs that run on FreeBSD systems, while OpenBSD
tracks only the core distribution.  We'd probably want to be on the FreeBSD
security list even if we run OpenBSD, just to get these.

One important issue we need to address before building our next machine is
our future upgrade plan.  Up till now we've always just run the same system
without major upgrades for many years, and then built a whole new machine.
If we go with Solaris, we can continue to do this.  If we go with FreeBSD
or OpenBSD, I don't think we can.  The problem is the way they handle security
alerts.  Sun only just stopped releasing security alerts and patches for
SunOS 4.1.4 - about eight or nine years after release.  (The fact that they
seem to have mostly stopped doing so is another argument for getting off
the Sun).  The Open/FreeBSD people, however, are not so eager to keep patching
old releases.  With OpenBSD, the security advisories for one release stop
pretty soon after the next release.  FreeBSD keeps them going a little longer,
having a staged release policy with various stable and development releases.
But mostly after a year or two, you're going to be reading security alerts
for a newer version of the OS you're running.  Security holes won't be
reported for the old version unless they are also in the new version.  Fixes
will be written for the new version, and it will be up to us to adapt them
to the old version.  Before long the only sane thing to do is to upgrade
to the new version.  My guess is that with OpenBSD or FreeBSD we will be
under substantial pressure to do an OS upgrade every year and a half.

If that's true, the Grex staff is going to have to make a lot of changes to
how we make modifications to the system.  At a minimum, we have to record them
better so we can redo them every 18 months.  We might want to try harder to
get some of our modifications integrated into the distributions.  (I'd guess
that the FreeBSD people would be substantially more receptive than the
OpenBSD people).  We might prefer an OS that already has more of the features
we need built in.  That'd be FreeBSD instead of OpenBSD.  For example, Marcus
wants PAM.  FreeBSD has it, OpenBSD doesn't.  Marcus can certainly install
PAM in OpenBSD.  But does he want to do it every 18 months?

For me this is one of the strongest reasons for going with Solaris.  The
current 4/670 machine is wonderfully stable.  I think there are a lot of
reasons for this, including good hardware.  But part of it is that SunOS
4.1.4 is eight or nine years old now, and for all that time has been
continuously fixed without adding any new features.  Neither FreeBSD nor
OpenBSD is going to do that.  Sun will do that with Solaris just as it
has with SunOS.

So when I look at the sustained effort required to continue to maintain
security over the years rather than just the one time effort to build a
secure system, then I see Solaris as the "most secure", with FreeBSD a
long ways behind, and OpenBSD a bit further back.

Hmmm...maybe someone should start FlatBSD project.  Pick a current release
of free OS for a single defined hardware platform.  Doesn't matter what.
Commit to maintaining it for five years.  Bug fixes and security alerts,
but no new features.  Probably be some interest in that.  You'd be able
to run away with the prize for most secure open OS within a couple years.