response 75 of 181:

Oct 18 10:23 UTC 2001

Another advantage of running several different systems is we can take
advantages of the differences and strengths of each.  For instance, we
could more easily run mysql if we had a separate machine that ran linux
or solaris, because mysql likes to make heavy use of threads.  We might
not want to let random users lose on the same hardware, because of
various assorted security issues.

The worst case update scenario is when running a general purpose machine
that is fully exposed to potentially hostile users--ie, grex itself.  A
dedicated machine that is stripped down to provide just one purpose may
be much easier to secure, and it may be much less important to update it
regularly or to compile everything under the sun for it.  A kerberos
KDC, for instance, should not need a web server on it or X software on
it.

response 77 of 181:

Oct 18 15:10 UTC 2001

How do Sun and i386 machines of the class we're talking about
compare in cost?

response 79 of 181:

Oct 18 18:58 UTC 2001

Regarding #76; yes, but if 64-bit SPARC is what you're talking about in
terms of SPARC hardware (sun4m is probably too close to the end of it's
useful lifecycle to be anything other than a satellite server of some
kind), and OpenBSD is the only software platform you feel realistic, then
what to do about not having a sun4u version of OpenBSD?  I'd say FreeBSD
or OpenBSD running on i386 hardware would be better, even without the
hardware context switching stuff (does OpenBSD even know how to deal with
that?  BSD Unix on the VAX didn't use the VAX's hardware context switch
instructions, for instance, though VMS did).  But, bearing in mind that
grex could get something substantially faster than a 350MHz Pentium, I
wouldn't worry too much about context switch overhead.  Another example,
soda.cs.berkeley.edu is a mutli-processor PC running FreeBSD, and supports
hundreds of interactive users at any given time.  Also, ftp.freesoftware.org
is a FreeBSD machine running on Intel hardware that supports upwards of
5000 active FTP sessions at any given time, and moves over a terabyte of
data a day.  That's an impressive concurrency rate.  However, those
systems are turned for interactive, highly concurrent workloads, while
you're average out of the box system isn't.  Such tuning (or lack thereof)
could easily account for the performance difference you noted.

response 81 of 181:

Oct 19 01:27 UTC 2001

I disagree with Steve on several points.

First, I don't think security stands head and shoulder above all other
considerations.  Stability is right up there with it.  With all our
machines before the 4/670 we had the joy of regularly having he machine
crash, and having to reboot things.  No individual crash was quite as
annoying as a root break-in, but having a machine that constantly breaks
by itself is at least as bad as having one that occasionally gets broken
by someone else.  For this reason, I'm have little confidence in any
hardware/software combination that I do not see used by large numbers of
sites in intensive applications.

I think the three serious contenders are OpenBSD/x86, FreeBSD/x86, and
Solaris/sparc.  I've seen lots of reports of very heavy duty applications
of OpenBSD/x86 and FreeBSD/x86, but not of either on a sparc server.  There
is a big difference between running reliably on someone's desktop, and
running reliably as a busy server.  If you want to sell me on OpenBSD/sparc
then you'll have to provide some documentation of several such configurations
running reliably as heavy duty servers.

My other objection to OpenBSD/sparc is that many of the advantages of the
sparc architecture are probably going to be negated by running OpenBSD
instead of Solaris on it.  Solaris is, I like to imagine, extremely good
at making optimal use of the special features of Sun hardware.  Does OpenBSD
do as well?  I doubt it.  I think if you want OpenBSD then you should run
it on an x86 machine, where it gets most of its usage and development.  If
you want a sparc machine you should run Solaris, which is best optimized to
take advantage of the system.  Trying to mix and match is just sitting between
two chairs.

Second, I don't see OpenBSD as the only OS alternative.  I think good cases
can be made for each of the three architectures described above.  In fact,
I think OpenBSD would be my third choice.  I agree that OpenBSD has a security
advantage over FreeBSD, but I don't think it's as big as it appears at first
glance.  An awful lot of their idea of security is achieved by running a
minimum of remote servers and securing those very well.  Their security
releases tend to say things like:

  July 5, 2000: Just like pretty much all the other unix ftp daemons on the
   planet, ftpd had a remote root hole in it.  Luckily, ftpd was not enabled
   by default. The problem exists if anonymous ftp is enabled. (patch
   included) 

Of course, on our system ftpd *would* have been enabled, as it would be on
many OpenBSD sites.  The only thing that is "lucky" here is that they were
able to keep their claim of "no remote exploits in the default install" in
spite of the fact that they were no better than anyone else at spotting this
bug early.  I sense in this an orientation toward castle-building.  Strong
outer walls, narrow points of entry, carefully guarded.  Fine for lots of
people, but utterly useless for Grex.  Once you throw away most of the outer
defenses, as Grex certainly will do, then the case for the security difference
between OpenBSD and FreeBSD is less compelling.  FreeBSD actually has some
features (like being able to set up binaries on /suidbin so even root can't
edit them without rebooting the machine to a lower security level or setting
up log files so they can only be appended to unless you do the same thing)
that might be noticably useful to Grex.  FreeBSD issues security advisories
for most of the common programs that run on FreeBSD systems, while OpenBSD
tracks only the core distribution.  We'd probably want to be on the FreeBSD
security list even if we run OpenBSD, just to get these.

One important issue we need to address before building our next machine is
our future upgrade plan.  Up till now we've always just run the same system
without major upgrades for many years, and then built a whole new machine.
If we go with Solaris, we can continue to do this.  If we go with FreeBSD
or OpenBSD, I don't think we can.  The problem is the way they handle security
alerts.  Sun only just stopped releasing security alerts and patches for
SunOS 4.1.4 - about eight or nine years after release.  (The fact that they
seem to have mostly stopped doing so is another argument for getting off
the Sun).  The Open/FreeBSD people, however, are not so eager to keep patching
old releases.  With OpenBSD, the security advisories for one release stop
pretty soon after the next release.  FreeBSD keeps them going a little longer,
having a staged release policy with various stable and development releases.
But mostly after a year or two, you're going to be reading security alerts
for a newer version of the OS you're running.  Security holes won't be
reported for the old version unless they are also in the new version.  Fixes
will be written for the new version, and it will be up to us to adapt them
to the old version.  Before long the only sane thing to do is to upgrade
to the new version.  My guess is that with OpenBSD or FreeBSD we will be
under substantial pressure to do an OS upgrade every year and a half.

If that's true, the Grex staff is going to have to make a lot of changes to
how we make modifications to the system.  At a minimum, we have to record them
better so we can redo them every 18 months.  We might want to try harder to
get some of our modifications integrated into the distributions.  (I'd guess
that the FreeBSD people would be substantially more receptive than the
OpenBSD people).  We might prefer an OS that already has more of the features
we need built in.  That'd be FreeBSD instead of OpenBSD.  For example, Marcus
wants PAM.  FreeBSD has it, OpenBSD doesn't.  Marcus can certainly install
PAM in OpenBSD.  But does he want to do it every 18 months?

For me this is one of the strongest reasons for going with Solaris.  The
current 4/670 machine is wonderfully stable.  I think there are a lot of
reasons for this, including good hardware.  But part of it is that SunOS
4.1.4 is eight or nine years old now, and for all that time has been
continuously fixed without adding any new features.  Neither FreeBSD nor
OpenBSD is going to do that.  Sun will do that with Solaris just as it
has with SunOS.

So when I look at the sustained effort required to continue to maintain
security over the years rather than just the one time effort to build a
secure system, then I see Solaris as the "most secure", with FreeBSD a
long ways behind, and OpenBSD a bit further back.

Hmmm...maybe someone should start FlatBSD project.  Pick a current release
of free OS for a single defined hardware platform.  Doesn't matter what.
Commit to maintaining it for five years.  Bug fixes and security alerts,
but no new features.  Probably be some interest in that.  You'd be able
to run away with the prize for most secure open OS within a couple years.

response 83 of 181:

Oct 19 01:51 UTC 2001

I asked what we would distribute.  Someone said mail.  We've talked about this
a lot.  We could have the mail delivered to a separate machine.  How do we
read it?  Many mail clients here don't do POP or IMAP (hmm...maybe a wrapper
could be put around /bin/mail so that it always runs fetchmail once before
executing).  NFS is insecure.  AFS is weird and far from happening.  Plan
needs work.

Marcus said mysql.  Currently we have two rather small applications that use
an SQL server, so that would be kind of overkill in the short term.  In the
longer term I think they may find more uses, especially as we spread over
multiple machines and have more data we want to be able to access from
any machine.  However, remember that you have to at least be able to build
the client software on all servers.  I'm not entirely sure that will be easy
on SunOS.

We could certainly serve Grex's official web pages off another server.  User
pages might be a possibility, but then we'd have to find a way for users to
edit those pages.  Maybe restricted ftp access.

I can see some viability for these ideas, but frankly it'd be simpler to
have one fast machine that does it all than spreading the load among many
slow machines.  I like being able to edit my web pages without logging into
a different server.  We can afford one server fast enough to do it all
(heck, the current one does) so why split it up?  There might be some
security advantages, but in most cases as soon as you put something
interesting on a machine you have to start opening the machine up so
people can access it.

response 85 of 181:

Oct 19 08:59 UTC 2001

Lots of things to distribute: conferencing, mail, web pages,
authentication, file space, multiple login machines, ...

The OpenBSD folks are said to have a strong interest in sun hardware,
and are rumoured to be diligently at work on a sun ultra port.  That
means we could very likely be considering an ultra-5 or ultra-170 as a
possible hardware platform in the not too distant future.

So far as security goes, I have to strongly disagree with Jan and agree
with STeve.  Whatever sytsem we put up, there *will* be vandals trying
to break in *Every* day.  A lot of those vandals will be very stupid,
but some will not as stupid, and there is, somewhere far from our view,
some Very clever vandals who are going to be much more up to date on the
latest vulnerabilities than we can ever hope to be.

The cost of failure for us is also high.  A successful vandal attack is
likely to be a *major* nuisance to us.  A vandal attack could mean: user
private mail and files are stolen, every user has to change their
password, some users may find their accounts elsewhere are stolen, a
possible visit from the FBI or secret service, substantial staff time to
figure out how the vandals broke in and to fix the hole, substantial
down-time, user files may be lost and have to be restored from the last
backup, one or more user accounts may be stolen by malicious users, a
huge drop in user population and donations especially if downtime is
excessive, multiple outages if staff can't figure out how the vandal
broke in, or guess wrong, possible court time, legal expenses, &etc&etc.
We've been *VERY* lucky not to have faced any of these on grex in a long
time.  Many other systems have not faired nearly so well; m-net is a
good local example of this; the Well out in California had a famous
break-in, and arbornet had a crucial incident very early in its history.

Vandals usually like to attack the most popular hardware and software
technologies.  Unfortunately for us, those are also the ones that tend
to be "best" supported.  Fortunately for us, there is some difference
between quality and popularity.  If you think reliability is important,
then you ought to be *very* concerned about vandals, because the
simplest way to look at vandals is they're just another source of
instability.  Another way to look at security is that it is a min-max
problem: how to make the system as security as possible, while
compromising utility as little as possible.  If we were only concerned
about popularity, we'd obviously want to run under Windows NT or even
Windows 98 -- I don't think anyone here is willing to argue that should
be our future server architecture.

Regarding hardware cache contexts and all that -- this is one of those
messy hardware issues where it's all *too* easy to get bound up in the
details.  It is also an area where different systems can be *radically*
different, and yet not be visible to software.  A simple example:
openbsd runs on both Macintosh IIci's and Sun-3/160's.  The IIci uses
the built-in MMU of the 68030.  This provides for a minimal number of
CPU visible registers, stores most of the actual MMU map information out
in memory, and works in short very much like the DAT box of an IBM
360/67.  This is cheap in hardware terms and has only a modest effect on
straight-line performance.  It also degrades in a fairly
straight-forward fashion when doing lots of context switches.  The
SUN-3/160 uses the 68020, which has no built-in MMU, and a special
external sun MMU design which has *lots* of CPU visible registers.  It
is capable of mapping not only CPU references, but doing scatter-gather
I/O as well, which helps DMA performance.  The hardware contexts store
everything directly, so nothing lives in memory during actual use.  This
makes them *very* fast to use.  There are also multiple "contexts", so
the data for multiple address spaces can be resident in the MMU at the
same time.  This makes context switching (including interrupts, system
calls, and switching between users) relatively cheap.  There is a cost
for this however -- if you *exceed* the # of hardware contexts, the
system has to reload a whole bunch of MMU registers all at once.

What this means is: for the 2ci, if you were to plot out system
performance vs. run average, you would find a nice inverse relationship
-- and you'd probably want to keep the run average to not much more than
1 if you want to get the most out of your hardware.  This fits in well
with the IIci's general architecture.  For the 3/160, you would find
that performance remains relatively constant up to about a run average
of around 10 (8 or 14, but I forget which), then gets *much* worse.
Since most users aren't busy most of the time, that means you can
provide services for 20-30 users without too much trouble.

Our current hardware is descended from the 3/160, and has a somewhat
similar MMU, but features many more hardware contexts.  This means the
sharp knee in performance isn't a problem for us.  Generally the load
average is pretty low on grex, but there is one case where the ability
to handle lots of processes at once is pretty crucial, and that's mail
handling -- if we've been down for a few days, when we come back, we get
*hammered* -- by hundreds of mail messages all being processed "at
once".  Our current hardware does an astounding job of doing this pretty
gracefully -- much better than the sun-3/160 ever did.

Ever since the invention of the first computer speed-up, computer
performance has been *very* difficult to compare.  Cache memory,
buffering, multi-processing, translation look-aside buffers, and
pipe-lining all have their effects, which are usually not at all easy to
predict.  Usually, the best way is to find something as similar as
possible to the proposed load, measure it, then add a huge fudge factor
of uncertainty into the result.  I think it would be surprising to hear
that the average desktop PC, which is *not* designed for computationally
intensive context switching, is anywhere near as good at this as
specially designed hardware.  Stranger things happen all the time, on
the other hand.  It's definitely a 'try it and see' thing though.

response 87 of 181:

Oct 19 13:31 UTC 2001

If user web pages could be accessed only by restricted FTP, this is
essentially allowing only paying members to have web pages on grex.
Is this consistent with grex's mission?  I have made little (1k) web pages
for grex users as educational tools.

response 89 of 181:

Oct 19 14:17 UTC 2001

Keesan:  No.  In that particular scenario, the web pages would be on a
seperate machine.  You wouldn't be able to dial up or telnet directly to
that machine.  You would be able to ftp to that machine giving your login
and password, and use ftp to place files on it.  This could be done from
any machine with an internet connection.  People who don't have internet
connections could dial into Grex, edit their files there, and when they
are ready to put up on the web, ftp them to Grex.  The web server would
still be free and open to the public.  Anyone at all could put things up
there.  It'd just take an extra level of indirection.  It'd be a bit awkward,
which is why I'm not overwhelmingly delighted with the multiple server
approach, but it would change what people could do.

response 91 of 181:

Oct 19 15:14 UTC 2001

Marcus:  I agree that security is important and vandals are a really big
problem.  So you can consider me to have said "uh-huh" to your three
paragraphs of "vandalism is bad, bad, bad."  The issue here isn't whether
to deal with it, but how.  And I think STeve is wrong that OpenBSD is a
necessary part of the answer.  There are several phases to securing the
system.

  (1) Set up.  The easy part.  OpenBSD's default install is very, very
      secure.  But it's completely useless to us.  We'd need to open up
      OpenBSD a lot.  Whatever we use, we are going to have to do a lot
      of careful work to get it fairly secure.  Which operating system
      we start with isn't going to make all that much difference by the
      time we are done, as our security will depend as much on the software
      we install (which is largely the same no matter which OS we use) as
      on the OS.

  (2) Maintainance.  We need to track announcements of security holes and
      apply them fast.  For some OS's, this will require regular OS upgrades.
      Whatever operating system we choose, how well we succeed at this will
      depend mostly on us.  But the organizations behind the different OS's
      give us differing levels of support.  My assessment is that Sun does
      better than FreeBSD which does better than OpenBSD.  I may not be right.
      My reasons are given above.  I'm ready to be convinced otherwise.

  (3) Repair.  We've been lucky for a while now, and haven't had any root
      break ins.  We can't assume that this will continue to be true, even
      if we stay on the current OS.  So putting the system back together after
      a root break-in is part of life.  FreeBSD provides some notable
      advantages that make it easier to detect and limit the damage that
      can be done to the integrity of the system.

As for the security-by-obscurity argument - I don't believe it.  Yes,
stupid hackers will run Linux exploit scripts on your SunOS server, but
stupid hackers aren't really the ones we are concerned about.  We can
keep ahead of them.  As I see it, the various OS options just aren't
different enough to make any of the obscure enough to be invulnerable
to an intelligent attacker.  We've seen ftpd exploits that effected just
about every ftpd in circulation.  We've seen sendmail exploits that effected
just about every system running sendmail, including Grex's much hacked
sendmail.  I don't think security-by-obscurity buys you a substantial edge,
unless you are willing to run Plan 9 or OpenVMS.  (Hmm...plan 9 might
be an interesting alternative - I don't think I've ever heard of a plan 9
exploit.)

Running OpenBSD on a sparc instead of an x86 won't actually 'close'
more than a tiny fraction of the possible security holes.  It is likely to
bugger up a fair number of prepackaged exploit scripts, but no reasonably
intelligent hacker should have difficulty adapting them to the different
architecture.  So maybe running on weird hardware buys you a little time
before a new exploit is actually used on your system, but only a little.

Any you pay for it in reliability.  I think in open source OS's, you don't
get very far from "the more people are using it, the more thoroughly debugged
it gets".  Nobody writes bug-free process scheduling and memory management
algorithms on the first try.  Many eyes are needed.  In open OS's, the
search for stability/performance bugs and security bugs operate by almost
exactly the same process, sometimes done by exactly the same people.  They
tend to go in proportion to each other.  Only in commerical OS's are these
more decoupled.

response 93 of 181:

Oct 19 17:12 UTC 2001

A few general points: OpenBSD (and indeed, any 4.4BSD derivative) supports
the chflags stuff to change files to append-only, immutable, etc.

Marcus, that was a wonderfully clear exposition of hardware context issues
for non-technical people, but I'm still wondering to what extent OpenBSD
takes advantage of the capabilities of any given hardware platform.  Also,
I'm really not sure when SPARC-64 support is going to appear; I've been
waiting for it since the introduction of the Ultra.

One option for the upgrade issue is to track the -STABLE branch of the OS,
and do periodic rebuilds from source.  That negates the need to do a,
``wipe the disk and reinstall a binary distribution periodically'' issue
until that particular -STABLE branch is EOL'ed (usually longer than 18
months).  Nightly `cvsup's' of the system source provide a really easy
way to track changes and get the latest patches when needed.

Plan 9 would be interesting for grex, but probably not suitable, for several
reasons.  1) Compared to modern Unix distributions, it's rather, er, sparse
in terms of user features.  2) Much Unix software just won't run on it due
to missing facilities, among other things.  There is the APE (ANSI Posix
Environment) that they use to bring over useful programs, but it won't do
more exotic things like pthreads.  3) In an effort to reduce the complexity
of the system, the idea of a terminal line discipline has disappeared, as
have anything relating to termcap/terminfo/curses; say goodbye to vi.  This
more or less forces you to use a multiplexed bitmap display under control
of the Plan 9 window system (called rio), and the window system controls
quite a bit of the system's basic operation.  For instance, to interupt a
process (as one would using, say, ^C or whatever the stty intr was set to
under Unix), one types Del.  Del is then trapped by the window system, not
the OS (remember, there's no terminal line discipline!) and a note (there
aren't any signals, either!) is sent to the process group associated with
the window the Del was typed in.  What's this mean?  Well, you can't interupt
a process when logged in via telnet or ssh, for one.  Also, since there's
no terminal line discipline, there's no job control.  This would probably be
bad for most of the user's of grex.  4) There aren't many Plan 9 exploits
because there aren't many Plan 9 systems.  And finally, 5) you don't get
a lot of support from Bell Labs for the system; it's very much sink or swim.

That said, Plan 9 is a really cool OS, but it's very much a researchers tool
as opposed to something for general consumption.

response 95 of 181:

Oct 19 19:04 UTC 2001

Tried to do some looking around to see what other free shell providers run
on.  Mostly found there aren't many other free shell providers and even fewer
that are actually up and running at the moment.

http://sdf.lonestar.org/
   modified Linux on Athlon 750.  Or possibly two alphas.  Hard to tell. 
   Seems busy:
      6:47PM  up 9 days, 13:40, 161 users, load averages: 3.20, 3.50, 3.65
http://freeport.xenos.net/
    Redhat Linux.  Has had some hacking problems and no longer newusers.
http://www.nyx.net/
    Mostly Sun.  Various SparcServers, mostly inferior to Grex, running
    SunOS 4.1.4.

Not exactly a wealth of information there.

response 97 of 181:

Oct 20 00:20 UTC 2001

Security by obscurity is useful, but not reliable, and (as jan pointed out)
this is the case almost exclusively with no-brain script kiddies.  Any systemc
craker with the education to run gdb to generate shell code for that
particular os/hardware combination can find the stack pointer, and execute
the shellcode as desired, the only obstacle in the way is figuring out what
__asm__() function/instruction will return the actual stack pointer for the
buffer overflow.

currently, openbsd runs on sparc64 (ultra) machines, but questionably.  i say
questionably because of what is mentioned at
http://www.openbsd.org/sparc64.html, namely that many hardware facilities
are not supported (extensive pci and sbus expansions), and the boast is that it
runs multi user, but there are only "more or less functional snapshot" made
quite often.  most of the code is ported over from netbsd, and needs time to
settle into robustness.  it is indeed (again, as jan noted) secure by default,
which means very little for grex.  a quick edit of inetd.conf demonstrate that.
 this is not to say that openbsd is not secure, but rather that openbsd is just
that:  secure by default, not necessarily with thrid party software, as you
will run much of.

as for the distribution, consider this:

 *----------*---------------(inbound/outbound)---
 *   grex   *
 *----------*--------
                     |
 *-----------*     *----------------*
 * syslog/   *---- -* internal hub   *
 * backup    *     *----------------*
 *-----------*        |
                      |
 *-----------*        |
 * database  *--------
 * server    *
 *-----------*


a database (mysql/postgres or whatever) and syslog/backup server can both
remain resident on an internal network (hub, crossovers, what have you).  the
syslog/backup server can simply serve as a redundant syslog server (keep logs
locally on grex and copies on the syslog server) and as a backup server (be
it rdist or even a cron job that tars everything up and passes it over a
mounted filesystem, maybe ssh tunneling even though nothing's visible outside
of this network).  the database server can also be just that:  a database
server/.  both can handle minimal services, have only one or two accounts
(excluding root, of course) all the while not making you worry about having
additional static ip's or services visible to the rest of the world.  a syslog
server is very nice to have when your machine goes down, and it's comforting
to know that even though you may have append-only flags set on your log files,
there's a more reliable mirror residing within the same network.  just some
thoughts.

response 99 of 181:

Oct 20 02:45 UTC 2001

Could be.  I've been doing the patch work on Grex since the move to
the 4/670.  I haven't paid much attention to the numerous Sun sendmail
patches because we haven't been running Sun sendmail.  It has definately
taken some scrounging to get some of the patches - the security ones are
free, but some others aren't.  You'd think that now that they are giving
some operating system versions away almost for free that they'd give
away the patches too, but it's not obvious to me that that is the case.
Patch availability has somewhat improved, but not as much as one would
like.  This isn't ideal.  But if we were on OpenBSD they would have stopped
issuing patches or security reports for this OS version seven years ago.  

M-Net apparantly hasn't had any security breaches since their switch to
FreeBSD.  I think they've been steadily applying patches, and have done
two version upgrades.  There is a moderate degree of flakiness - some
early problems with "runaway processes" that they wrote a daemon to zap,
but which seem to have gone away in later versions of the OS; sendmail
stops working occasionally and needs a reboot to get going again.  They
follow the security mailing list and apply patches as soon as they are
announced.