response 26 of 137:

Jul 7 00:58 UTC 1995

Yes. Now. You have no idea how strong you are. I agree with 
Rane, except for one thing -- you are ready for the next step.
rcurl - srw - Steve - popcorn - danr have the "image" - now translate!

response 28 of 137:

Jul 7 16:46 UTC 1995

I think moving to an Intel platform would kill reliability.
I've yet to see an Intel box (one that doesn't cost as much as
a mid-range SPARC, at least) that is as reliable as the Suns.

response 30 of 137:

Jul 7 19:27 UTC 1995

Re #28:  What reliability?  :-/

response 32 of 137:

Jul 7 23:03 UTC 1995

I have particular and serious reservations about using pentium machines
for a "login" server.  I believe it's a virtual certainty that current
or future versions of the pentium chip will be found to have serious
holes in the protection logic, similar to the floating point bug.
Unlike the floating point bug, I believe Intel will be able to justify
ignoring these bugs, due to the "small" part of their market that would
be in fact affected by such a bug.  In fact, the only people who are
likely to be seriously affected are systems such as grex, and that's
such a small part of Intel's market, it's not even worth talking about.
I do see plenty of uses for the intel architecture in other things grex
is doing, such as our router.  But I very much do not believe it's
appropriate as our *login* machine.

Fortunately, there are a number of other architectures that do make
plenty of sense.  The sparc architecture is one such.  The newest
machines are certainly quite competive with the intel machines, both in
terms of price, and performance.  A large variety of older machines are
available in the used market.  The very oldest machines are even
compatible with the very cheap memory that we have currently invested
heavily in, for our current sun-3 CPU.  All of these machines are
software-wise, compatible, so, we only really need bite the
"incompatible software upgrade" bullet once.  This is exactly what the
sun-4 we're looking into represents - that's the next logical step on
our upgrade path, and it puts us into the first step of using the sparc
architecture.  If we care to spend real $$$, we can then make a massive
jump onto a much newer machine in this series, and having made this
step, that step will be *much* easier.

response 34 of 137:

Jul 9 08:08 UTC 1995

Without a detailed comparison of m-net vs. grex system crashes and
security incidents, I think it would be quite impossible to support any
such claim that either m-net or grex have had "more" incidents.
Certainly, I'd feel quite uncomfortable making such a claim, because I'm
not privy to any such information at m-net.

M-net has been lucky in that its always enjoyed better access to its
hardware.  Until recently, grex has been significantly handicapped in
access to its hardware, and that's really hurt us in quite a number of
ways.  The problems we have today, with the router, and with the disk
drive, are really problems we've inherited from that era.  I know I've
heard stories about an m-net upgrade to a faster CPU, that didn't work.
That's where good access really wins big.

So far as the intel architecture goes, ok, let's dig in.  When arbornet
originally went live, they used a 186 based Unix system.  The 186
doesn't have hardware protection, which means any would be dullard can
readily write a program to crash the machine.  Sure enough, they did,
and as an initial result, the people running arbornet limited access to
the C compiler & other facilities, and in the long run, they shifted to
a series of slower 68010 & 68000 based systems.  Performance is only one
leg of a triangle that also includes reliability and cost, and all 3
corners matter.

When the 286 came out, I knew friends who were working with the chip -
and I remember that, for a while, they were practically dealing with the
"chip du jour", many different masks, with a succession of problems.
Actually, there was one bug that Intel didn't fix until the pentium,
that allowed you to enable paging but stay in real mode.  One company
even introduced a product based on this "bug" in the 386 & 486.

That's all in the past, of course, and what really matters is the
future.  Unfortunately, the pentium is not exactly encouraging in this
regard.  With all past chip generations, intel has generally been quite
open about their chip design - if you pay them for all the manuals, and
read carefully, you can expect to know as much as anybody about what the
chip will do.  With the pentium, there is an "infamous" appendix H that
documents extra instructions and features of the pentium chip.  You only
get appendix H if intel likes you, and you sign a secret non-disclosure
agreement with intel.  Now, probably, most of that appendix is concerned
with instruction set timing, and it may only be that intel was so
embarassed about that part of their documentation that they decided to
only show it to their most trusted customers.  But, since the rest of us
can't look at it, there's no way for us to know if there's some special
bug or feature of the protection logic that's crucial to an air-tight
design.

Intel can only document their intended features.  Unfortunately, with
the pentium floating point bug, we have plenty of evidence how intel
might treat a crucial flaw in the protection logic.  This is a flaw that
is fairly rare and would not likely appear unless you sought it out, but
it's also a flaw that can be readily reproduced in one line of code from
any number of high level mass market applications.  Even so, intel has
always tried to underplay the seriousness of this flaw, and it was only
in the face of a massive public relations disaster that intel relented.
With a similar flaw in the protection logic, it's quite clear, intel is
not likely to do anything at all.

At various points, I've played around with successive generations of
intel based Unix systems, to see how rugged they are.  The 286 is really
the first to have any credibility at all; indeed, its protection logic
is basically quite similar to the pentium today.  When I tried some
mildly exotic things, I was able to readily crash a 286, by generating
some unusual kinds of faults that the kernel didn't know how to handle
properly.  I'm not greatly surprised by that, the intel documentation is
not especially clear on fault handling, and the chip is complex enough
that there are a large number of odd exceptions that can happen.  I've
tried similar things on more recent chips and operating systems, with
similar results.

Fairly recently, I had an opportunity to see the source code to an old
version of BSDI.  I was quite fascinated to see a way, deep inside the
guts of the kernel, that would allow any application to gain access to
the I/O space of the machine.  That would certainly allow a malicious
intruder to crash the machine.  With enough cleverness, I think an
intruder could use that to break root.

Now, I'll grant you, I'm probably a lot smarter than the average
cracker.  I think, too, the number of public access sites that actually
allow access to the compiler and encourage people to play around with
the Unix shell and such, is vanishingly small.  So I'm not really
surprised that m-net hasn't had any obvious problems with holes in the
pentium chip, or in BSDI.  Yet.  But I also think that some of these
crackers are fairly bright, and have a *lot* of time on their hands.  I
don't think anybody living in 1985 could have predicted the rise of
computer viruses.  I'm not sure what we can expect to see in the next
decade as more and more software takes advantage of memory protection.
But I think it would be foolish to assume there will never be any
incidents involving memory protection.

I think M-net has been quite lucky in having as few problems as you
claim, and can be justifiably proud of that.  I also think there is
every chance that the horrible things I've discussed above will not be a
problem for m-net.  I don't think it's a certainty however.  I don't
think grex's present architectural path has served it that badly - we
may be behind on CPU, but in fact, we're ahead on RAM; and really, the
largest bottleneck that faces both grex & m-net is not CPU but network
bandwidth.  I think there's an advantage to be had in terms of having
grex & m-net do *different* things (if nothing else, it means a bug some
cracker finds in one system may not be present in the the other).  And
I'm pretty confident that we'll be able to shortly overcome some of the
historical bugaboos we've been having and become quite a bit more
reliable.

One of the things I'm interested in seeing grex do is explore the
possibility of putting up a number of different machines and offloading
quite a bit of processing onto those machines.  Eventually, that might
take the form of something like this:

        +----- 3 file servers
        |
        +----- db server
        |
        +----- dns, mail hub
        |
        +----- news server
        |
        +----- 8 login servers
        |
        +----- 2 routers, & connections to outside world
        |
        +----- 4 terminal servers, and local terminals

this would be a far larger system than either m-net or grex today.  But
there is a limit to how far you can expand a single system, and past
that point, you really need more than one machine.  There *is* a certain
penalty to distributing services, so it makes sense to pick a solution
that scales well, and to have enough extra machines to make up for the
penalties of distributing service.  This is kind of a miniature version
of the sort of computing environment that exists today at CMU, or UofM,
and so, in a sense, I look at this as the natural direction of things.
Today, it's clearly much too expensive, but computing hardware is only
going to get cheaper, and grex & m-net both show every sign of growing.

response 36 of 137:

Jul 9 08:54 UTC 1995

re #34:  I wouldn't have expected that M-Net would have experienced any
         problems with "holes" in the Pentium chip.

response 38 of 137:

Jul 10 01:52 UTC 1995

        If you remember the I/O bug in enough detail, Marcus, I'd appreciate 
it very much if you'd e-mail information about it to me, or better yet, 
to jfk@arbornet.org.
        I appreciate your insights, and also your concerns.  I'm not aware of
the details of Grex's security problems, the causes or the solutions.  If
having better access to the hardware will help a lot in preventing those
problems, that's great, of course.  I agree that it's a good thing that
Grex and M-Net are not subject to all of the same security problems -- the
question for Grex has to be, is it a good enough thing to make it
worthwhile for Grex to remain on the hardware that it's on?  Are those
security advantages enough that it's worth not being able to use your
modems at their maximum speed, or to have files disappearing off Grex's
hard drive all of the time because the hardware cannot support the hard
drives that Grex is using?
        I'm not a security expert, not on your level at any case, but I'm a
cheeky enough guy that I can still question how the staff of Grex has
evaluated it's security concerns, as a knowledgeable user, anyway.  I
would think security should be measured in part by it's effect on 
functionality.  Is Grex so much more secure than it would be on an Intel
based machine that it's worth the heavy cost in performance and
reliability?
        By the way, I don't agree 100% that M-Net is limited greatly by it's
slow Internet connection.  For one thing, we're getting an ISDN connection
soon, we expect.  But for another, our emphasis is on local service, not
global service.  We've been offered direct access to CICnet's T3.  The
prevailing opinion is that this would be bad for M-Net because it would
change the way people would use M-Net very drastically.  I don't know the
end of this story yet.  I don't even know what I want to do yet.  My point
is only that a really fast Internet connection isn't an unmitigated
blessing.

response 40 of 137:

Jul 10 12:49 UTC 1995

I've heard that BSDi makes better use of memory than SunOS, so we could
probably get by on something closer to your 32 megabyte estimate, rather
than the full 128 megs.

response 42 of 137:

Jul 10 17:38 UTC 1995

I find it unlikely that static linking would produce efficient use of
memory, though I'm told the CISC/RISC distinction helps some.

response 44 of 137:

Jul 10 22:12 UTC 1995

M-net made a very good decision when they moved to a new 486 a few
years ago.  They probably didn't even anticipate back then that
within a few years they'd be down to a very precious few techies
capapble of futzing with an older, non-standard setup. Yet that's
where they are.  Lucky ducks.

I would think Grex could learn something from their experience.
That maybe right along with the money and reliablity we must
assume that our volunteer Sun staff might not always be so 
generous with their time.  Would it be easier to find qualified
staff on any other platform?  Would buying new save us from needing
as much technical expertise?

In my humble opinion, our weakest link at this point is not
community support, supply vs. demand, or even available money.
It's the very small group of folks who essentially keep the old
girl going, in their spare time, as best they can.  Grex needs
to move beyond what these few can do.

response 46 of 137:

Jul 11 13:40 UTC 1995

Just a minor correction Marcus. BSDI was orriginally called BSD/386. It
began life from the BSD4.3reno source code. At the same time, the Jolitz's
also began work on 386Bsd from the same sources. NetBSD and FreeBSD are
decendants of 386BSD, but are very different from BSD/386. They are
the product of 2 completely different development groups.

NetBSD currently supports Sun style Dynamic linking. BSDI announced that
it would support shared libraries in BSDI 2.0(the 4.4lite version), but when
it arrived, it turned out that they only implemented *static* linked shared
libraries. A technique that I find a PITA to use.

response 48 of 137:

Jul 13 10:30 UTC 1995

I'm not sure when BSDI & Jolitz split off; but it certainly wasn't
4.3bsd.  It may have been the "networking 2" tape; or it may have been
at some other point.  If the two did split off that early; they
certainly borrowed pretty freely.  Jolitz is identifed by name in the
BSDI 1.x kernel source; and the vm subsystem in both bsdi & bsdj is not
the 4.3bsd vm code at all, but one derived from mach, and adopted for
use in bsd by folks at the University of Utah.  Clearly, there are
differences - and especially in the device drivers you can tell that
bsdi branched off well before bsdj, and netbsd & freebsd in turn are
rapidly diverging from their common ancestor bsdj.

The most important difference between freebsd/bsdi/bsdj/netbsd/whatever
is not the actual code, or its history, but the price, support, &
development goals of the software.  Netbsd has been ported to many
platforms, but it was not intended for heavy production use and isn't
particularly robust.  The freebsd people have concentrated on the 386
platform, and their product is easier to install and more robust.  Both
products are free, and you get no support other than what you can do
yourself.  Bsdi cost money, but you get far better support.  The bsdi
folks are willing to take responsibility to make their product work for
the customer, not something either the freebsd or netbsd folks are
prepared to do.  M-net had money, users, and a burning need to replace
their suddenly defunct hardware, but a comparitively small technical
base.  Bsdi makes perfect sense for them.  Grex started off with quite a
few technically savvy people, hardware loans and donations, but little
money.  I believe SunOS has done us well.

Rob mentioned "security/reliability"; they seem theoretical until you
have to actually deal with them.  I've seem some fiendishly clever
crackers; I've spent plenty of time making things more reliable.  One of
the things I've learned is, if you're having problems, you analyze the
problem, and you attack the root causes.  If you don't attack the root
causes, trying to make massive changes in things has every chance of not
fixing the root cause, and is very likely to introduce new and even more
serious problems.  In this case, the root cause is not sunos; if you
look around the country, you will find numerous universities, colleges,
and freenets that have found sunos is perfectly capable of delivering
massive amounts of computing power in a thrifty and reliable manner.

With grex, you're seeing trailing edge technology.  Here's the leading
edge technology:
        a SunStation 20
        solaris 2.4
        dual processors.
        64 M ram.
I'm not sure of the cost -- $10k? perhaps 20k?  This is a machine that
will readily deliver high quality computing (ethernet speeds) to 50-100
users.  Or, if we downscale our expectation to grex speeds, probably
200-500 users.  Solaris has some significant differences from sunos.
The important thing, though, is that it supports SMP, which means the
dual processors of this configuration will cream a pentium in terms of
throughput.  Solaris has been less reliable than it ought to be.  With
2.4, however, it seems they've finally getting things right.  However,
that's one of the prices you pay for using leading edge technology, you
get to debug it.  If you want to be less aggressive: back down to one
processor, & sunos.  It's still a highly competitive machine, and it's
using extremely well debugged software.  This is a well tested, well
respected configuration, and really, basically the standard in the
industry.  Speaking of "well" tested- actually, the well has been using
solaris for some time now with no problems.  This is the state of the
art; and it's really not that far out of our reach financially.