M-Net Downtime Update -- 06/07/2000

Jun 7 05:24 UTC 2000

Hello.  I thought I'd take a few minutes to recap our current situation,
fill in some of the gaps in the information that has been disclosed, and
offer up a more focused forum for questions and answers about our current
downtime.

A few months back, M-Net was compromised by a user named "xdr".  If I
recall correctly, he took advantage of an unchecked buffer (and a local
configuration boo-boo) in webyapp to gain root.  After that hole was
closed, mjb uncovered a back door that had been left behind.  Entering
"vtxdr" at the "terminal type" prompt in newuser dumped the user into a 
root shell prompt.

M-Net's entire staff was advised of the breech, and for several days,
several of us concentrated our efforts on locating any other holes that
had been opened.  We found none.

Everything was normal for awhile.

xdr showed up in party one afternoon, participating in some rather
pleasant conversations with a few people.  Seemed like a nice guy.  He
showed up at our board meeting, and introduced himself as "Jason".  I
commented on where I recognized his username from.

A few days after our board meeting, I tried logging into M-Net, and found
that my password had been changed.  I contacted Rex immediately.  After
recovering my account, I helped casper and mjb get back online -- it
turned out that every root's password had been changed, except for Rex's.

We looked around for holes, but found nothing.  Whatever he did, he
carefully covered his tracks.  Over the next few days, I periodically
noticed "footprints" -- root activity on a pty with no corresponding wtmp
record, etc.  

We started discussing strategies for securing M-Net.  We came up with
several ideas, and started working together some of the implementation
plan.  Then M-Net went down.

It was decided fairly quickly that no matter what, we wanted to get some
new equipment in place, and we wanted to start fresh with M-Net.  Our
logic was this:  If M-Net was down due to hardware failure, we'd need new
hardware anyways, and if M-net was down due to vandalism, we'd probably
want a clean slate, too.

tod coordinated the purchase of new system, rounding up financial
contributions from himself, trex, jerryr and me.  We collectively
purchased an AMD Athlon 750MHz with 128MB of RAM, which we are 
donating to Arbornet.  This system is currently in Rex's posession.

Rex has picked up the M-Box from WWNet in Livonia.  He was unable 
to log into the system in multi-user mode.  Details of the backup 
process were discussed in another item.

From here, the plan (as I understand it) is as follows:

1) Install FreeBSD-4 on the new M-Box,
2) Carefully compile and test all of our various 3rd party software (Yapp,
   Apache, Orville-Write, Party, and so forth),
3) Recover as much data as possible (user mail, conferences, user home
   directories, and so forth),
4) Restore M-Net to service.

This is not a "snap our fingers" process.  It requires a lot of skilled
man-hours.  We ARE doing our best to get M-Net back online as quickly as
we can.

I can't speak for everyone, but this was among the worst possible weeks
for us to have this kind of problem, as far as my work schedule is
concerned.  I'm buried up to my nose at the office.  I'm not sure how many
more processor cycles my brain can spare towards this nightmare. 

Hang in there, people.  We'll be back as soon as we can, bigger and better
than before.  Details here as available.  Rex, please fill in any holes or
make any corrections.  I'm running on empty.  Time for sleep.

response 2 of 123:

Jun 7 11:05 UTC 2000

it's been noted before, but to keep the update complete it should be mentioned
that the hdd in the new box was donated by seldon.

response 4 of 123:

Jun 7 12:52 UTC 2000

Thanks for the update and thanks for all the effort. It isnt going to kill
me if Mnet is down for a bit. 

response 6 of 123:

Jun 7 14:11 UTC 2000

Thanks for your hard work, guys.  Take the time to do it right.  Let
us know if we should send pizza.

response 10 of 123:

Jun 7 14:41 UTC 2000

Thanks for the update, Mike.  We should try to get this post into the
website for folks that don't normally check Grex.

BTW, I saw Jason drive away from the BoD meeting in a white Saturn with
a fishing cap. If you have further info on 'xdr', please forward it to me.
(We could use more root talent like his! *smirk*)

response 14 of 123:

Jun 7 15:03 UTC 2000

Back to the topic at hand, I'm going to take care of some stuff and then
head into the office.  I'll be back online in awhile.  I'm on ICQ at
8744004 or AIM at Smike430 if anyone needs to track me down when I'm not
on Grex.  

response 16 of 123:

Jun 7 16:56 UTC 2000

        And in general, both M-Net and grex present themselves
        as targets because of the talent gathered, making them
        a prize to hack, an accomplishment.

response 20 of 123:

Jun 7 18:00 UTC 2000

        an unfortunate choice of terminology iggs

response 22 of 123:

Jun 7 18:24 UTC 2000

interesting that all the roots users pw's were changed except t-rex's--
and whywould xdr put in a backdoor that included his login id as part of
it (vtxdr)?  I think signs point to xdr being framed as the vandal.  

response 24 of 123:

Jun 7 18:32 UTC 2000

i admit it, while i was admiring igbor's crack (i had no idea at the time it
was wise) i cranked up my jr. space ranger ray gun and zapped the system. 
i am so ashamed.