|
Grex > Helpers > #140: Grex System Problems - Spring 2005 | |
|
| Author |
Message |
| 25 new of 457 responses total. |
tod
|
|
response 188 of 457:
| 
May 2 19:23 UTC 2005 |
MOVE ON -Mary Remmers
|
gull
|
|
response 189 of 457:
|
May 2 20:23 UTC 2005 |
And I'm sure many of us will remember being told to "move on," when it
comes time to renew our memberships...
|
naftee
|
|
response 190 of 457:
|
May 2 21:20 UTC 2005 |
GreX should adopt a no-ID policy wrt memberships
|
steve
|
|
response 191 of 457:
|
May 2 21:31 UTC 2005 |
I have no evidence that Grex was vandalized. Believe me, I looked
and I've not seen anything that indicates that. Based on previous
examples of vandalism on Grex and M-net and other such systems, a
rm -rf / is one of the more common things done to systems.
The Sun-4/670 was incredibly reliable. It's tough to match that,
but I think we can. I realize that the current system has had problems,
which combined with other stuff has made for long periods of downtime.
STeve of Oz, eh? Ok.
|
steve
|
|
response 192 of 457:
|
May 2 21:38 UTC 2005 |
Changing one's password is never a bad idea. How often do people actually
change their passwords, I wonder.
Another reason I don't think Grex was vandalized was the fact that the
nulogfile that newuser writes data into was completely weird for many
accounts. Someone breaking in wouldn't do that.
|
tod
|
|
response 193 of 457:
|
May 2 21:40 UTC 2005 |
re #191
I wasn't referring to "vandalism". Did you miss the bit about "cracking"?
A corrupt passwd file should prompt staff to request all users to change their
passwords ASAP, don't you think?
|
steve
|
|
response 194 of 457:
|
May 2 21:43 UTC 2005 |
Cracking is vandalism.
|
mary
|
|
response 195 of 457:
|
May 2 22:03 UTC 2005 |
Re: 188 & 189 You guys are such drama queens.
Grex is unreliable at the moment and probably will be for some time
to come. If you have important files here, make sure they are
backed up. If you need mail, I mean NEED mail, Grex should not be
your primary mail drop. Realize that, find reliable email
elsewhere, and stop acting like clients not getting their expected
service. We aren't into service. We're into community. Don't
donate based on services you expect to receive. That's expecting
more than we're prepared to give. We are run by volunteers who are
pretty thinly spread at the moment. And we can't beat them into
giving more of their time. So it's up to the users to back off and
be as supportive and patient as they can. Not into that? Then you
really should move on. Grex isn't going to work for you.
|
jor
|
|
response 196 of 457:
|
May 2 22:19 UTC 2005 |
But what if the "paying members"
stamp their little feet?
Just teasin' ya.
tod has motivated me to change passwords.
man I wish I could lern to be one o'them thar Drama Queens
|
tod
|
|
response 197 of 457:
|
May 2 22:23 UTC 2005 |
re #195
Re: 188 & 189 You guys are such drama queens.
I prefer "prima attore" if you don't mind! 8P
|
naftee
|
|
response 198 of 457:
|
May 2 22:43 UTC 2005 |
i changed my passwd. thanks, tod !
|
janc
|
|
response 199 of 457:
|
May 2 23:00 UTC 2005 |
I'm not that sure that we need a bigger mail partition. The one time I
saw it fill up, it was due to a single user's mail file that grew way,
way big. When I deleted that file, mail was only like 46% full. I
probably should have studied the file before deleting it, because
theoretically no mail file should ever be able to grow that big. The
mailbox quota software I wrote should prevent it. But evidentally there
is a weakness in that somewhere.
If we got a bigger mail partition, but didn't fix this bug, then I
presume it would still fill up. If we do fix the bug, maybe we don't
need a bigger mail partition.
|
steve
|
|
response 200 of 457:
|
May 3 00:36 UTC 2005 |
Thats a good point, Jan. I saw the account thats been hogging so
much of /var/mail, so I did a talk at him. Turns out he's been
harassed by someone where he is, who subscribed him to a bunch of
things. He said he had in the process of getting off them, and I
think that perhaps he did it, since I haven't seen much activity
on that account lately.
|
cross
|
|
response 201 of 457:
|
May 3 02:47 UTC 2005 |
This response has been erased.
|
steve
|
|
response 202 of 457:
|
May 3 04:33 UTC 2005 |
Anything that is misconfigured is going to be a problem, Dan.
You know that. I don't understand your negative attitude towards
OpenBSD. I think you know that the panics we've seen have been
related to the networking card. OpenBSD 3.5 had a few problems
with it, and we've seen them.
But I'll tell you what we haven't seen: random breakins with
filesystems being destroyed, mail from root to staff ala "we 0wn u",
mail sent from others accounts by someone with root, or any of
the other marvelous little things that vandals do. And believe
me, people are trying, *all the time*. The number of exploits
I've seen brought over here, and the number of strange little
one screen programs making odd kernel calls is as high as I
have ever seen, only they're more tailored for OpenBSD.
Given the history of security on SunOS, we're a trophy. I've
been told this at least three times. There are people who really
dislike us just because we had the sense to lock down the system
back when NFS explots and remote mountings were common enough
that Grex not doing those things was rare.
I wish I'd had more time in the last several months to collect
some of the crap I've seen people try and run. But that we're
still here, haven't lost the system says *a lot*.
Have we had problems? Yes. We started out on the wrong
version of OpenBSD, by not using the latest. I do not fault
Jan for this; after all, he just about single handedly got the
system up. I was certainly useless during most of that time.
So we've been running a version behind from the start. Our
upgrade is going to change a lot of things, and I'm going to
be very surprised if we don't see an elimination of the nic
problem. The quota problem we saw I think is fixed but I
haven't looked at the cvs logs closely.
Nothing is perfect. As Marcus and I have said in the past
we've embarked upon an interesting journey here, in going to
a modern operating system. SunOS was stable, but also had
the advantage of being obscure enough so as to not to attract
a lot of vandal attention. Now that we're out of that world
we live in an enviroment where people do try active exploits,
and have access to the source. Ultimately the open source
model makes for much better security, but if we do find that
an exploit is out there we'll have to act quickly. I'll point
out that FreeBSD is in exactly the same position here. If
something comes out, swift reaction is needed.
Had we a pile of money I'd go for the raid design. When
we were first talking of building a new Grex it hadn't occured
to me that we wouldn't have constant access to the hardware.
I guess I fumbled on that one--it should have been talked
about more. Given Grex's budgetary restraints at the moment
I don't think that this is the best thing to do at the
moment. We need to get Grex 1) reliable, 2) deal with the
amazing Spam problem. Once we have a system that resembled
the stability of the Sun-4/670 we can start thinking of
other improvements; raid would certainly be a nice thing
to have.
Lastly, Grex does beat on its disks Dan. You've never
been in the Pumpkin and listened to the disks. I remember
a time some years ago that Marcus and I sat next to Grex
just listening to the noises the disks were making. The
activity lights were furiously blinking away and we could
hear the HP disk and at least one other as people were doing
things. Long time Grex folks will remember the various
disks we beat to death over the years, going all the way
back to the little 40M (not gig--meg) SMD disk that Marc
Unangst and I battled. What may be different now is our
current system; given the extra ram we have we might be
significantly be reducing disk movement. I don't know.
|
mcnally
|
|
response 203 of 457:
|
May 3 06:12 UTC 2005 |
I agree with Dan: Grex is not (or, not quite the same thing, should not
be) an exceptionally disk-intensive system for a multi-user server.
If disk is really thrashing that much we ought to (a) examine the partitioning
scheme, (b) look at other tuneable disk parameters, (c) maximize caching.
Even with 30-40 users logged on most of the time it doesn't make sense to
me that Grex's disks are getting pounded so hard, especially not the user
partitions.
I also agree with him that the explanation for the recent crash was quite
surprising to me. Did I gather correctly that an ordinary unprivileged
user can take Grex down with a fork bomb? Haven't we set per-user file,
memory, and process limits to reasonable values? I think I read recently
that OpenBSD doesn't set those in the default install, but are they set now?
|
gull
|
|
response 204 of 457:
|
May 3 13:31 UTC 2005 |
Re network card-induced panics: I recently had that same problem with
FreeBSD. Near as I can tell, the RealTek driver is buggy. OpenBSD
shares most of the same driver code, so if you have any RealTek cards in
use, you may want to change them. This may be a bug that only rears its
head on Alpha platforms, though.
|
janc
|
|
response 205 of 457:
|
May 3 14:04 UTC 2005 |
I've seen no clear evidence on what cause the last crash. We do have a core
dump and kernel image saved from that panic. It looked like something went
very bad while attempting to close a file descriptor. That could be a network
issue, or it could be just about anything else.
We've twice now had the password file corrupted. STeve attributed the first
one to a disk error. I assume he had evidence for that, but the few times
I tried to access that disk, I never got a single error message off it. Now
we have a second crash where the passwd database got corrupted and I can't
help wondering if something else might be going on. I wish somebody had time
to properly analysize these things.
|
tod
|
|
response 206 of 457:
|
May 3 15:14 UTC 2005 |
Yea, Dan. Its the DISKS, Dan. It gives worms to ex-girlfriends, Dan.
<pats self on back like bird flapping wings>
What's wrong with the RAID suggestion? It makes sense. If RAID won't fix
the problem then the OS needs to be replaced.
|
cross
|
|
response 207 of 457:
|
May 3 15:39 UTC 2005 |
This response has been erased.
|
cross
|
|
response 208 of 457:
|
May 3 15:41 UTC 2005 |
This response has been erased.
|
tod
|
|
response 209 of 457:
|
May 3 15:42 UTC 2005 |
re #207
Okay, I'll let the cat out of the bag: Staff had a report of a
security problem where a random, unauthorized users could run *cat*
on a tty device and see users connecting and typing their passords.
This was on Grex? Were the users notified that they should change their
passwords?
|
gull
|
|
response 210 of 457:
|
May 3 15:46 UTC 2005 |
FWIW, I think the security arguments for OpenBSD over FreeBSD are
overstated. FreeBSD gets the benefits of OpenBSD's code audits, because
a lot of code is shared. I also suspect FreeBSD has a larger installed
base, which tends to flush out driver problems sooner. I never ran into
them on my x86 machines. I've run into a few on my AlphaPC, but Alpha
is a minority platform that doesn't receive as much testing.
I'm not trying to weigh in on one side or the other here. I'm just
saying that the two operating systems are, from my perspective,
extremely similar, so I think in many ways it's an arbitrary decision.
Migrating from OpenBSD to FreeBSD, if you choose to do so, would
probably be fairly painless; much of the configuration is kept in the
same places. It still may not be easier than fixing what you've got,
though.
(Incidentally, keep in mind that OpenBSD's much-touted "only one hole in
the last 8 years" security claim applies only to *remote* exploits.
That suggests to me that security in a situation where you have local
shell users may not be their first priority.)
|
cross
|
|
response 211 of 457:
|
May 3 16:32 UTC 2005 |
This response has been erased.
|
tod
|
|
response 212 of 457:
|
May 3 16:36 UTC 2005 |
re #203
Did I gather correctly that an ordinary unprivileged
user can take Grex down with a fork bomb? Haven't we set per-user file,
memory, and process limits to reasonable values?
That's what I read from the explanation.
re #211
What's more, most of the security auditing that happens is poorly done
by amateurs. I wouldn't rely on it to run a bank.
I would hope you would want better security for the financial or healthcare
sector than for Grex, too. ;) At least with Grex, I'd hope we could find a
way to keep the system from crashing for several days at a time.
|