|
Grex > Helpers > #140: Grex System Problems - Spring 2005 | |
|
| Author |
Message |
| 25 new of 457 responses total. |
steve
|
|
response 194 of 457:
| 
May 2 21:43 UTC 2005 |
Cracking is vandalism.
|
mary
|
|
response 195 of 457:
|
May 2 22:03 UTC 2005 |
Re: 188 & 189 You guys are such drama queens.
Grex is unreliable at the moment and probably will be for some time
to come. If you have important files here, make sure they are
backed up. If you need mail, I mean NEED mail, Grex should not be
your primary mail drop. Realize that, find reliable email
elsewhere, and stop acting like clients not getting their expected
service. We aren't into service. We're into community. Don't
donate based on services you expect to receive. That's expecting
more than we're prepared to give. We are run by volunteers who are
pretty thinly spread at the moment. And we can't beat them into
giving more of their time. So it's up to the users to back off and
be as supportive and patient as they can. Not into that? Then you
really should move on. Grex isn't going to work for you.
|
jor
|
|
response 196 of 457:
|
May 2 22:19 UTC 2005 |
But what if the "paying members"
stamp their little feet?
Just teasin' ya.
tod has motivated me to change passwords.
man I wish I could lern to be one o'them thar Drama Queens
|
tod
|
|
response 197 of 457:
|
May 2 22:23 UTC 2005 |
re #195
Re: 188 & 189 You guys are such drama queens.
I prefer "prima attore" if you don't mind! 8P
|
naftee
|
|
response 198 of 457:
|
May 2 22:43 UTC 2005 |
i changed my passwd. thanks, tod !
|
janc
|
|
response 199 of 457:
|
May 2 23:00 UTC 2005 |
I'm not that sure that we need a bigger mail partition. The one time I
saw it fill up, it was due to a single user's mail file that grew way,
way big. When I deleted that file, mail was only like 46% full. I
probably should have studied the file before deleting it, because
theoretically no mail file should ever be able to grow that big. The
mailbox quota software I wrote should prevent it. But evidentally there
is a weakness in that somewhere.
If we got a bigger mail partition, but didn't fix this bug, then I
presume it would still fill up. If we do fix the bug, maybe we don't
need a bigger mail partition.
|
steve
|
|
response 200 of 457:
|
May 3 00:36 UTC 2005 |
Thats a good point, Jan. I saw the account thats been hogging so
much of /var/mail, so I did a talk at him. Turns out he's been
harassed by someone where he is, who subscribed him to a bunch of
things. He said he had in the process of getting off them, and I
think that perhaps he did it, since I haven't seen much activity
on that account lately.
|
cross
|
|
response 201 of 457:
|
May 3 02:47 UTC 2005 |
This response has been erased.
|
steve
|
|
response 202 of 457:
|
May 3 04:33 UTC 2005 |
Anything that is misconfigured is going to be a problem, Dan.
You know that. I don't understand your negative attitude towards
OpenBSD. I think you know that the panics we've seen have been
related to the networking card. OpenBSD 3.5 had a few problems
with it, and we've seen them.
But I'll tell you what we haven't seen: random breakins with
filesystems being destroyed, mail from root to staff ala "we 0wn u",
mail sent from others accounts by someone with root, or any of
the other marvelous little things that vandals do. And believe
me, people are trying, *all the time*. The number of exploits
I've seen brought over here, and the number of strange little
one screen programs making odd kernel calls is as high as I
have ever seen, only they're more tailored for OpenBSD.
Given the history of security on SunOS, we're a trophy. I've
been told this at least three times. There are people who really
dislike us just because we had the sense to lock down the system
back when NFS explots and remote mountings were common enough
that Grex not doing those things was rare.
I wish I'd had more time in the last several months to collect
some of the crap I've seen people try and run. But that we're
still here, haven't lost the system says *a lot*.
Have we had problems? Yes. We started out on the wrong
version of OpenBSD, by not using the latest. I do not fault
Jan for this; after all, he just about single handedly got the
system up. I was certainly useless during most of that time.
So we've been running a version behind from the start. Our
upgrade is going to change a lot of things, and I'm going to
be very surprised if we don't see an elimination of the nic
problem. The quota problem we saw I think is fixed but I
haven't looked at the cvs logs closely.
Nothing is perfect. As Marcus and I have said in the past
we've embarked upon an interesting journey here, in going to
a modern operating system. SunOS was stable, but also had
the advantage of being obscure enough so as to not to attract
a lot of vandal attention. Now that we're out of that world
we live in an enviroment where people do try active exploits,
and have access to the source. Ultimately the open source
model makes for much better security, but if we do find that
an exploit is out there we'll have to act quickly. I'll point
out that FreeBSD is in exactly the same position here. If
something comes out, swift reaction is needed.
Had we a pile of money I'd go for the raid design. When
we were first talking of building a new Grex it hadn't occured
to me that we wouldn't have constant access to the hardware.
I guess I fumbled on that one--it should have been talked
about more. Given Grex's budgetary restraints at the moment
I don't think that this is the best thing to do at the
moment. We need to get Grex 1) reliable, 2) deal with the
amazing Spam problem. Once we have a system that resembled
the stability of the Sun-4/670 we can start thinking of
other improvements; raid would certainly be a nice thing
to have.
Lastly, Grex does beat on its disks Dan. You've never
been in the Pumpkin and listened to the disks. I remember
a time some years ago that Marcus and I sat next to Grex
just listening to the noises the disks were making. The
activity lights were furiously blinking away and we could
hear the HP disk and at least one other as people were doing
things. Long time Grex folks will remember the various
disks we beat to death over the years, going all the way
back to the little 40M (not gig--meg) SMD disk that Marc
Unangst and I battled. What may be different now is our
current system; given the extra ram we have we might be
significantly be reducing disk movement. I don't know.
|
mcnally
|
|
response 203 of 457:
|
May 3 06:12 UTC 2005 |
I agree with Dan: Grex is not (or, not quite the same thing, should not
be) an exceptionally disk-intensive system for a multi-user server.
If disk is really thrashing that much we ought to (a) examine the partitioning
scheme, (b) look at other tuneable disk parameters, (c) maximize caching.
Even with 30-40 users logged on most of the time it doesn't make sense to
me that Grex's disks are getting pounded so hard, especially not the user
partitions.
I also agree with him that the explanation for the recent crash was quite
surprising to me. Did I gather correctly that an ordinary unprivileged
user can take Grex down with a fork bomb? Haven't we set per-user file,
memory, and process limits to reasonable values? I think I read recently
that OpenBSD doesn't set those in the default install, but are they set now?
|
gull
|
|
response 204 of 457:
|
May 3 13:31 UTC 2005 |
Re network card-induced panics: I recently had that same problem with
FreeBSD. Near as I can tell, the RealTek driver is buggy. OpenBSD
shares most of the same driver code, so if you have any RealTek cards in
use, you may want to change them. This may be a bug that only rears its
head on Alpha platforms, though.
|
janc
|
|
response 205 of 457:
|
May 3 14:04 UTC 2005 |
I've seen no clear evidence on what cause the last crash. We do have a core
dump and kernel image saved from that panic. It looked like something went
very bad while attempting to close a file descriptor. That could be a network
issue, or it could be just about anything else.
We've twice now had the password file corrupted. STeve attributed the first
one to a disk error. I assume he had evidence for that, but the few times
I tried to access that disk, I never got a single error message off it. Now
we have a second crash where the passwd database got corrupted and I can't
help wondering if something else might be going on. I wish somebody had time
to properly analysize these things.
|
tod
|
|
response 206 of 457:
|
May 3 15:14 UTC 2005 |
Yea, Dan. Its the DISKS, Dan. It gives worms to ex-girlfriends, Dan.
<pats self on back like bird flapping wings>
What's wrong with the RAID suggestion? It makes sense. If RAID won't fix
the problem then the OS needs to be replaced.
|
cross
|
|
response 207 of 457:
|
May 3 15:39 UTC 2005 |
This response has been erased.
|
cross
|
|
response 208 of 457:
|
May 3 15:41 UTC 2005 |
This response has been erased.
|
tod
|
|
response 209 of 457:
|
May 3 15:42 UTC 2005 |
re #207
Okay, I'll let the cat out of the bag: Staff had a report of a
security problem where a random, unauthorized users could run *cat*
on a tty device and see users connecting and typing their passords.
This was on Grex? Were the users notified that they should change their
passwords?
|
gull
|
|
response 210 of 457:
|
May 3 15:46 UTC 2005 |
FWIW, I think the security arguments for OpenBSD over FreeBSD are
overstated. FreeBSD gets the benefits of OpenBSD's code audits, because
a lot of code is shared. I also suspect FreeBSD has a larger installed
base, which tends to flush out driver problems sooner. I never ran into
them on my x86 machines. I've run into a few on my AlphaPC, but Alpha
is a minority platform that doesn't receive as much testing.
I'm not trying to weigh in on one side or the other here. I'm just
saying that the two operating systems are, from my perspective,
extremely similar, so I think in many ways it's an arbitrary decision.
Migrating from OpenBSD to FreeBSD, if you choose to do so, would
probably be fairly painless; much of the configuration is kept in the
same places. It still may not be easier than fixing what you've got,
though.
(Incidentally, keep in mind that OpenBSD's much-touted "only one hole in
the last 8 years" security claim applies only to *remote* exploits.
That suggests to me that security in a situation where you have local
shell users may not be their first priority.)
|
cross
|
|
response 211 of 457:
|
May 3 16:32 UTC 2005 |
This response has been erased.
|
tod
|
|
response 212 of 457:
|
May 3 16:36 UTC 2005 |
re #203
Did I gather correctly that an ordinary unprivileged
user can take Grex down with a fork bomb? Haven't we set per-user file,
memory, and process limits to reasonable values?
That's what I read from the explanation.
re #211
What's more, most of the security auditing that happens is poorly done
by amateurs. I wouldn't rely on it to run a bank.
I would hope you would want better security for the financial or healthcare
sector than for Grex, too. ;) At least with Grex, I'd hope we could find a
way to keep the system from crashing for several days at a time.
|
mcnally
|
|
response 213 of 457:
|
May 3 16:53 UTC 2005 |
Since I don't want to see this dissolve into a BSD-vs-BSD flame-war
death match, I'm going to try to subvert the conversation by proposing
some concrete suggestions that don't require immediate consensus on
the OS issue and don't require planning an OS upgrade or replacement
any time soon.
How about if we begin by:
Immediate, Critical:
1) Fixing the TTY security problem if it isn't already solved.
2) Make sure that sensible run-time limits are enforced and that
no ordinary user can cripple or crash the system with a fork
bomb.
Very Important:
3) Ensure that Exim version is sufficient to use recent SpamAssassin
integration features and begin testing system load under more
aggressive spam-filtering program.
4) Verify whether network driver support for RealTek NICs really is
affected by known bugs and add new ethernet card based on better-
supported chipset to system if so.
5) Consider setting up CVS or other versioning system to checkpoint
multiple backup copies of critical system files like /etc/passwd.
6) Research OpenBSD disk problems to see whether others are experiencing
similar crashes, in which case we should reconsider OpenBSD, or,
if not, we should consider the possibility of hardware problems as
a root cause.
|
nharmon
|
|
response 214 of 457:
|
May 3 17:42 UTC 2005 |
> I wouldn't rely on it to run a bank.
With most financial institutions, security is concentrated on the perimeter,
usually because the mainframe systems that run banking software use insecure
operating systems (Windows 2000 Datacenter comes to mind).
|
tod
|
|
response 215 of 457:
|
May 3 17:50 UTC 2005 |
re #214
With most financial institutions, security is concentrated on the perimeter
Actually, security is concentrated "in depth" as in at multiple layers like
a fortress with a moat, gate, guard tower, huge wall, etc
A firewall simply doesn't cut it anymore when you have GLBA worries, IT
productivity problems, password headaches, etc..
The least you should have are 2 firewalls with different flavors at the
perimeter of a financial institution but this is not a DMZ or IPS discussion.
The fact is, Grex had a security flaw and it wasn't reported to the users.
I'm disheartened at how this and the subpoena discussions have been buried
from the public discussion.
|
marcvh
|
|
response 216 of 457:
|
May 3 17:58 UTC 2005 |
Sure, and also financial security is based on the concept of transactions
and auditability. Grex doesn't have such beasts.
|
cross
|
|
response 217 of 457:
|
May 3 18:30 UTC 2005 |
This response has been erased.
|
nharmon
|
|
response 218 of 457:
|
May 3 18:32 UTC 2005 |
Re #215 - In Grex's defense, perhaps the Coop conference is a more appropriate
place to discuss Grex policies regarding notifying users. I've posted an item
that hopefully attracts some comments on the pros/cons.
Re #216 - It used to be that Banks didn't have to care very much about their
customer's names and addresses, etc...because this data was regularly bought
and sold to other companies. But the GLBA now requires us to safeguard this
information with the utmost diligence...to the extent that some banks will
fire employees for not locking their PCs and leaving them with customer
information still on the screen.
|