|
Grex > Helpers > #140: Grex System Problems - Spring 2005 | |
|
| Author |
Message |
| 25 new of 457 responses total. |
tod
|
|
response 212 of 457:
| 
May 3 16:36 UTC 2005 |
re #203
Did I gather correctly that an ordinary unprivileged
user can take Grex down with a fork bomb? Haven't we set per-user file,
memory, and process limits to reasonable values?
That's what I read from the explanation.
re #211
What's more, most of the security auditing that happens is poorly done
by amateurs. I wouldn't rely on it to run a bank.
I would hope you would want better security for the financial or healthcare
sector than for Grex, too. ;) At least with Grex, I'd hope we could find a
way to keep the system from crashing for several days at a time.
|
mcnally
|
|
response 213 of 457:
|
May 3 16:53 UTC 2005 |
Since I don't want to see this dissolve into a BSD-vs-BSD flame-war
death match, I'm going to try to subvert the conversation by proposing
some concrete suggestions that don't require immediate consensus on
the OS issue and don't require planning an OS upgrade or replacement
any time soon.
How about if we begin by:
Immediate, Critical:
1) Fixing the TTY security problem if it isn't already solved.
2) Make sure that sensible run-time limits are enforced and that
no ordinary user can cripple or crash the system with a fork
bomb.
Very Important:
3) Ensure that Exim version is sufficient to use recent SpamAssassin
integration features and begin testing system load under more
aggressive spam-filtering program.
4) Verify whether network driver support for RealTek NICs really is
affected by known bugs and add new ethernet card based on better-
supported chipset to system if so.
5) Consider setting up CVS or other versioning system to checkpoint
multiple backup copies of critical system files like /etc/passwd.
6) Research OpenBSD disk problems to see whether others are experiencing
similar crashes, in which case we should reconsider OpenBSD, or,
if not, we should consider the possibility of hardware problems as
a root cause.
|
nharmon
|
|
response 214 of 457:
|
May 3 17:42 UTC 2005 |
> I wouldn't rely on it to run a bank.
With most financial institutions, security is concentrated on the perimeter,
usually because the mainframe systems that run banking software use insecure
operating systems (Windows 2000 Datacenter comes to mind).
|
tod
|
|
response 215 of 457:
|
May 3 17:50 UTC 2005 |
re #214
With most financial institutions, security is concentrated on the perimeter
Actually, security is concentrated "in depth" as in at multiple layers like
a fortress with a moat, gate, guard tower, huge wall, etc
A firewall simply doesn't cut it anymore when you have GLBA worries, IT
productivity problems, password headaches, etc..
The least you should have are 2 firewalls with different flavors at the
perimeter of a financial institution but this is not a DMZ or IPS discussion.
The fact is, Grex had a security flaw and it wasn't reported to the users.
I'm disheartened at how this and the subpoena discussions have been buried
from the public discussion.
|
marcvh
|
|
response 216 of 457:
|
May 3 17:58 UTC 2005 |
Sure, and also financial security is based on the concept of transactions
and auditability. Grex doesn't have such beasts.
|
cross
|
|
response 217 of 457:
|
May 3 18:30 UTC 2005 |
This response has been erased.
|
nharmon
|
|
response 218 of 457:
|
May 3 18:32 UTC 2005 |
Re #215 - In Grex's defense, perhaps the Coop conference is a more appropriate
place to discuss Grex policies regarding notifying users. I've posted an item
that hopefully attracts some comments on the pros/cons.
Re #216 - It used to be that Banks didn't have to care very much about their
customer's names and addresses, etc...because this data was regularly bought
and sold to other companies. But the GLBA now requires us to safeguard this
information with the utmost diligence...to the extent that some banks will
fire employees for not locking their PCs and leaving them with customer
information still on the screen.
|
steve
|
|
response 219 of 457:
|
May 3 18:39 UTC 2005 |
We use a Broadcom 5702x nic.
Grex isn't a transaction system. I will agree that such a thing presents
more of a load than Grex does, but it also has hardware better suited to that
task. We've *listened* to the disks Dan. Honestly. There were times on the
Sun-4/670 that you could just sit there and hear them madly running around.
Perhaps I didn't say it well enough but OpenBSD may be significantly different
from SunOS in this regard; maybe it will be kinder on the disks due to caching
issues. I guess we'll see.
|
tod
|
|
response 220 of 457:
|
May 3 18:49 UTC 2005 |
Maybe IDE would be kinder than SCSI?
|
naftee
|
|
response 221 of 457:
|
May 3 18:51 UTC 2005 |
i use FreeBSD and Realtek and am pleased by the performance of both.
|
steve
|
|
response 222 of 457:
|
May 3 18:56 UTC 2005 |
I don't think the disk interface matters much. However, it has occured
to me in the last few minutes that we're swimming in disk compared to what
we had under SunOS: 256M there, and 1.5G here. That will eliminate swapping
and use about 75M ram for file caching which will also help.
I just changed the default limits in /etc/login.conf for maxproc to 32.
Maxproc-max was at 128.
|
steve
|
|
response 223 of 457:
|
May 3 19:06 UTC 2005 |
Now, as for sd0 having a problem, I just mounted it and tried copying
spwd.db to /dev/null. It failed. The message in /var/log/messages is
May 3 15:00:59 grex /bsd: sd0(ahc1:0:0): Check Condition on opcode 0x28
May 3 15:00:59 grex /bsd: SENSE KEY: Media Error
May 3 15:00:59 grex /bsd: INFO FIELD: 116647
May 3 15:00:59 grex /bsd: ASC/ASCQ: Unrecovered Read Error
May 3 15:00:59 grex /bsd: FRU CODE: 0xe4
May 3 15:00:59 grex /bsd: SKSV: Actual Retry Count: 134
May 3 15:01:00 grex /bsd: sd0(ahc1:0:0): Check Condition on opcode 0x28
May 3 15:01:00 grex /bsd: SENSE KEY: Media Error
May 3 15:01:00 grex /bsd: INFO FIELD: 116647
May 3 15:01:00 grex /bsd: ASC/ASCQ: Unrecovered Read Error
May 3 15:01:00 grex /bsd: FRU CODE: 0xe4
May 3 15:01:00 grex /bsd: SKSV: Actual Retry Count: 134
There are other errors on the disk as well. When I tried to dd the
entire disk I brought the system down, the day Joe said that newuser
was failing.
Have to go back and do work work now...
|
naftee
|
|
response 224 of 457:
|
May 3 19:12 UTC 2005 |
work work work
|
steve
|
|
response 225 of 457:
|
May 3 19:15 UTC 2005 |
work plod work
|
nharmon
|
|
response 226 of 457:
|
May 3 19:29 UTC 2005 |
plod no work,... abort, retry, or ignore?
|
cross
|
|
response 227 of 457:
|
May 3 20:02 UTC 2005 |
This response has been erased.
|
steve
|
|
response 228 of 457:
|
May 3 20:11 UTC 2005 |
Dan you are sliding off into fantasy land here. THE DISK HAS PROBLEMS.
It is as simple as that. If no one else saw the errors it was because no
one looked at /var/log/messages. I will point out that you could have
rummaged around there yourself to find errors. Sigh, I don't know why
I'm bothering to respond to some of your comments, but I will say that I
think Marcus and I know the difference between the sound of bearings
and the noise a disk makes when the heads are constantly moving.
|
tod
|
|
response 229 of 457:
|
May 3 20:17 UTC 2005 |
We're not worthy.
|
gull
|
|
response 230 of 457:
|
May 3 20:28 UTC 2005 |
Re resp:211: In my (admittedly limited) experience, banks run on Windows
and proprietary mainframes. The bank I worked for had *no* Internet
connections at all, though. All branch-to-branch connections were on
leased lines.
Re resp:213: #3 is a minor issue. AFAIK there are no major bugfixes in
recent versions of Exim. While versions earlier than 4.50 do not come
with Exiscan out of the box, it's easy to patch in, and the OpenBSD port
probably already includes a flag you can toggle to include it.
FreeBSD's does.
Re resp:219: Are we swapping? Maybe we need more RAM. Even if we're
not swapping, more RAM means more disk cache. RAM is cheap.
Re resp:227: Those messages pretty clearly indicate a hardware problem,
and if they were the result of an incorrect request on the part of the
driver we'd be seeing them on the other disks, too. I think you're
really reaching to blame OpenBSD here, which is unfortunate, because it
makes this look like a matter of religion on your part instead of a
technical argument.
If you're really convinced that OpenBSD is somehow causing the illusion
of a hardware failure on this disk, I suggest connecting it to another
system running a different OS and trying to access it. That should
settle the issue.
|
steve
|
|
response 231 of 457:
|
May 3 20:32 UTC 2005 |
No, we're doing fine for ram, now. The 256M swapping situation was
on the Sun-4/670. Unless spam processing eats up Grex's hardware I
think the 1.5G we have will last for some time. Sorry I wasn't clear
on that.
|
gull
|
|
response 232 of 457:
|
May 3 20:40 UTC 2005 |
I just took a look, and we've currently got 1.2 gigabytes free, so I
think you're right. I don't know how to find out how much OpenBSD is
using as disk cache. In FreeBSD it's reported by "top", but that
doesn't seem to be the case here.
|
gull
|
|
response 233 of 457:
|
May 3 20:43 UTC 2005 |
Also, someone on staff should please read my last set of comments in the
Exim item in the Garage conf. I clear up a couple of apparent
misunderstandings in Grex's current exim.conf file, and point out some
stuff that was copied verbatim from my example and shouldn't have been.
It looks like those things still haven't been fixed.
|
steve
|
|
response 234 of 457:
|
May 3 20:52 UTC 2005 |
Want to make up a diff?
|
cross
|
|
response 235 of 457:
|
May 3 22:00 UTC 2005 |
This response has been erased.
|
steve
|
|
response 236 of 457:
|
May 3 22:39 UTC 2005 |
1) The problem with permissions on the tty was our fault, I
believe. Do you think that this was a part of the release, and
that no one ever found it? We messed up, not the OS itself. I
ask you to prove otherwise. If its a real bug in the distribution
others would have seen it.
2/3) At least some of the crashes have been due to our nic. That
code was worked on post 3.5 release. I won't say that we've not
crashed for other reasons but have we properly analyzed it? The
quota code could well have some problems. I'll bet we're pushing
it. You are right that in our current configuration we are more
disk intensive than if softupdates were on. I'm pretty sure that
the softupdate code was changed post 3.5 and in visiting the
changelog between 3.5 and 3.6, we find
"Big FFS softdep merge with FreeBSD, fixing a number of bugs."
In the changelog between 3.6 and 3.7 we find
"Fix a soft dependencies problem that caused processes to get stuck."
This was a part of some stuff from FreeBSD which wasn't complete
apparently, and was then fixed.
You know as well as I do that saying something "will be fixed
in ..." is a dangerous thing to say. I'm not going to let your
bias against OpenBSD make me say things that shouldn't be
promised. But yes, I *do* think that 3.7 is going to be a good
move for Grex, as will 3.8 and so on.
|