|
Grex > Helpers > #142: Grex System Problems - Summer 2005 | |
|
| Author |
Message |
| 25 new of 281 responses total. |
albaugh
|
|
response 250 of 281:
| 
Aug 31 17:51 UTC 2005 |
Is it known yet whether the reboots are hardware-initiated,
software-initiated, or both?
|
remmers
|
|
response 251 of 281:
|
Sep 1 12:54 UTC 2005 |
Not known to me.
No reboots in two days. (cross fingers)
|
albaugh
|
|
response 252 of 281:
|
Sep 1 16:36 UTC 2005 |
As opposed to "finger cross". ;-)
|
cross
|
|
response 253 of 281:
|
Sep 1 20:32 UTC 2005 |
This response has been erased.
|
mcnally
|
|
response 254 of 281:
|
Sep 2 01:15 UTC 2005 |
Think pretty highly of yourself, don't you?
|
cross
|
|
response 255 of 281:
|
Sep 2 17:03 UTC 2005 |
This response has been erased.
|
tod
|
|
response 256 of 281:
|
Sep 2 17:04 UTC 2005 |
Would you settle for an MRE?
|
cross
|
|
response 257 of 281:
|
Sep 2 17:13 UTC 2005 |
This response has been erased.
|
happyboy
|
|
response 258 of 281:
|
Sep 2 17:16 UTC 2005 |
/send dan a big bucket of popeye's wings and a soady-pop
|
drew
|
|
response 259 of 281:
|
Sep 3 17:47 UTC 2005 |
Now it's refusing to let me enter stuff direct-dialed using vi. Just got a
"nasty error message" or something when I tried to enter a response.
|
richard
|
|
response 260 of 281:
|
Sep 13 18:54 UTC 2005 |
grex is back! thanks to staff for what sounds like a lot of work to
repair the labor day attack.
what exactly happened that caused this mess anyway?
|
aruba
|
|
response 261 of 281:
|
Sep 14 14:16 UTC 2005 |
Thanks to the staff member(s) who got Grex back up. Could we hear the
story?
|
eprom
|
|
response 262 of 281:
|
Sep 14 16:28 UTC 2005 |
The response time was outragious! We need some accountability here.
People need to be fired or demoted and a contigency plan should be
drafted up just incase this happens again!
|
remmers
|
|
response 263 of 281:
|
Sep 14 16:35 UTC 2005 |
The staff member who got Grex back up was me, aided by Jan Wolter's
life-saving mirroring software and some helpful advice in email from
Marcus Watts. I'm only sorry that I wasn't able to devote much
attention to it sooner, due to other commitments last week.
What happened: Some files in the /etc disk partition (in particular,
the password file) became corrupt, for reasons unknown to me but
probably due to a software glitch (don't know if it was OS software or
application software, either). I made a trip to our colo and was able
to run some tests and verify that the disks and filesystems were
healthy, but didn't have time to investigate further. On a subsequent
trip, I booted into single user mode and took some time to look around
the filesystem, eventually discovering that the password file (and
possibly others) had been corrupted.
Grex's important file systems (system directories, user directories,
bbs) are backed up to a spare hard drive every few hours, thanks to some
mirroring software that Jan Wolter wrote. Because of this, I was able
to restore "good" versions of the files in /etc from the state they were
in about 4 hours before the crash. Thankfully, that's all it took to
get Grex to boot successfully. The most that was lost was whatever new
accounts were created via newuser in that 4-hour period, I think.
Diagnosis of the cause of the problem will have to be left to someone
who knows more about OpenBSD than I do. Until the cause is addressed,
the problem may well recur. If it does, at least we know where to look
now, and Grex should be up a lot sooner. I'm sorry that it all took so
long this time.
|
edina
|
|
response 264 of 281:
|
Sep 14 16:43 UTC 2005 |
John, thank you for your assistance. It is appreciated.
|
jiffer
|
|
response 265 of 281:
|
Sep 14 17:01 UTC 2005 |
I say thanks to all the staff for spending their PRECIOUS time to help restore
grex. So, if you want to complain that it wasn't up faster, get the knowledge,
skill and volunteer to do it.
|
twenex
|
|
response 266 of 281:
|
Sep 14 20:25 UTC 2005 |
Re: #264, #265. Hear, hear!
|
naftee
|
|
response 267 of 281:
|
Sep 14 20:42 UTC 2005 |
Har, har !
|
rcurl
|
|
response 268 of 281:
|
Sep 15 05:09 UTC 2005 |
Re #265: while I agree that the staff are to be thanked heartily for their
efforts in maintaining Grex, I think it is unreasonable to expect everyone
to be come equally skilled before they can complain. After all, the members
of Grex that do not have the skills to do what staff does are still donating
the funds required for staff to do what they do. I think some thanks are
due for even just that - and that members do gain some license to complain
thereby. In addition, it would be a huge waste of time and money for
*everyone* using Grex to become equally skilled as staff, as then how could
all that talent possibly be used simultaneously? Isn't there a suitable
maximum to the number of staff required to adequately service Grex?
|
nharmon
|
|
response 269 of 281:
|
Sep 15 12:10 UTC 2005 |
As the number of talented staff increases, the better the chances that
someone will be available to work on the system at all hours of the day
and night.
|
bru
|
|
response 270 of 281:
|
Sep 15 12:13 UTC 2005 |
How about grex spend a little cash on the machine, hire a tech to come in and
FIX whatever is actually wrong and stop it from crashing.
|
jep
|
|
response 271 of 281:
|
Sep 15 14:07 UTC 2005 |
Minor thing, but isn't it time to remove the MOTD item that some
loginids are missing but the staff is working to recover them? It's
been there for something like 6 months, if not longer. The staff is
not working on recovering them at this point, or at lrast so I prefer
to believe. That announcement is kind of painful to see, day after day.
|
dpc
|
|
response 272 of 281:
|
Sep 15 14:08 UTC 2005 |
Thanks to remmers for saving us!
A few minutes ago, *both* dialin lines were ringing open. I thought
the system had crashed again. I am pleasantly surprised to see that
it's only the dialins that are hosed.
|
davel
|
|
response 273 of 281:
|
Sep 15 14:13 UTC 2005 |
|
davel
|
|
response 274 of 281:
|
Sep 15 14:15 UTC 2005 |
It apparently had crashed again. It wasn't available over the network,
either. last shows entries with end times of "crash" and a reboot about 9:50.
|