|
Grex > Coop13 > #380: Cyberspace Communications finances for November 2006 | |
|
| Author |
Message |
| 25 new of 124 responses total. |
keesan
|
|
response 86 of 124:
| 
Dec 21 02:11 UTC 2006 |
Ric, what percentage of spam do you eliminate? Remmers, when do you expect
to have something set up for people to use who are averse to copying over two
files and changing the login name and want a script to do it for them?
|
cross
|
|
response 87 of 124:
|
Dec 21 03:02 UTC 2006 |
Regarding #85; Personally, I'd like to see the entire system on hot-swappable
media: both the user data, and the operating system. We had an occassion
where the root filesystem got lost once, and grex was down for at least
several days. If that filesystem had been RAIDed, we could have avoided that
downtime. I don't believe you can boot fram RAIDframe, either, which implies
that the root filesystem cannot be as redundant as we'd (perhaps) like.
I'm in favor of moving to a rack mount case with the storage system you
proposed, and disposing of the SCSI disks. Perhaps selling them and the SCSI
controller would be a way to offset the cost --- at least partially --- of
getting this new hardware.
|
maus
|
|
response 88 of 124:
|
Dec 21 04:59 UTC 2006 |
If I remember correctly, the way you do it is to make every filesystem
except / on software RAID and make an identical copy of / on the first
slice of the second drive, so no, it is not fault tolerant live, but if
you cannot boot from the normal /, then you just issue your boot command
to bring you up on the alternate copy of /. Things could have changed,
though, since I have not done RAIDframe-based RAID in a while, mostly
relying on 3ware boards for mirroring Serial ATA or IDE drives, and LSI
or Adaptec boards for mirroring SCSI drives.
|
cross
|
|
response 89 of 124:
|
Dec 21 14:12 UTC 2006 |
That is what you're supposed to do, but then you have to have some mechanism
for mirroring the root filesystem over to the spare partition; that gets
ugly after a bit. What's more, if one of the disks running the root
filesystem goes down, you still have to manually reboot. A hardware RAID
solution is better in the sense that this is handled for you automatically;
if one of the disks holding / dies, you just throw in another hot-swappable
disk and go on your merry way. Sure, one can approximate this using our
existing SCSI disks and RAIDframe and mirroring the root filesystem, but
why bother?
|
maus
|
|
response 90 of 124:
|
Dec 21 14:50 UTC 2006 |
I recommended keeping our existing SCSI infrastructure to capitalize on
sunk costs and because by keeping the system separate from the data, we
decrease a bottle-neck. I agree, with good, inexpensive hardware RAID,
RAIDframe kind of blows in comparison.
|
cross
|
|
response 91 of 124:
|
Dec 21 15:50 UTC 2006 |
You decrease a bottleneck, but at what cost? Then you have the associated
maintenance costs if the root disk fails, which is what we're trying to avoid.
I'd say that the goal of an increase in performance at the expense of added
(or, unchanged) administrative burden is the opposite of what we're trying
to achieve (or, what we *should* be trying to achieve). As for the sunk
costs.... Well, grex cold do several things with the existing SCSI disks and
controller. (1) Put up a satellite machine ala gryps to offload some of the
processing from the main machine. For instance, a basic spam/virus blocker
for mail before it gets to the `main' grex machine, or running proxy servers
for web and/or DNS, having a serial port plugged into the serial console of
grex itself, etc. (2) Sell them and use the proceeds to offset the new
hardware costs. (3) I'm sure there are others.
The other factor is that I *really* don't believe that grex gets enough usage
to worry about bottlenecks throught he I/O controller right now.
|
maus
|
|
response 92 of 124:
|
Dec 21 16:43 UTC 2006 |
That's a fair assessment. Consider my mumblings about the bottleneck the
ramblings of a weary mind, and please ignore said mumblings.
So, silly question: if we are thinking about moving the system-space
onto a new disc subsystem, does this mean a fresh, new installation? Can
we use the opportunity to request new commands to be added and to
implement new controls and move to standards from odd Grex-isms?
|
cross
|
|
response 93 of 124:
|
Dec 21 17:01 UTC 2006 |
No, not at all; I think it's good to be challenged and be asked to justify
one's conclusions. I thank you for that.
I think you can always request the installation of additional software. And
yes, I *do* think it would mean a new installation of the basic system. But,
that might not be a bad thing. Any opportunity to move to standard commands
from weird customizations is a plus, in my opinion.
|
maus
|
|
response 94 of 124:
|
Dec 21 17:15 UTC 2006 |
I agree that moving to standards would be a good thing, provided nothing
is broken in the process (if a command that users or staff depend on is
broken by the move to standardize, then the standardization is crap; if
no-one is hurt and we make the system easier to maintain and easier to
upgrade and actually match what the man pages and web-pages say, then we
have done a good thing by standardizing and deserve doughnuts).
I didn't have specific commands or software in mind (or, at least,
nothing appropriate for this system), but I figured that if we were
facing a fresh installation, this would be the time to ask people what
commands they would like to see on here, and also see if there are
commands that users would want to see upgraded or replaced.
|
maus
|
|
response 95 of 124:
|
Dec 21 17:29 UTC 2006 |
On the new commands front, I was just looking through /usr/local/bin and
noticed javac and java and jar. I thought the port of native java to
OBSD was still a couple of years away. Did we build this by way of RHEL
emulation or something else entirely? Have we published somewhere how we
managed it? Hurray f
|
remmers
|
|
response 96 of 124:
|
Dec 21 17:44 UTC 2006 |
Hmm... I seem to recall that I saw the Java stuff sitting in either the
OpenBSD ports or packages collection a few months ago and installed it.
Didn't do any testing or anything (I'm not a Java person), so whether it
all works is another issue.
Oh, I remember now. Have a look at /usr/ports/lang/kaffe.
|
cross
|
|
response 97 of 124:
|
Dec 21 18:11 UTC 2006 |
Regarding #94; That can be relative. For instance, on the Sun4, staff
depended on a custom command to edit password information, because the
password stuff was so hacked. But, the standard commands are better;
we made a net gain by leaving behind the old stuff we *had* depended on
and moving to a newer system. Someone definitely deserved a Krispy
Kreme on that one.
|
maus
|
|
response 98 of 124:
|
Dec 21 19:17 UTC 2006 |
Krispy Kreme? Ewwww!!!! Give me a nice Shipley's or a Dunkin' Donuts any
day.
giggle
|
cross
|
|
response 99 of 124:
|
Dec 21 19:27 UTC 2006 |
Blasphemy!
|
cmcgee
|
|
response 100 of 124:
|
Dec 21 19:43 UTC 2006 |
Colleen hands out hot fresh Krispy Kremes to Cross, and warm, sugary Dunkin
Donuts to maus.
Any other staff members wanting special donuts should post here. If donuts
were all it took to keep staff motivated, I'd take on the whole job myself.
|
gelinas
|
|
response 101 of 124:
|
Dec 22 02:00 UTC 2006 |
Someone, somewhere, mentioned vipw not working. Today, while not thinking of
anything important (or maybe it was yesterday; they all run together any
more), I realised why vipw is a bad idea on grex: newuser is updating the
password file fairly often. Since "last write wins," it's possible to
_really_ screw things up with two things modifying the password file at the
same time.
I'm hoping that "moduser" will either lock the password file or be agile
enough not to interfere with newuser.
|
cross
|
|
response 102 of 124:
|
Dec 22 02:05 UTC 2006 |
vipw didn't work on old grex; it's all right on grex now; it does proper
locking of the passwd database as well rebuilding the DB files after changes.
If newuser is conflicting with it, then that's because newuser is broken.
|
remmers
|
|
response 103 of 124:
|
Dec 22 18:27 UTC 2006 |
A cursory glance at the newuser source code indicates that it's using
pw_mkdb(), pw_lock(), and pw_abort() to do its password file updating, so
I suspect things are probably ok.
|
cross
|
|
response 104 of 124:
|
Dec 22 18:29 UTC 2006 |
If it's calling those routines, then you're right; it's doing the same locking
that vipw is doing (essentially) and therefore vipw and newuser will play
nicely with each other (as will useradd, moduser, etc).
|
spooked
|
|
response 105 of 124:
|
Dec 23 09:50 UTC 2006 |
For those hardware-inspired among you, please lend me your advice in item
30 of the Garage conference.
|
maus
|
|
response 106 of 124:
|
Jan 5 22:04 UTC 2007 |
Just for another thought, do we want to investigate something like
http://www.lacie.com/products/product.htm?pid=10876 as a NAS solution
instead of putting drives into the server itself? It looks fairly
inexpensive, and is very pretty. A good gigabit NIC that is known to
work well with OpenBSD is dirt cheap (under 50$ most of the time) and we
could grow storage space as needed by adding additional units. Lacie has
a pretty good reputation, especially amongst Apple people. We would also
probably want to see if Lacie guarantees the drives inside their device.
According to what i have seen, it looks like this would provide us
full-speed RAID 5 + hot spare if we wanted it.
Do we want to think about this or stay with the idea of drives inside
the server?
|
nharmon
|
|
response 107 of 124:
|
Jan 5 22:13 UTC 2007 |
I think drives inside a server would be best in order to minimize our
physical footprint in case we ever need to move to a rack.
|
scg
|
|
response 108 of 124:
|
Jan 6 08:22 UTC 2007 |
Hi. I'm doing my very occasional look through Grex, and saw this.
I'm a network person, not a systems person, so I'm way over my head when
talking about specific systems components. That said, I do manage a lot
of "critical infrastructure" type services on a low budget with a small
staff, so I spend a lot of time thinking about how to keep services
reliable while also cheap and easy to operate. And, for those of you
who are new in the last few years and don't know me, I'm a former Grex
staff and board member.
I'm a big fan of systems where the answer to a problem is to turn off
the malfunctioning component, and the users don't notice. For that
reason, I like that hardware RAID systems and the like are being
discussed. For partitions with lots of dynamic data that needs to stay
up to date, like the conferences, RAID or some equivalent is absolutely
the right way to go.
I also like that this is being thought about now, at a time when I
gather Grex has been relatively stable. "If it ain't broke, don't fix
it" and "the number one cause of network outages is network engineers"
are both appropriate rules to keep in mind, but if you've got something
that looks really likely to fall apart it is easier to fix it before it
becomes an emergency.
However, there are a few other things about this discussion that worry
me. Doing piecemeal upgrades to several year old hardware seems like a
good way to run into unexpected incompatibilities. Using internal RAID
enclosures with the idea of moving them to as yet unspecified new
hardware seems like a big loss of flexibility.
If I were specing this out, and if it could fit within the budget,
here's what I would probably do:
Get a networked disk array, such as the one maus talks about in #106
(rack mountable, consuming as few rack units as possible), and put all
the dynamic data on it. Then get a couple of 1U servers, standard and
self contained, with serial consoles and ideally some sort of "lights
out manager" thing. Put the static non-changing stuff on the internal
hard drives, and set them up as clones of each other. Add in a cheap
Ebay-purchased console server to manage it. If the applications support
it, run the two 1U servers side by side, accessing the same data off the
RAID array and sharing the load; otherwise, keep one as a hot spare.
If one of the disks in the RAID array fails, pull and replace it. If
one of the servers fails, turn it off and run on the other one.
I suspect you could fit this all in six U or so of rack space, which is
still not huge.
That said, I'd also question somewhat whether Grex should still be in
the hardware business. It might be worth looking at some of the
"dedicated server hosting" companies, and see how what they're charging
to rent a server that includes colo space, network connectivity, and
hands on hardware support, compares to what you're paying ProvideNet.
|
mary
|
|
response 109 of 124:
|
Jan 6 12:33 UTC 2007 |
Steve, nice to see you here and thanks for taking the time to jump in to
the discussion.
I'm especially interested in hearing any comments on your last paragraph.
For a while now I've been thinking about ways to get off of our own
hardware.
|
nharmon
|
|
response 110 of 124:
|
Jan 6 13:08 UTC 2007 |
It has been a while since I have priced it out but if I remember
correctly, ISP-provided servers are either very expensive, or
virtualized. So I'm not sure if it would be workable.
|