|
Grex > Oldcoop > #380: Cyberspace Communications finances for November 2006 | |
|
| Author |
Message |
| 25 new of 124 responses total. |
maus
|
|
response 92 of 124:
| 
Dec 21 16:43 UTC 2006 |
That's a fair assessment. Consider my mumblings about the bottleneck the
ramblings of a weary mind, and please ignore said mumblings.
So, silly question: if we are thinking about moving the system-space
onto a new disc subsystem, does this mean a fresh, new installation? Can
we use the opportunity to request new commands to be added and to
implement new controls and move to standards from odd Grex-isms?
|
cross
|
|
response 93 of 124:
|
Dec 21 17:01 UTC 2006 |
No, not at all; I think it's good to be challenged and be asked to justify
one's conclusions. I thank you for that.
I think you can always request the installation of additional software. And
yes, I *do* think it would mean a new installation of the basic system. But,
that might not be a bad thing. Any opportunity to move to standard commands
from weird customizations is a plus, in my opinion.
|
maus
|
|
response 94 of 124:
|
Dec 21 17:15 UTC 2006 |
I agree that moving to standards would be a good thing, provided nothing
is broken in the process (if a command that users or staff depend on is
broken by the move to standardize, then the standardization is crap; if
no-one is hurt and we make the system easier to maintain and easier to
upgrade and actually match what the man pages and web-pages say, then we
have done a good thing by standardizing and deserve doughnuts).
I didn't have specific commands or software in mind (or, at least,
nothing appropriate for this system), but I figured that if we were
facing a fresh installation, this would be the time to ask people what
commands they would like to see on here, and also see if there are
commands that users would want to see upgraded or replaced.
|
maus
|
|
response 95 of 124:
|
Dec 21 17:29 UTC 2006 |
On the new commands front, I was just looking through /usr/local/bin and
noticed javac and java and jar. I thought the port of native java to
OBSD was still a couple of years away. Did we build this by way of RHEL
emulation or something else entirely? Have we published somewhere how we
managed it? Hurray f
|
remmers
|
|
response 96 of 124:
|
Dec 21 17:44 UTC 2006 |
Hmm... I seem to recall that I saw the Java stuff sitting in either the
OpenBSD ports or packages collection a few months ago and installed it.
Didn't do any testing or anything (I'm not a Java person), so whether it
all works is another issue.
Oh, I remember now. Have a look at /usr/ports/lang/kaffe.
|
cross
|
|
response 97 of 124:
|
Dec 21 18:11 UTC 2006 |
Regarding #94; That can be relative. For instance, on the Sun4, staff
depended on a custom command to edit password information, because the
password stuff was so hacked. But, the standard commands are better;
we made a net gain by leaving behind the old stuff we *had* depended on
and moving to a newer system. Someone definitely deserved a Krispy
Kreme on that one.
|
maus
|
|
response 98 of 124:
|
Dec 21 19:17 UTC 2006 |
Krispy Kreme? Ewwww!!!! Give me a nice Shipley's or a Dunkin' Donuts any
day.
giggle
|
cross
|
|
response 99 of 124:
|
Dec 21 19:27 UTC 2006 |
Blasphemy!
|
cmcgee
|
|
response 100 of 124:
|
Dec 21 19:43 UTC 2006 |
Colleen hands out hot fresh Krispy Kremes to Cross, and warm, sugary Dunkin
Donuts to maus.
Any other staff members wanting special donuts should post here. If donuts
were all it took to keep staff motivated, I'd take on the whole job myself.
|
gelinas
|
|
response 101 of 124:
|
Dec 22 02:00 UTC 2006 |
Someone, somewhere, mentioned vipw not working. Today, while not thinking of
anything important (or maybe it was yesterday; they all run together any
more), I realised why vipw is a bad idea on grex: newuser is updating the
password file fairly often. Since "last write wins," it's possible to
_really_ screw things up with two things modifying the password file at the
same time.
I'm hoping that "moduser" will either lock the password file or be agile
enough not to interfere with newuser.
|
cross
|
|
response 102 of 124:
|
Dec 22 02:05 UTC 2006 |
vipw didn't work on old grex; it's all right on grex now; it does proper
locking of the passwd database as well rebuilding the DB files after changes.
If newuser is conflicting with it, then that's because newuser is broken.
|
remmers
|
|
response 103 of 124:
|
Dec 22 18:27 UTC 2006 |
A cursory glance at the newuser source code indicates that it's using
pw_mkdb(), pw_lock(), and pw_abort() to do its password file updating, so
I suspect things are probably ok.
|
cross
|
|
response 104 of 124:
|
Dec 22 18:29 UTC 2006 |
If it's calling those routines, then you're right; it's doing the same locking
that vipw is doing (essentially) and therefore vipw and newuser will play
nicely with each other (as will useradd, moduser, etc).
|
spooked
|
|
response 105 of 124:
|
Dec 23 09:50 UTC 2006 |
For those hardware-inspired among you, please lend me your advice in item
30 of the Garage conference.
|
maus
|
|
response 106 of 124:
|
Jan 5 22:04 UTC 2007 |
Just for another thought, do we want to investigate something like
http://www.lacie.com/products/product.htm?pid=10876 as a NAS solution
instead of putting drives into the server itself? It looks fairly
inexpensive, and is very pretty. A good gigabit NIC that is known to
work well with OpenBSD is dirt cheap (under 50$ most of the time) and we
could grow storage space as needed by adding additional units. Lacie has
a pretty good reputation, especially amongst Apple people. We would also
probably want to see if Lacie guarantees the drives inside their device.
According to what i have seen, it looks like this would provide us
full-speed RAID 5 + hot spare if we wanted it.
Do we want to think about this or stay with the idea of drives inside
the server?
|
nharmon
|
|
response 107 of 124:
|
Jan 5 22:13 UTC 2007 |
I think drives inside a server would be best in order to minimize our
physical footprint in case we ever need to move to a rack.
|
scg
|
|
response 108 of 124:
|
Jan 6 08:22 UTC 2007 |
Hi. I'm doing my very occasional look through Grex, and saw this.
I'm a network person, not a systems person, so I'm way over my head when
talking about specific systems components. That said, I do manage a lot
of "critical infrastructure" type services on a low budget with a small
staff, so I spend a lot of time thinking about how to keep services
reliable while also cheap and easy to operate. And, for those of you
who are new in the last few years and don't know me, I'm a former Grex
staff and board member.
I'm a big fan of systems where the answer to a problem is to turn off
the malfunctioning component, and the users don't notice. For that
reason, I like that hardware RAID systems and the like are being
discussed. For partitions with lots of dynamic data that needs to stay
up to date, like the conferences, RAID or some equivalent is absolutely
the right way to go.
I also like that this is being thought about now, at a time when I
gather Grex has been relatively stable. "If it ain't broke, don't fix
it" and "the number one cause of network outages is network engineers"
are both appropriate rules to keep in mind, but if you've got something
that looks really likely to fall apart it is easier to fix it before it
becomes an emergency.
However, there are a few other things about this discussion that worry
me. Doing piecemeal upgrades to several year old hardware seems like a
good way to run into unexpected incompatibilities. Using internal RAID
enclosures with the idea of moving them to as yet unspecified new
hardware seems like a big loss of flexibility.
If I were specing this out, and if it could fit within the budget,
here's what I would probably do:
Get a networked disk array, such as the one maus talks about in #106
(rack mountable, consuming as few rack units as possible), and put all
the dynamic data on it. Then get a couple of 1U servers, standard and
self contained, with serial consoles and ideally some sort of "lights
out manager" thing. Put the static non-changing stuff on the internal
hard drives, and set them up as clones of each other. Add in a cheap
Ebay-purchased console server to manage it. If the applications support
it, run the two 1U servers side by side, accessing the same data off the
RAID array and sharing the load; otherwise, keep one as a hot spare.
If one of the disks in the RAID array fails, pull and replace it. If
one of the servers fails, turn it off and run on the other one.
I suspect you could fit this all in six U or so of rack space, which is
still not huge.
That said, I'd also question somewhat whether Grex should still be in
the hardware business. It might be worth looking at some of the
"dedicated server hosting" companies, and see how what they're charging
to rent a server that includes colo space, network connectivity, and
hands on hardware support, compares to what you're paying ProvideNet.
|
mary
|
|
response 109 of 124:
|
Jan 6 12:33 UTC 2007 |
Steve, nice to see you here and thanks for taking the time to jump in to
the discussion.
I'm especially interested in hearing any comments on your last paragraph.
For a while now I've been thinking about ways to get off of our own
hardware.
|
nharmon
|
|
response 110 of 124:
|
Jan 6 13:08 UTC 2007 |
It has been a while since I have priced it out but if I remember
correctly, ISP-provided servers are either very expensive, or
virtualized. So I'm not sure if it would be workable.
|
maus
|
|
response 111 of 124:
|
Jan 7 00:40 UTC 2007 |
Working for a hosting provider (and having worked for others and knowing
the workings of lots of them), I would definitely say be cautious.
Hosting providers are notorious for using cheap (often recycled) gear,
and virtually every service incurs a nontrivial charge. Most of these
places have minimally skilled staff that try to make everything that
happens the direct result of something the customer did (even
catastrophic hardware failure) so that it is not their problem and is
something that they can charge an arm and leg to fix. Additionally,
most hosting providers will not run phone lines for you.
Just as a single datapoint, consider Layered Technologies (one of the
larger players in the hosting business):
- Celeron 2 GHz (their lowest spec chassis)
- 2 GBytes non-ECC Reg RAM
- 2 x 200 GByte Serial ATA hard drive
- Serial ATA RAID controller (set up in a mirror set)
- OpenBSD
- No control panel
- a /29 netblock
- 10 Mbit/sec uplink
- 1500 GBytes total network throughput per month
This would run about 214$ per month
I do think that scg has some good insight, and his cluster spec is
sound.
|
maus
|
|
response 112 of 124:
|
Jan 7 00:44 UTC 2007 |
resp:108
scg,
Just curious how you handle high-availability. I have looked over a few
solutions, though i have only ever implemented them in a lab setting, so
I am not sure how they would behave in the wild. I particularly like:
- HSRP/CARP with synchronization data sent over a dedicated seperate
LAN - Round-robin NAT with IP-takeover (and using IPMI/LOM to "shoot
the
other guy in the head")
I am sure there are much better solutions, and would love to learn them.
|
scg
|
|
response 113 of 124:
|
Jan 7 02:03 UTC 2007 |
My favorite dedicated server hoster is ServePath
(http://www.servepath.com). I designed their network a few years ago,
and they treat me very well. My view that they're doing things right
isn't unbiased. I've also heard good things about RackSpace and
Affinity (http://www.valueweb.com), but I've never actually dealt with
them.
I also wouldn't be so quick to dismiss server virtualization, if you
find a hosting provider you like who is doing that. True, you only get
a fraction of the server's processing power, but the servers will likely
be a lot more powerful than what Grex is running on now. From what I
hear (no direct experience yet), it's hard to tell the difference from a
distance between a real server and a virtual server, and a hosting
provider will probably be far more motivated to avoid or deal quickly
with hardware problems taking down lots of customers' virtual servers
than with issues affecting only a single customer.
How to handle load balancing and failover depends on what you're doing.
Simplest is round robin DNS with a low TTL, perhaps accompanied by a
nanny script that watches to see if one of the servers goes away and
removes its DNS entry. That's what used to be done in the old days
before all those fancy load balancing boxes were invented, and is more
or less what some of the fancy load balancing boxes do.
(What I do at work is to scatter the servers around the world and source
BGP announcements from the servers (google for "anycast"), but that's
not very well suited for what Grex is doing.)
|
maus
|
|
response 114 of 124:
|
Jan 7 02:17 UTC 2007 |
I need to amend my previous statement. I had forgotten about RackSpace;
they have a good rep. I don't know ServPath or Affinity.
|
pfv
|
|
response 115 of 124:
|
Jan 7 20:24 UTC 2007 |
I guess I fail to see what all this "powerful" gains anyone.
|
maus
|
|
response 116 of 124:
|
Feb 14 04:21 UTC 2007 |
While sitting in a hollowed out baseboard and chewing on some wires, I
found an Ultra320 SCSI RAID board in what appears to be new condition.
Do we want me to send it in so that when we reinstall, we can mirror the
root volume on high-speed SCSI drives?
|