We discussed this a bit at the nextgrex meeting last night, but my
thinking wasn't very clear at the time. Too long since I last looked at it.
With the new OpenBSD Grex, we will have to do OS upgrades frequently.
Opinions vary between every 6 months and every 18 months. How do we do
them? We discussed two alternatives:
(1) Spare identical Grex computer. Configure it, swap it in, let the
old one be the new spare. This is expensive.
(2) Take Grex off-line, load new OS release on top of old one. This
costs down time, risks loss of data on /etc, generally annoying.
I'd like to advocate an in-between approach:
(3) Spare disk. We need enough extra high-performance disk to hold
an extra set of system particitions - /, /usr, /etc, /usr/local
and so on. When we upgrade, we erase the spare partitions, and
do a new install of the new OS release onto those partitions.
We modify them, probably copying over some stuff from the
active partitions. When done, we change the new partitions to
be the default boot partitions, and the old ones become spare.
The advantages are:
- Less down time. Theoretically you could do the whole thing while
Grex is up and running and users are on it. This may not be
convenient. It may be more convenient to take Grex off line while
work is being done. However, if you get the job half done, you
can boot Grex back up on the old OS, and go get a night's sleep.
OS upgrades don't have to be done in marathon sessions.
- Only need to maintain one set of Grex install scripts/directions.
All Grex builds are from scratch. We don't also have to figure
out how to build a new Grex from an old Grex.
- If the new OS install has problems, we have the option of booting
back into the old OS.
The cost is a bit of extra disk. Since we are talking high-performance
SCSI for these partitions, that's not totally cheap, but I think it's a
good deal.
24 responses total.
I have another proposal: keep the system source online, and rebuild the system from source periodically. If you track one of OpenBSD's stable release branches, this is more like throwing in a vendor patch than doing a brand-new-system-installation.
We plan on doing that anyway, it's when we must move from one stable branch to another (as OpenBSD obsoletes stable branches more than two revs old) that the two disk system comes into play. Personally, I've used a two disk system for upgrading my desktop for a while now. It worked great. Theoretically, you don't even need two sets of SCSI disks, as you can build the new system on cheap IDE disks, and then blow them over the real SCSI disks when you're sure of the results of the install. I've done it both ways, and the two-live-disks system is slightly better, but the cheap-build-disk system is certainly cheaper.
Now linked to Coop as Item 175; Garage 146
#0 sounds like a very clever solution to a knotty problem. Would a removable drive enclosure create reliability issues for the new box? If not, using one has a further advantage: someone can hack away on a new system setup at home, install it on the drive, test it out, and just swap drives at Grex and reboot.
The reliable way to do removable drives would be hot swappable drives. From a system security standpoint, it's much better to do system builds and updates on a 2nd system, then ship built binaries over to the main system. Building binaries on the main system creates an opportunity for vandals to sneak in their own local hacks. Recording a snapshot of what was built also makes it possible to more quickly restore the system. In general, openbsd updates come in 2 forms, "compatible" upgrades, and "incompatible" upgrades. Compatible upgrades can be done relatively transparently. New stuff fixes bugs or enhances existing APIs. Old binaries continue to run. Incompatible upgrades break everything - anything that is a binary has to be rebuilt, and the question becomes not what changed, but what remained the same. Just as a "for instance", the current i386 openbsd is still a.out based, but there's a chance it could become elf within the next few years. The use of "propolice" is another good example of why "incompatible" upgrades happen. "Compatible upgrades" are obviously easier than "incompatible" ones. There are going to be 3 necessary steps for doing updates: (1) dealing with etc (2) dealing with /usr/local (3) dealing with any system mods we make that aren't just etc hacks. Of the 3, dealing with etc is the most obvious one. "etc" is the package that contains most of what lives in /etc and /var. These are mostly configuration files, some of which are relatively canned things we don't want to change, and some of which are explicitly hooks for us to customize, and some of which contain both. /etc/master.passwd is the outstanding example - the bulk of this will obviously consist of our own entries, but the start contains system IDs. Dealing with /usr/local - this is where things get really messy. OpenBSD packages build to go here. We will likely also want to build our own things to go here. There can be complicated interdependencies. Packages can and often are updated independently of the core OS. Some of those updates may be important to us. Local system mods may not be hardest to handle, but they'll be hardest to anticipate. One sample we might have: safefork, which is a modification to fork to deal with fork bombs. This could go any of 3 ways: a loadable kernel module - potentially the most modular of the 3, but lkms aren't well supported in obsd. A patch that we apply at build time to the kernel source. Obviously this precludes using precompiled kernel binaries, but gives us the most flexibility. Convince the obsd developers to adapt "safefork" as a standard feature. In this case it may even be likely, but it's not certain.
Regarding safefork; if you made use of it controllable via a sysctl parameter, I think the OpenBSD people would be very receptive to including it. I personally think that local mods *will* be the hardest to handle, which is one reason why I'm pushing so strenuously for non-hacked software. As for /usr/local; most of what grex has installed in /usr/local now is either (a) included in the base OpenBSD distribution (e.g., a lot of the GNU utilities), or (b) available via the ports collection. Those few things that are specific to grex that cannot be maintained via ports should, I believe, be installed in a seperate directory (I suggest a /grex or /local filesystem for such things, and then using encap to maintain them).
Tell us about encap, in 25 words or less.
Encap is a methodology for maintaining local software; it uses symbolic links to point, e.g., /usr/local/bin/foo to /usr/local/encap/pkg-current/foo where /usr/local/encap/pkg-current is a symlink to /usr/local/encap/pkg-version.
(twenty-five even. nice job!)
Thanks. btw- encap is the approach used in Brian Ward's book, ``The Linux Problem Solver,'' which I've always recommended to sysadmins as an excellent reference (regardless of whether you're using Linux). Just about every large site I've ever seen uses some tricks with symbolic links to manage large installations of local software. There's a slight performance penalty, but on grex's new hardware, I really don't think that'll matter. And besides, most things will come from ports; only a few smaller utilities would use encap.
Before the development machine croaked, I tried the "build on another system and move the binaries over" thing a couple times with a couple different packages. I didn't like it one bit. One of the packages was mysql. It was a "configure; make; make install" kind of thing (except for all the SunOS incompatibilities I had to fix). "Make install" runs a long time, creating all sorts of library files, directories, config files, header files, you name it, all with appropriate permissions and ownerships, in many different directories. The output from "make install" is rarely totally illuminating, so figuring out exactly what it installs where is not simple. So how do I "move the binary over"? That may work for Picospan, but few modern packages are quite that simple. Backtalk I can mostly manage, but only because I know it so well. In the end, with mysql, I tar'ed up the entire build directory after doing the 'make', ftp'ed it over, unpacked it, and run "make install" on Grex. Simple enough, but hardly nice - that build directory is gigantic.
Having another disk that you physically manipulate the base operating system on could be handy. That is, build a new, stable release on another machine, tweak it to make sure it works right, and then move the physical disk into the `real' grex machine. Of course, you'd then expect that most of the data partitions would live on disks seperate from what the operating system as installed on.
a
Most modern packages (including mysql) have various options for where
to install, and where the installed copy finds bits of itself. Most
commonly, there is a
configure --prefix /run-time/path
option to configure to tell it where it lives once installed, and at
install time you can say
"make DESTDIR=/readwrite/path install"
to tell it to to prefix all the runtime pathnames with
"/readwrite/path". This is commonly done for packages installed in
AFS, for instance, where the run-time path may point to a read-only
path that can't be written directly. This is how most gnu utilities
work, if they're properly configured (not always the case), and how
many other things can be made to work (depending on how faithfully they
follow the model.) (Gnu configure scripts also allow you to segment
and additionally customize prefixes for various parts of the package,
but that's an unimportant point of confusion.) Mysql is packaged like
most gnu software, so knows about both
--prefix (mentioned in the documentation) and DESTDIR (which they
somehow omitted.) Even the documented --prefix alone can be easily
used to identify which bits are supposed to be installed. For most
generic distributions, this can certainly be be made to work, modulo
some work which is often slight.
For packages that are complicated but somehow managed to omit the
customary install options, a simple alternative on a guinea pig system
would be to just install it, then use some tool like tripwire or aide
to find out what changed.
If you just install everything in "/usr/local" (which is how most of
these things work) you can end up with filename collision problems, and
package A breaking package B. This is especially problemmatical when
installing multiple versions of the same package. There are lots of
ways to solve this, but they generally involve installing the thing in
its own per-package directory, then making symlinks to some more
central place to publish its existance. In solaris 2, sun has
installed many of its things in /opt .
For OpenBSD, many things, including mysql, are available as precompiled
packages. Source is available via the "ports" stuff. The nice thing
about the precompiled packages is that somebody else has worried about
filename collisions, obsd configuration and patching, and also
installing and removing the thing. This can greatly simplify life,
unless you *really* need bleeding edge stuff or some arcane option that
isn't available via the packages, and are willing to endure the pain.
The source that's available via the ports includes the necesary bits to
compile it locally and turn it into a package that can then be
installed on a different system.
It should be fairly easy to take random source and turn it into a
package. I haven't tried this so can't speak from personal experience,
but the package management bits mostly seem to include local patches
and some wrapper stuff to decribe which files get installed & such.
This doesn't look hard to reproduce.
Backtalk is one of the few things where this would not be sufficient,
mostly because of anything associated with authentication that
might need to get done in httpd.conf .
It should be noted that, for the option in Marcus's second paragraph (basically encap) there are several tools for managing trees of symlinks. Stow is a good example of one, for instance.
Dan probably meant my 3rd paragraph ("/usr/local") not my 2nd ("how to
figure out what components get installed by package X".)
Yes, that's correct. Sorry for the confusion.
This topic is still one that I feel Grex staff lacks a solution for. This is a major problem.
Call in the feds!
I could write a php interface that would be twice as good as backtalk. I could also built a java client for party. I could bring Grex into the 21st century. This interface is from the 90's See an example of my work here: http://www/antionline.com You guys have no vision.
Go ahead. I judge people's abilities by what they do, not what they say. And this interface is not from the 90s. It's from the 80s.
re #19 - if you can't even format a link properly, I'd be a little scared to try any software you write.
TROGG IS DAVID BLAINE
Having looked at the http://www.antionline.com/ webpage, I will say that I respect the poster's technical abilities, but would not want to see that as the default interface. I like the simple, uncluttered look of backtalk. Having to deal with a metric assload of information in a day, it is a nice rest for the eyes to see something straight-forward, in which information flows linearly, rather than coming at me from all sides. And yes, a metric assload is roughly 1.0296 imperial assloads. You can do the conversion into absolute terms yourself.
I also *much* prefer backtalk to any of the other forum systems in wide use. I wish other places would convert. Please get them to. ;)
You have several choices: