Grex software, maintainability, and libgrex.

Feb 12 22:17 UTC 2005

It seems to me that we've developed quite a bit of software on grex
over the years.  Some of it is good, some not so good.  One thing
that I realized when rewriting the watch utility for the nextgrex
transition is that it is very beneficial to periodically review and
rewrite software applications.  This prevents code-rot and eases
the job of forward maintainability.

There was stuff in watch, for instance, dating back to System III
Unix.  These were the days prior to reasonaly efficient malloc()
implementations, prior to standard dynamic sorted container libraries,
prior to decent signal handling.  A lot of the stuff that got ripped
out had made watch brittle and difficult to move to more modern
systems, and paradoxically, a lot of that was kept in version mnet
ported to FreeBSD!  Going back and rewriting it to move to the new
system gave us a chance to fix some problems and update it to use
standard interfaces.  it was an instructive exercise to see how we
could modify a program that's evolved over many years.

And watch is a program that's fewer than 400 lines of code!

I think we might benefit from more of this sort of thing on grex.
I'd like to review the software we use, and selectively rewrite
portions of it.  We should take what we have that's custom, and
remove the use of old interface, standardize things like signal
handling and command line option parsing, etc.  We should work on
improving testability and maintainability, and in the process we
might fix a couple of stupid bugs.  We should also identify segments
of code that are similar between different programs and try to
extract those into reusable components that we can put into a
library.  This will improve maintainability of our code and make
developing new code a bit easier.

Some specific things I'm thinking about:

        * Examining our local C programs looking for things like
          argument processing loops, signal handling, etc, and
          making these fit standard interfaces (getopt(3), signal(2),
          etc).  There's a lot of historical baggage in there.
        * Changing error reporting code to use standard interfaces.
          BSD has err(3); let's use it!  If it's not on another
          platform, we can build a facade for it.
        * Modifying whatever local daemons we have to use standard
          logging interfaces, etc.
        * Looking at the command line arguments we pass to local
          programs and standardizing them.  Make sure each program
          can spit out a useful `Usage' message, and so on.
        * Try to find segments of code that are repeated frequently
          that we can put into a library for re-use.
        * Building Unit tests for library functions and functions
          in largish applications.
        * Building regression testing frameworks for programs and
          applications.
        * Having a consistent build procedure for our local programs.
          E.g., standard Makefile targets and structure, etc.
        * Building standard templates for writing new programs,
          Makefiles library functions, etc.

To this end, I've started working on a small library of useful
functions that I call, `libgrex'.  It's currently in my directory
under ~cross/src/libgrex.  As part of this, I found and modified
a very small unit testing tool called `minunit' and have written
some unit tests for some of the functions in libgrex.

Is anyone else interested in this effort?  If so, should discussion
stay here in the conferences or should I create a mailing list?

It'd be nice to get more of the community involved in this effort.

response 2 of 36:

Feb 13 02:26 UTC 2005

I also am interested.  I'm sadly out of practice in coding; I haven't
done much in five years or so.

response 4 of 36:

Feb 13 02:53 UTC 2005

That's all right; you don't need to be a wizard to fix up some of grex's
utilities.

Some places to start: look over the existing libgrex code.  Let me know
if I've made any errors or if you have any additions (hey, I'm not a
saint either!).  Read up on some of the `standard' programming interfaces:

        err(3)
        getgrent(3)
        getopt(3)
        getpwent(3)
        printf(3)
        signal(3)
        string(3)
        tsearch(3)

All of these things conform to some sort of Unix standard, such as
Unix98 or the Single Unix spec; err(3) is an exception, but it's
easy to implement those functions in a compatibility layer if we
ever want to move libgrex to another machine.  Note that I only
mention printf(3) because there's a lot of `puts' cruft in our
source code, and I put in string(3) because there seems to be a lot
of hand-rolled stuff for skipping delimiters and parsing apart
strings, etc.  It'd be nice to replace that stuff with standard
functions like strsep(3), strcspn(3), etc.

Read the `libgrex/test/minunit.h' header file that describes the
unit testing framework (the comments give a terse discussion of how
to use it, and there are plenty of examples in the libgrex/test
subdirectory).  Write yourself some simple tests and suites to get
a hang of how it works; see libgrex/test/skeltest.c,
libgrex/test/skelsuite.c, and libgrex/test/testrunner.c as well
as the Makefile in libgrex/test.

Also read up on how to use the `doc++' documentation framework.
I'm using this for libgrex; it's not terribly complex, and looks
sort of like Javadoc but works on C and C++ code, too.

We need to come up with a standard for code layout with respect
to spacing, where to put the braces, etc.  I'm reasonably satisfied
with the output of the indent(1) tool for this.  It's sort of ugly,
and not precisely how I'd do it, but at this point I think
consistency is more important than individual preference.  Either
use indent or do it by hand, but make it look reasonably close.

Then start combing through /grex/grexdoc/misc/src looking for things
that you're interested in working on.  Some initial suggestions for
things that could be updated to be much simplifed or just generally
improved are the members and twit commands.  A lot of commands have
their own argument processing loops (complete with goto's to break
out of them) and don't use getopt(), which would be nice to fix.

Fortunately, most programs can also be tested in isolation, without
affecting the rest of the systems.  Once you've got something
working, put a notice here or send me an email.  I'll look over
your changes and put it into production if it looks good.

Note that this last step is rather cumbersome.  I'd much rather
have a shared source repository where multiple people, not necessarily
staff, can commit.  Part of that might mean divorcing our source
base from the grexdoc change management system that Jan put together.
Grexdoc is great for controlling changes to grex's configuration;
it might not be so great for controlling changes to grex's software.
This is something that needs a bit more thought and discussion, but
right now, I'd like to see something like a Subversion repository
that grexdoc hooked into to install grex's locally written and
maintained software.  As part of this, I might move libgrex under
another directory, called `grexsoft' that includes some source files
from elsewhere on the system.

Eventually, I'd like to wrap the scripts in /usr/local/grex-scripts
into this effort as well.  There's a lot of SunOS cruft left in those
(it's outstanding the number of places that /usr/ucb appears in
those files!), but the C stuff is enough of a start.

response 6 of 36:

Feb 13 05:53 UTC 2005

Ok, so we need CVS for maintaing the code tree. I'd like to sign up but
only if all code is GPL. Is it?

response 8 of 36:

Feb 13 21:57 UTC 2005

Another task that needs to be accomplished is writing man pages for
the grex utilities.  This is going to require reading the source code
to them, figuring out what they do, and then writing the troff -man
source itself.  Note: if you write a manpage, use -man and not -mandoc,
which isn't easily portable.

response 10 of 36:

Feb 14 16:39 UTC 2005

Well, if you write something new, you can certainly put it under
the GPL.  I don't see why libgrex can't be GPL'ed, for that matter.
On the other hand, a bit of the Grex software already has a BSD
style license (most things by Marcus Watts, for instance).  I'm not
too into license holy wars, and I don't care if someone uses the
source code for a closed-source thing or not, but that's just me.

response 12 of 36:

Feb 16 20:42 UTC 2005

I put grexsoft into a CVS repository under /a/n/c/ncvs.  I'm work on
it more later, but presently, I can commit from multiple places (which
is a lot nicer than what I had been doing.

response 14 of 36:

Feb 17 03:59 UTC 2005

I added a couple of new functions to libgrex: warning() and fatal().
These do basically what their names imply: print error-style messages
to standard output, and fatal() calls exit with EXIT_FAILURE.

Each function works like printf(): you can specify variable arguments
using normal printf(3) style verbs.  Additionally, if the verb "%r"
appears in the format string, it is replaced with the contents of
`strerror(errno)' for the current value of errno.  %r can appear
anywhere in the format string, and may appear multiple times.
Finally, if you've called the libgrex function `progname()' and it
has returned a meaningful value, the error message will be proceeded
by the return value of progname() and ": ".

Note that format specifiers to warning() and fatal() need not include
a trailing newline: that will be automatically appended, just like
in the perror() function.

warning() and fatal() have to possibly interpolate other strings
into the passed-in format string.  They try to do this in an
intelligent way that will minimize potential errors - after all,
your error reporting routine erroring out and producing no output
doesn't do you much good.  If they fail in their interpolation
attempts, they make a best-effort attempt to print out your error
message, but some %r verbs might not be interpreted correctly.

Typical uses might be:

        /* ... Say this is invoked as: /usr/local/bin/myprog */
        const char *OUTPUT_FILE = "/etc/output";
        prog = progname(argv[0]);
        while ((file = *++argv) != NULL) {
                fp = fopen(file, "r");
                if (fp == NULL) {
                        warning("Failed to open %s: %r", file);
                        continue;
                }
                /* ... */
                fclose(fp);
        }
        outfp = fopen(OUTPUT_FILE, "w");
        if (outfp == NULL)
                fatal("Can't open %s for writing: %r", OUTPUT_FILE);
        /* ... etc etc etc */

If one of the input files couldn't be opened and you couldn't write
to /etc/output, you would see the following errors:

myprog: Failed to open /some/file: No such file or directory.
myprog: Can't open /etc/output for writing: Permission denied.

And the program would exit after the printing the second message.
Exit status would be whatever EXIT_FAILURE is defined to (usually 1).

response 16 of 36:

Feb 27 05:34 UTC 2005

I rewrote the twit filter and installed it today.  The original had
a lot of garbage like, `#define then {' in it, and was overly complex
for no good reason.  It also had a lot of pre-ANSI C'isms in it that I
removed, like forward declarations of functions inside of other
functions and stuff like that.  Ick.

I also have an awk version of the filter that's less than 15 lines
long.  The C version shrank to something like 60 lines.

response 18 of 36:

Feb 27 09:21 UTC 2005

I've configured CVS to log its commit messages to BBS.  See item
9 in garage.

response 20 of 36:

Mar 16 21:04 UTC 2005

I installed the X11 programmer suite here on grex, and reinstalled
doc++ and DejaGNU.  DejaGNU is actually quite complex, so might be
overkill.  I'm investigating simple regression level testing
frameworks, but there isn't much out there.  We'll probably have
to build something.

This and documentation are the big priorities right now.  If you'll
notice from the CVS commit messages logged in item 9, many of the
things I'm checking in are fixes to little bugs I'm introducing.
This is due primarily to three reasons:  1) I'm human and I make
mistakes, especially with something I don't have too much time for.
2) Most grexsoft commands are undocumented, and it's not always
easy to figure out what the expected output for them should be.
3) I'm not testing commands after building them, generally, because
there's no cheap, easy way to automate doing so.

Well, being aware of my own limitations, I've figured out that I've
got to do things somewhat differently.  I've got to structure the
development process a little more efficiently....  How to do this?

I propose the following model for modifying grexsoft code:

        1) Pick a command to change.
        2) If it's not already documented, document it.
           Write a man page for it describing what it's
           supposed to do.  If documentation exists and
           is wrong, tweak it.
        3) Write a regression test for it.
        4) Peak through the source code and try and
           document that.
        5) Write unit-level tests for the major components
           inside the source (this may not be applicable).
        6) Refactor.
        7) Iterate from step 2.

In this model, documentation describes function and comes first,
followed by testing.  Testing is defined by function and drives the
process.  The process iterates, and development and refactoring and
refactoring are repeated throughout the entire procedure in a process
of successive refinement.

Now, we just need to come up with a regression testing framework.
Something like this looks promising:

        http://diveintopython.org/regression_testing/

Except that we have a lot more perl hackers on grex than python types,
I think.  It'd be interesting to take something like that and adapt it
to Perl and use it here.

response 24 of 36:

Sep 21 21:24 UTC 2005

Hm, I use a 1440x900 display, and the pistachio buttons are plenty large
enough for me.

There are Unixes without mkdir()???