It seems to me that we've developed quite a bit of software on grex
over the years.  Some of it is good, some not so good.  One thing
that I realized when rewriting the watch utility for the nextgrex
transition is that it is very beneficial to periodically review and
rewrite software applications.  This prevents code-rot and eases
the job of forward maintainability.

There was stuff in watch, for instance, dating back to System III
Unix.  These were the days prior to reasonaly efficient malloc()
implementations, prior to standard dynamic sorted container libraries,
prior to decent signal handling.  A lot of the stuff that got ripped
out had made watch brittle and difficult to move to more modern
systems, and paradoxically, a lot of that was kept in version mnet
ported to FreeBSD!  Going back and rewriting it to move to the new
system gave us a chance to fix some problems and update it to use
standard interfaces.  it was an instructive exercise to see how we
could modify a program that's evolved over many years.

And watch is a program that's fewer than 400 lines of code!

I think we might benefit from more of this sort of thing on grex.
I'd like to review the software we use, and selectively rewrite
portions of it.  We should take what we have that's custom, and
remove the use of old interface, standardize things like signal
handling and command line option parsing, etc.  We should work on
improving testability and maintainability, and in the process we
might fix a couple of stupid bugs.  We should also identify segments
of code that are similar between different programs and try to
extract those into reusable components that we can put into a
library.  This will improve maintainability of our code and make
developing new code a bit easier.

Some specific things I'm thinking about:

        * Examining our local C programs looking for things like
          argument processing loops, signal handling, etc, and
          making these fit standard interfaces (getopt(3), signal(2),
          etc).  There's a lot of historical baggage in there.
        * Changing error reporting code to use standard interfaces.
          BSD has err(3); let's use it!  If it's not on another
          platform, we can build a facade for it.
        * Modifying whatever local daemons we have to use standard
          logging interfaces, etc.
        * Looking at the command line arguments we pass to local
          programs and standardizing them.  Make sure each program
          can spit out a useful `Usage' message, and so on.
        * Try to find segments of code that are repeated frequently
          that we can put into a library for re-use.
        * Building Unit tests for library functions and functions
          in largish applications.
        * Building regression testing frameworks for programs and
          applications.
        * Having a consistent build procedure for our local programs.
          E.g., standard Makefile targets and structure, etc.
        * Building standard templates for writing new programs,
          Makefiles library functions, etc.

To this end, I've started working on a small library of useful
functions that I call, `libgrex'.  It's currently in my directory
under ~cross/src/libgrex.  As part of this, I found and modified
a very small unit testing tool called `minunit' and have written
some unit tests for some of the functions in libgrex.

Is anyone else interested in this effort?  If so, should discussion
stay here in the conferences or should I create a mailing list?

It'd be nice to get more of the community involved in this effort.

36 responses total.

        I want do to serious homework in this area.
        Will try soon.

        I'm experienced as a maintenance/cleanup coder,
        e.g. with C/unix. 

        Fell free to push me forward. Even with one small area.

I also am interested.  I'm sadly out of practice in coding; I haven't
done much in five years or so.

If you need me to do Java, I'm in

That's all right; you don't need to be a wizard to fix up some of grex's
utilities.

Some places to start: look over the existing libgrex code.  Let me know
if I've made any errors or if you have any additions (hey, I'm not a
saint either!).  Read up on some of the `standard' programming interfaces:

        err(3)
        getgrent(3)
        getopt(3)
        getpwent(3)
        printf(3)
        signal(3)
        string(3)
        tsearch(3)

All of these things conform to some sort of Unix standard, such as
Unix98 or the Single Unix spec; err(3) is an exception, but it's
easy to implement those functions in a compatibility layer if we
ever want to move libgrex to another machine.  Note that I only
mention printf(3) because there's a lot of `puts' cruft in our
source code, and I put in string(3) because there seems to be a lot
of hand-rolled stuff for skipping delimiters and parsing apart
strings, etc.  It'd be nice to replace that stuff with standard
functions like strsep(3), strcspn(3), etc.

Read the `libgrex/test/minunit.h' header file that describes the
unit testing framework (the comments give a terse discussion of how
to use it, and there are plenty of examples in the libgrex/test
subdirectory).  Write yourself some simple tests and suites to get
a hang of how it works; see libgrex/test/skeltest.c,
libgrex/test/skelsuite.c, and libgrex/test/testrunner.c as well
as the Makefile in libgrex/test.

Also read up on how to use the `doc++' documentation framework.
I'm using this for libgrex; it's not terribly complex, and looks
sort of like Javadoc but works on C and C++ code, too.

We need to come up with a standard for code layout with respect
to spacing, where to put the braces, etc.  I'm reasonably satisfied
with the output of the indent(1) tool for this.  It's sort of ugly,
and not precisely how I'd do it, but at this point I think
consistency is more important than individual preference.  Either
use indent or do it by hand, but make it look reasonably close.

Then start combing through /grex/grexdoc/misc/src looking for things
that you're interested in working on.  Some initial suggestions for
things that could be updated to be much simplifed or just generally
improved are the members and twit commands.  A lot of commands have
their own argument processing loops (complete with goto's to break
out of them) and don't use getopt(), which would be nice to fix.

Fortunately, most programs can also be tested in isolation, without
affecting the rest of the systems.  Once you've got something
working, put a notice here or send me an email.  I'll look over
your changes and put it into production if it looks good.

Note that this last step is rather cumbersome.  I'd much rather
have a shared source repository where multiple people, not necessarily
staff, can commit.  Part of that might mean divorcing our source
base from the grexdoc change management system that Jan put together.
Grexdoc is great for controlling changes to grex's configuration;
it might not be so great for controlling changes to grex's software.
This is something that needs a bit more thought and discussion, but
right now, I'd like to see something like a Subversion repository
that grexdoc hooked into to install grex's locally written and
maintained software.  As part of this, I might move libgrex under
another directory, called `grexsoft' that includes some source files
from elsewhere on the system.

Eventually, I'd like to wrap the scripts in /usr/local/grex-scripts
into this effort as well.  There's a lot of SunOS cruft left in those
(it's outstanding the number of places that /usr/ucb appears in
those files!), but the C stuff is enough of a start.

Regarding #3; There's not a lot of Java on grex.  For that matter, we
don't even have a Java compiler....  Hmm.

Ok, so we need CVS for maintaing the code tree. I'd like to sign up but
only if all code is GPL. Is it?

It's not really anything.  It's more or less public domain....

I was thinking Subversion instead of CVS, but that's not a huge
difference.

Hmm, actually, now I am curious what license the grex source code
is under.  Does anyone know?  I suppose it's up to the individual
authors.  For my own contributions, I'd be fine with any open
source or public domain license.  Siddhartha, how would you feel
about a BSD-style or public domain license?

Another task that needs to be accomplished is writing man pages for
the grex utilities.  This is going to require reading the source code
to them, figuring out what they do, and then writing the troff -man
source itself.  Note: if you write a manpage, use -man and not -mandoc,
which isn't easily portable.

Re #7, I prefer good ol' GPL. That way no one can take the code and use
it for a closed-source product. GPL will ensure that wherever the code
is used the source is also distributed alongwith.

Well, if you write something new, you can certainly put it under
the GPL.  I don't see why libgrex can't be GPL'ed, for that matter.
On the other hand, a bit of the Grex software already has a BSD
style license (most things by Marcus Watts, for instance).  I'm not
too into license holy wars, and I don't care if someone uses the
source code for a closed-source thing or not, but that's just me.

Okay, so in an effort to move things along, I've done a slight reorganization
of the directory structure for all this, putting a `grexsoft' directory
under ~cross and putting some commands there.  Currently, `grexsoft'
contains two subdirectories: cmd, and libgrex.  Libgrex is obvious, `cmd'
is the source to some of our local commands.  I'm going to symlink
~cross/src/libgrex to ~cross/grexsoft/libgrex and move my work to there.

I put grexsoft into a CVS repository under /a/n/c/ncvs.  I'm work on
it more later, but presently, I can commit from multiple places (which
is a lot nicer than what I had been doing.

unl ul . lucky

I added a couple of new functions to libgrex: warning() and fatal().
These do basically what their names imply: print error-style messages
to standard output, and fatal() calls exit with EXIT_FAILURE.

Each function works like printf(): you can specify variable arguments
using normal printf(3) style verbs.  Additionally, if the verb "%r"
appears in the format string, it is replaced with the contents of
`strerror(errno)' for the current value of errno.  %r can appear
anywhere in the format string, and may appear multiple times.
Finally, if you've called the libgrex function `progname()' and it
has returned a meaningful value, the error message will be proceeded
by the return value of progname() and ": ".

Note that format specifiers to warning() and fatal() need not include
a trailing newline: that will be automatically appended, just like
in the perror() function.

warning() and fatal() have to possibly interpolate other strings
into the passed-in format string.  They try to do this in an
intelligent way that will minimize potential errors - after all,
your error reporting routine erroring out and producing no output
doesn't do you much good.  If they fail in their interpolation
attempts, they make a best-effort attempt to print out your error
message, but some %r verbs might not be interpreted correctly.

Typical uses might be:

        /* ... Say this is invoked as: /usr/local/bin/myprog */
        const char *OUTPUT_FILE = "/etc/output";
        prog = progname(argv[0]);
        while ((file = *++argv) != NULL) {
                fp = fopen(file, "r");
                if (fp == NULL) {
                        warning("Failed to open %s: %r", file);
                        continue;
                }
                /* ... */
                fclose(fp);
        }
        outfp = fopen(OUTPUT_FILE, "w");
        if (outfp == NULL)
                fatal("Can't open %s for writing: %r", OUTPUT_FILE);
        /* ... etc etc etc */

If one of the input files couldn't be opened and you couldn't write
to /etc/output, you would see the following errors:

myprog: Failed to open /some/file: No such file or directory.
myprog: Can't open /etc/output for writing: Permission denied.

And the program would exit after the printing the second message.
Exit status would be whatever EXIT_FAILURE is defined to (usually 1).

I set up the RT `request tracker' system for use in maintaining grexsoft.

I rewrote the twit filter and installed it today.  The original had
a lot of garbage like, `#define then {' in it, and was overly complex
for no good reason.  It also had a lot of pre-ANSI C'isms in it that I
removed, like forward declarations of functions inside of other
functions and stuff like that.  Ick.

I also have an awk version of the filter that's less than 15 lines
long.  The C version shrank to something like 60 lines.

More on RT:

RT is usually associated with system administration: It's essentially
helpdesk software.  Something like Bugzilla is usually associated
with software development, so why not Bugzilla?

Well, the grexsoft effort sort of straddles the worlds of system
administration and software development.  So it doesn't make sense
to use tools that are primarily targetted at software engineering
(like Bugzilla).  RT is an issue tracking system that works well
for both system administration and software development (for instance,
the RT developers use RT for their own efforts), and a tool for
issue tracking *is* needed, so that's what I went with.  Besides,
it'd be nice to wrap other grex management tasks into RT, and this
is a start in that direction.

For more information, have a look at Best Practical:

http://www.bestpractical.com/

I've configured CVS to log its commit messages to BBS.  See item
9 in garage.

I'd like to start building program-level tests for the grexsoft
commands.  Right now, we have unit tests for the libgrex functions,
which is good, but not the only thing we need.  In particular, we
need a suite of regression tests for the rest of our commands.

Using a framework like DejaGNU for this would be really nice, and
quite powerful, but there are problems installing DejaGNU on grex
without X11 support.  Actually, it's not so much X11 support that's
needed as much as it is expect and tk, which depend on the X11
programming libraries.  I've suggested in another item that we
install these libraries during the next grex upgrade; this will
make installing things like doc++ easier.

I installed the X11 programmer suite here on grex, and reinstalled
doc++ and DejaGNU.  DejaGNU is actually quite complex, so might be
overkill.  I'm investigating simple regression level testing
frameworks, but there isn't much out there.  We'll probably have
to build something.

This and documentation are the big priorities right now.  If you'll
notice from the CVS commit messages logged in item 9, many of the
things I'm checking in are fixes to little bugs I'm introducing.
This is due primarily to three reasons:  1) I'm human and I make
mistakes, especially with something I don't have too much time for.
2) Most grexsoft commands are undocumented, and it's not always
easy to figure out what the expected output for them should be.
3) I'm not testing commands after building them, generally, because
there's no cheap, easy way to automate doing so.

Well, being aware of my own limitations, I've figured out that I've
got to do things somewhat differently.  I've got to structure the
development process a little more efficiently....  How to do this?

I propose the following model for modifying grexsoft code:

        1) Pick a command to change.
        2) If it's not already documented, document it.
           Write a man page for it describing what it's
           supposed to do.  If documentation exists and
           is wrong, tweak it.
        3) Write a regression test for it.
        4) Peak through the source code and try and
           document that.
        5) Write unit-level tests for the major components
           inside the source (this may not be applicable).
        6) Refactor.
        7) Iterate from step 2.

In this model, documentation describes function and comes first,
followed by testing.  Testing is defined by function and drives the
process.  The process iterates, and development and refactoring and
refactoring are repeated throughout the entire procedure in a process
of successive refinement.

Now, we just need to come up with a regression testing framework.
Something like this looks promising:

        http://diveintopython.org/regression_testing/

Except that we have a lot more perl hackers on grex than python types,
I think.  It'd be interesting to take something like that and adapt it
to Perl and use it here.

In Backtalk, please create larger buttons for the listing options
(i.e. 'New Items', 'Unforgotten', 'Forgotten', 'All')

On a 800x600 monitor they are ok, but on anything larger than 
1024x768 they are too tiny to click on.

Maybe we need a "large print edition" interface. :)

I do have a certain tendancy to retain code for obsolete Unix's in my
programs.  I believe, for instance, Backtalk has code to handle versions of
Unix without mkdir(), which is, admittedly, insane.  I have no objection to
updating stuff, but it doesn't entirely thrill me either.  I still do a lot
of things in old-fashioned ways.  After all, I've been programming Unix
systems for at least 20 years now.  One tends to keep doing what works.

If the buttons are too tiny to click on, is the text too tiny to read?  Or
do you have your browser scaling up the text?  Maybe you just have young eyes.
Me, I wear progressive bi-focals and keep my chin up at the computer. 
Luckily, I mostly type near the bottom of the screen.  Anyway, an option for
larger buttons in Pistachio would be doable, but not exacly simple.  I've put
it on my Backtalk TODO list - which contains hundreds of items.

Hm, I use a 1440x900 display, and the pistachio buttons are plenty large
enough for me.

There are Unixes without mkdir()???

No.  According to the 1st Edition Unix manual (circa Nov., 1971), Unix
had a mkdir system call.  Perhaps Jan meant rename, which was rather
later, or opendir/readdir/closedir, which were BSD inventions.

rename? opendir?  Those are practically brand new.

My recollection is that seventh edition Unix didn't have a mkdir() call.
You used mknod() instead.

At http://www.freebsd.org/cgi/man.cgi you'll find a nice collection of
nice man page sets.  If you select "Seventh edition Unix" and search for
"mkdir" you'll only find the command line version.  If you search for
mknod, you'll find that the description of mknod(2) mentions the
creation of directories.  Note that mknod() only works for root users. 
The mkdir command was an suid-root program back in those days.

Interesting that the first edition had mkdir().  Maybe the renamed it to
mknod() when they added special devices and then brought back mkdir()
when the want a non-suid-root ability to create directories.

janc ! can you answer our fronttalk questions, please ?

Perhaps you're right.  The filesystem in 1st Edition was quite different.

Is this thread closed? Or still work is going on?

No one ever joined me and I got bored.  So, no work is going on, though I
wouldn't necessarily say that the thread is closed.

I am ready to C/Shell programming. I am ready to learn Perl if necessary. I
can do little bit of python programming
May I know how can I help you?

Well, you can check out the current version and peek through it.  If you see
something you'd like to work on, go ahead and edit it and email me the
changes.  If they're good, I'll commit them.  Otherwise, I'll email you back
with suggestions.

I e-mailed you sometime back. But it got bounced because it is getting
forwarded to some internet address. I dont have permissions to send e-mails
to internet addresses. I was going through the grex faq (!faq), still it has
got this SunOS listed as the OS used by grex. The web based faq seems to be
fine. I would like to know how to start reading the code. I mean where to find
the code? Do I have to use some version control system to checkout the code.
It will be better if you can send me some suggestions to do stuffs.

Yes, you need to check out the code first.  I was using CVS for version
control, because it came with OpenBSD, worked tolerably well, and subversion
(which would have been my first choice) wasn't in the ports collection at that
time.  For similar reasons, I used doc++ instead of doxygen.

Regardless, here is what you do:

1) create a directory where you'd like your working copy of the source code
   to live.  I usually call mine $HOME/work (that is, a subdirectory "work"
   under my home directory.  Change directories to that directory.
2) Check out a copy of the source code:
   cvs -d /a/n/c/ncvs co grexsoft
   This will create a directory called "grexsoft" under the current working
   directory.  You may get some CVS warnings related to being unable to update
   history files and the like; you can mostly ignore these.
3) Poke around the current grexsoft sources and see what's there; some of it
   is quite old, some much newer.  In either case, the best way to start is
   by reading what's there.  Given time, it becomes easier to spot the good
   code inside the bad, but at first this can be a bit difficult.

   A specific suggestion: the version of the "watch.c" program that's under
   grexsoft/cmd is much improved over the original; how could you improve it
   still further?  Look at the libgrex code; can you think of any other
   routines that would be useful to add?

That should get you started.

Also, read this item and the grexsoft commit item in this conference.  Get
familiar with the tools: make, indent, cvs, etc.

cvs checkout: warning: cannot write to history file
/a/n/c/ncvs/CVSROOT/history: Permission denied
cvs checkout: Updating grexsoft
cvs checkout: failed to create lock directory for `/a/n/c/ncvs/grexsoft'
(/a/n/c/ncvs/grexsoft/#cvs.lock): Permission denied
cvs checkout: failed to obtain dir lock in repository `/a/n/c/ncvs/grexsoft'
cvs [checkout aborted]: read lock failed - giving up

I got that error. The grexsoft directory got created but no source files.

Oh. Hmm.  Yeah, that's a problem, but one I don't have access to fix anymore,
really.  You might be out of luck there.

Response not possible - You must register and login before posting.

- Backtalk version 1.3.30 - Copyright 1996-2006, Jan Wolter and Steve Weiss