response 4 of 4:

Sep 10 15:12 UTC 2010

Let's talk a little bit about "test infection" and "test-driven 
development."

First, let me state plainly: I think that testing is good.  Tests can 
give you some assurance that your assumptions about how something 
works under some specific set of conditions are valid (or not).  Given 
that many, many bugs are simple "gotchas" that can be easily detected 
by testing, I think that having tests for code is a good way to reduce 
defect counts.  And working with a unit testing framework, even a 
minimalist one like the one I posted in resp:3, is a lot nicer than 
writing ad hoc test drivers.  In general, I think the formalization of 
test writing and emphasis on testing as an intrinsic part of the 
development process is a good thing that's doing a lot to eliminate 
the typical sort of bugs in edge cases and the like, and generally 
increase software quality.

However, tests cannot prove the absence of bugs.  Nor does a good body 
of passing tests guarantee that a program is "correct."  Tests are 
useful to give you some assurance that things are working the way that 
you expect, and libraries of tests can give you some confidence that a 
change you make in one part of the code doesn't break your assumptions 
in other parts of the code, but they are no excuse for not reasoning 
about the code and understanding it.

And that's something that I've got to take exception with; too often 
these days, the "test infected" crowd, as exemplified by practioners 
of "test-driven development" seems to think that working tests 
basically mean correct code.  Consider the short essay, written by 
Robert C. Martin, here:

http://www.objectmentor.com/resources/articles/craftsman5.pdf

In this story, the protagonist "arrives" at an algorithm, guided by 
the tests, but he doesn't really understand the algorithm, at least 
not at first, or the obvious ways to make it better.  The knowledge 
that the tests pass is "good enough" to declare the code correct.  And 
the tests don't really capture the stated assumptions encoded in the 
solution (there's a buffer overflow just waiting to happen in there if 
a currently-valid assumption ever changes).  And there's an obvious 
way to increase the efficiency of the solution.

My feeling is this: a solution should not just "sneak up" on a 
programmer guided by tests.  The solution should be *understood*.  If 
the programmer gets that understand by writing some tests, then that's 
one thing and that's fine, but if the programmer relies solely on the 
tests to decide "correctness" without understanding, then one's 
sitting on a powder keg.

Put another way, tests are wonderful for detecting simple flaws based 
on bad assumptions.  But they are no excuse for not understanding or 
being able to reason about one's programs.