Lesser Known Test Processes

Noel Rappin, Tuesday, February 19, 2008 @ 8:20 pm

While Test-Driven Development (TDD) and Behavior-Driven Development (BDD) get all the headlines, thousands if not millions of developers have worked out their own ad-hoc variations on these processes. This is a list of alternate testing processes, some of which are useful, some of which less so. It's a different way of thinking about the benefits of doing testing -- at the very least, it's a way to think about the costs of backsliding on TDD.

Data Driven Testing (DDT)

Also Known As: Begging The Question, Blank Box Testing

Process Description: Write your tests first as in normal TDD, however instead of actually asserting an expected value, assert against a blank value (as in assert_equal(nil, user.batting_average)). The test, of course, fails. You then fill in the expected value based on the result of the failing test. Bingo! The test passes.

Pitfalls: If you aren't careful, this turns in to Tautology Based Testing. Specifically, you need to verify that the returned value really is what you expect. If you are doing this too often, it may mean you don't really understand what your code is doing.

Actual Uses: There are a couple of benefits to this testing process. First off, it guarantees a failed test the first time around, which is proof that the basic test setup is correct, and that you are testing the item that you think. This is also useful in cases where the correct answer is easier to verify than to generate by hand (a long string, for example). It's preferable to use the literal result of the actual program then a separate non-literal calculation of the answer, even if you paste in the literal result.

Test Next Development (TND)

Also Known As: Almost TDD, I Really Mean to do TDD Next Time.

Process Description: Very similar to proper TDD, only in
reverse order: write a small bit of functionality, write a small test.
This really only works if you keep the tight feedback loop.

Pitfalls: The temptation to lengthen the feedback loop is
strong, and eventually you're doing Test Last Development. Also, it's
easy to forget to go back and test every little bit. Writing tests
after the code always takes a little bit longer than writing them
first, even if you've just written a few lines of code first.

Actual Uses: This is a useful mode of development when you
aren't exactly sure of what the details of the code are going to be. In
practice, I find myself falling into this when doing Rails controller
or view development -- I know what the code is going to do in general,
but I'm not quite sure exactly how I'm going to extract data into
variables, or exactly what HTML I want to test against an assert_select.
Usually, a quick run through the real code gets me to the point where I
can write a test, but I'll sometimes continue to do the refinement
backwards -- code first then test. As long as I'm reasonably
disciplined about getting the testing in, I feel like I get nearly all
the benefit of test-first.

Test Last Development (TLD)

Also Known As: Oops, I forgot to test. You mean testing is part of the process?

Process Description: Now, we're moving from relatively useful
variants of TDD, to more or less useless variants. In TLD, the
developer writes a large chunk of code, then goes back and writes tests
for all of it in one fell swoop.

Pitfalls: The main one is that it gets more and more
expensive to go back and write tests as you write more and more code
without testing. No matter how careful you are, it's easy to put in
test-defeating side effects or fail to make the code modular enough to
enable quick testing. Because of that, the tests you eventually write
in TLD tend to be of lower quality, more coarse-grained and often
closer to integration tests than unit tests. When developers complain
that tests take too long to write, they are usually talking about this
kind of testing or Big Setup Testing.

Actual Uses: Sometimes you have to do this because there are
no other options -- you've been handed a chunk of code that has no
tests, and you want to convert it to tested code. If you're doing
test-next, there's often a corner piece that is missed and has to be
done test-last. It's better than having no tests at all.

Coverage Driven Development (CDD)

Also Known As: Drive-By Coverage. Hey, look, we have tests!

Process Description: Tests without assertions -- the
automated test suite just calls the various functions but doesn't do an
actual testing. This can happen in a Rails project if the model testing
is incomplete but the model code is called via a controller action. The
coverage test will show the coverage from the controller action, but
the model code isn't actually being verified (which is why we check for
model coverage separately in our projects).

Pitfalls: Well, you're not actually testing anything, which
is a downside. Even worse, the test will report success even if the
underlying code is broken, which can lead to a false sense of security.

Actual Uses: I'm not actually advocating this, you
understand, but at least knowing that the code runs without crashing
does have some minimal value. From time to time, functionality will be
embedded weirdly into external modules such that more exact testing is
prohibitively difficult even with mocks (I've had this happen in Rails
where I was mocking ActiveRecord to not need a database at all).

Tautology Based Development (TBD)

Also Known As: Look, Our Tests Pass. Two == Two.

Process Description: This is not so much a process as it is a
common problem that happens writing tests. Due to some carelessness,
you wind up testing that an object is equal to itself. This can happen
if you are testing a derived value, with something like this,

  x = user.phone_number  #other code  assert_equal(x, user.phone_number)

It's also an occupational hazard of using mock objects but
continuing to test state rather than behavior -- the danger is that you
are just testing that the output is the value you manually set the mock
to return. (True mock testing, where you validate that the correct
method call is made, is a little bit less susceptible to this problem).

Pitfalls: Like Coverage-Driven Development, tautology testing
has the unfortunate side effect of being overly optimistic in reporting
the state of the code.

Workarounds: You can limit the likelihood of having
tautologies by trying to ensure that the expected value in your
assertions is a literal object (Jay Fields just wrote something similar).
At the very least, this ensures that you'll get a failure if your data
changes -- this is a good thing, because it shows that your tests are
actually protecting you. I'd rather have my tests be pessimistic than
optimistic.

Actual Uses: Practical effect is the same as Coverage-Driven
Development, but you might think you're doing something more elaborate.
Never really recommended.

Big Test Setup Development (BTSD)

Also Known As: I Really Hate Writing Tests And This Is Why.

Process Description: In order to get the tests going, you
need to write an elaborate amount of setup code just to create an
environment where the tests will run. This often happens in a system
which was not initially created using test-driven techniques,
especially if the original developers are not familiar with mock
objects.

Pitfalls: If tests are too painful to write, developers will stop writing them.

Workarounds: Mock objects can help create a testable
environment for specific tests quickly. It also helps to treat tests as
code, and factor out common setup tasks to helper classes so they can
be easily called from multiple tests.

Actual Uses: This is often an initial step in converting
untested legacy code to having complete tests. Ideally, it's an
intermediate step, and eventually the test setup becomes more
manageable either by mocks, refactoring, or simplifying the application
architecture.

What other variant forms of Test-Driven Development are out there,
both the somewhat useful kind, and the failed-process kind? I'd love to
hear your TDD-related habits.

Watch for  Professional Ruby on Rails.  Coming soon...

agile ajax

Comments

• Leave a comment
• Comment feed for this post
• TrackBack URL

None so far.

Leave a comment