Under the Waterfall: The Cost of Bugs After Launch

Dietrich Kappe, Monday, October 19, 2009 @ 1:44 pm

photo credit: bobistraveling

We have a few simple principles when it comes to software guarantees: either we have developed the software according to our rigorous testing standards or the third party code we being asked to support meets some basic criterea:

1. It has full functional specifications.
2. It has unit tests that provide better than 85% line and branch coverage.
3. It has automated functional tests that exercise 100% of the user stories.

That may seem pretty extreme, but if we're promising to fix bugs on our dime, we can't have it any other way. Doing without these basic criteria is like guaranteeing you can drive from New York to Los Angeles in 30 hours, keeping under the speed limit without a speedometer and without being told what the speed limit is. Spelling it out a little more, if you can't tell me what your application should do, how do I know if it is "broken?" If I can't run and inspect unit tests to tell if individual classes and methods are doing the right thing, how can I quickly tell if my changes have broken the system?

There's lots of damning studies of the software development industry out there, but the NIST study from 2002 is one of my favorites. In it there is a nice little table that shows that in the 1960's and 1970's fully 80% of software development effort was spent on Coding and Unit Testing, another 10% on Integration and System Testing. Only 10% was spent on the whole requirements and design shooting match. Fast forward to the 1990's, when only 30% of effort was spent on what previously consumed 90%.

The story isn't, of course, that programmers became less professional and more sloppy. Rather, people discovered that if you developed the wrong software it didn't really matter how bug free it was. So more effort was spent up front on requirements. In that sense, Agile software development is a little bit of a retro movement since it reasserts the primacy of unit and integration testing, though with Test Driven Development (TDD) and continuous integration you get there in a very different way than before.

Of course most of the 3rd party software we are asked to take over were developed using a waterfall approach. They usually have functional specifications, though they're typically out of date, and they usually have no unit tests and spotty functional tests. There's another nice table in the NIST study that shows that defects that are from 440 to 880 times as expensive to repair in the operation and maintenance phase of a software applications life-cycle as they were in the original development. I think that's too pessimistic. I think that 80% is a good figure. In other words, if you spent $200k developing your original application, you should expect to spend $800k on maintenance programming and testing over the application's life span.

If you've got good unit tests, functional tests and functional specs, you might carve that down to 50%, in other words, expect to spend another $200k over the lifespan of the application. If you've followed an Agile development approach, you should be able to achieve those criteria. Your choice.

Related posts:

1. 800 on Your Math SAT, Software Development and Bugs
2. Agile Development Improves ROI – But RFP Processes are Stuck in Waterfall.
3. Cucumber Rocks – But it’s not a replacement for unit tests
4. Bugs can’t be estimated
5. A mea culpa, and a launch date, for Really Simple History 0.8

Comments (0)

Software Development

Comments

• Leave a comment
• Comment feed for this post
• TrackBack URL

None so far.

Leave a comment