Saturday, October 9, 2010

Levels of Testing


A lot of people ask me if Functional testing replaces Unit testing and Test Driven Development (TDD).  My answer is no and here is my reasoning:

Unit tests verify that discreet units of logic are sound.
Integration tests ensure that the integrated parties adhere to the same contract.
Functional tests verify that the app as a whole performs as expected, from the perspective of the end user.

Using a water bottle as an example:
Unit test:
  The bottle should not leak when filled with water.
  The bottle should hold 12oz of liquid.
  Bottle & Cap material should be non-toxic.
  The cap should have no sharp edges.

Integration test:
  Both cap and bottle adhere to a 1" diameter, clockwise thread.
  When fastened, the cap and bottle should form a tight seal.

Functional test:
  Given that the bottle is full and the cap is fastened, a drop from 6' should not cause a spill.
  The capped-bottle shouldn't leak when heated or cooled within a normal range of temperatures.

Each works at a different level of abstraction.  While there is some overlap in test coverage, none is replaceable by the others.  Functional tests help to ensure that the whole app hangs together ok, but they don't give you the same visibility as a good, granular unit test.  

For example, the heating-cooling functional test would fail if the bottle leaked.  However, it would take some debugging to get to figure out whether the cap, bottle, or the combination of the two was at the root of the problem.  Good unit tests will alert us to any bottle-specific issues way earlier, eliminating the need to debug the fully deployed system.  

That said, I always try to lock in the desired behavior at the unit level first and lean on functional testing for things such as AJAX that don't unit test very well.