On Testing and Refactoring

The last few days there was quite a bit of commotion in the blogosphere around unit testing, TDD and testing in general. (InfoQ has some more background.) I don't want to get into that whole debate -- I'm a big fan of testing but I don't practice TDD religiously -- but I do want to touch on an interesting subject that came up: the way testing and refactoring relate.

There is a bit of a dual relationship between testing and refactoring. On one hand, good test coverage allows you to refactor with confidence: the tests tell you whether or not you broke things. On the other hand, if the refactoring you want to execute has a large impact on the test suite, you're held back in two ways: the extra work required to rework the tests, and the possibility of introducing bugs in the tests while reworking them, which reduces your refactoring confidence.

On the project I'm currently working on we've tackled this in an interesting way (nothing novel but worth pointing out nonetheless): next to the normal unit tests, we also have a large suite of what we call scenario tests. These scenario tests operate at a somewhat higher level and run against stable abstractions. Unlike unit tests they don't test isolated units of the application. Instead they test the system as an integrated whole and run real-life scenarios through it. However, the scenario tests are not really integration tests: all environmental dependencies are mocked out making it possible to run these scenario tests as part of the normal test suite with every build.

To make this a bit more tangible, lets consider a fictional event processing component:

public class CalculatingEventProcessor implements EventProcessor {

  private SomeCalculator calculator;
  private SomeRepository repository;

  ...

  @Override
  public void process(Event event) {
    if (weNeedToCalculateSomethingFor(event)) {
      SomeObject input = extractSomeObjectFrom(event);
      SomeOtherObject output = calculator.calculate(input);
      repository.store(output);
    }
  }

  ...
}

Instead of simply having unit tests for the SomeCalculator and SomeRepository classes, we build scenario tests at the level of the EventProcessor interface to verify that the correct things are calculated in the expected situations.

public interface EventProcessor {
  void process(Event event);
}

Such a scenario test would look something like this:

public class MyCalculationScenarioTest extends AbstractEventProcessorScenarioTest {

  @Override
  public EventProcessor setupEventProcessor() {
    return ...; // instantiate, or pull from a Spring app context, or ...
  }

  @Override
  public void runScenario() {
    // setup data as required by the scenario
    ...
    
    // create a relevant event
    Event event = ...;

    publishEvent(event);

    // the test infrastructure will run the event through the event processor

    // assert the necessary calculations have been done
    ...
  }
}

Note that the above test sets up all required data, typically in a throw-away database like HSQLDB. This is necessary to make sure that the scenario test can be run as a normal unit test: independent of any other tests or its environment.

It takes a bit of effort to setup the required scenario test infrastructure for your project, but the pay-off is big. Once you've got a sizeable set of scenario tests, you can refactor large parts of the internal design of your application without any impact on the scenario tests, which you can run at any time to see if things are moving in the right direction. The key here is the fact that the scenario tests run against a stable abstraction such as the EventProcessor interface in the example, which is unlikely to change.

UPDATE: It's worth pointing out that the scenario tests described above follow the basic form of a scenario in BDD (Behaviour-Driven Development):

Given some initial context (the givens),
When an event occurs,
then ensure some outcomes.

In BDD scenarios serve as an executable specification: a set of executable acceptance criteria that become proper end-to-end functional tests. The scenario tests described here serve the exact same purpose.