Monday, September 19, 2005

failure finder (or auto tests ++)

After I started writing autotests I began to noticed that I was still having bugs found in those classes that I had autotested. Every time a new bug was found I spent a fair amount of time trying to determining why my autotests hadn't found it. In the end I concluded that autotests can only get you so far because I wont think to call x(), y(), y(), y(), x() in my autotest even though I did happened to test x(), y(), x(). I can only comprehend a limited set of all permutations of interactions.

To automatically find some of these pairs would be set up. The first half of each pair is actions and the second set are verifiers.
For every action there is a verifier of the "state of the world" as known by that action. For example if you have the following functions:

size();
setSize(x) { last = first + x; mySize = x;}
int mySize;

The action is setSize(x) and the verifier confirms that size() will return last() - first(). With this pair you are then able to randomly verify and set things. Somewhat similar to how you generate genetic algorithm. Once you find a failure you simply perform the same genetic algorithm search, but this time the goal is the smallest set of operations that can still reproduce the failure.
Then when you add function

clear() { last = 0; first = 0; }

the test would find a bug very quickly.