Tuesday, July 24, 2007

Making testing enjoyable with parsers

A lot of people can find writing autotests to be tedious. But once you have a parser for your code there are a number of very interesting projects that you can write to help test your code and improve your auto tests.

The faster you can find errors the less time you will spend later on fixing bugs. The more I can automate things, the more tests I will get and the better the chance that I will find bugs sooner while I am still familiar with the code.

Generate autotests


The first things you should do with the parser is write a tool that will generate a stub auto test for you. Once you have an autotest adding new ones are not that hard, but creating that first one can be hard. Don't stop at just a stub, basic tests that just call every function (to make sure they don't crash) and more are very useful to generate. After that when writing the stub that will test function X the parser should walk inside of X and automatically add comment to the test for each control flow statement. Then all that has to be done is go through and implement each comment. This should remove a lot of the guesswork about what to test.

Mutation testing


Once you have autotests you can run your parser over your code only this time have it modify the code. There are plenty of simple things such as changing if(foo) to if (!foo) that can be done with very little work. After the modification run the auto tests and see if the test still passes, in which case the auto test coverage isn't very good. Record those changes that don't cause any failures in the tests.

Sameness


When adding a feature or fixing a bug one thing you want to do it to make sure that your changes didn't accidently change something that it wasn't meant to. Generate a known set of known things about your object. To start this database use your parser to call every function and record what happens. It doesn't need know if what it does is right or wrong just what it does. After making a change you can compare any differences to make sure you didn't break something you didn't mean to. The more data you can add to the database the more secure you can be in knowing that your changes didn't cause unintended regressions.

Test failure finder


I have previously written about this method here: Test failure finder.

Fuzz


A simpler version of the previous, test failure finder, using the parser to determine what can be called and just call the inputs with random data (from a seed) until it crashes. This is a way to test your code without having to write any code at all and can be run starting from day one.

I don't like to fix bugs, I like to code.

No comments: