For many years now it has been good practice to write unit tests for your source-code. And also to use test coverage reporting to see how much of your code is covered by tests. Although line + branch coverage reporting is quite useful, it doesn't tell you how good your unit tests actually are. Hence it's even possibly to achieve 100% coverage without even a single assert in your tests. Being interested in better ways of testing I attended the "Mutation testing" workshop during this years Joy of Coding conference. Mutation testing is a radical different approach of executing and analyzing the result and coverage of your unit tests. Instead of measuring how much of your code is "accessed from" your unit tests it determines how much of your code is actually "tested by" your unit tests.

The basic idea behind mutation testing is to make a small change (a mutation) to the (byte) code and then execute your tests to see if it is detected by the unit tests. Possible mutations are altering a ">" into ">=", replacing "++" with "--" and removing "void" method invocations. Each mutation therefor creates an altered version of your code called a "mutant". Prior to the actual mutation testing our unit tests first need to be executed against the original code to see if no tests are failing. Then the unit tests will be run for each "mutant" (making it possibly very time consuming) the see if:

Continue reading →