A year ago I had never heard of mutation testing. Then when looking at the list of presentations for TestBash Utrecht 2018, there was one on mutation testing which caught my eye.
I had heard of security testing, automation testing, penetration testing, regression testing, and many other types of testing, but never mutation testing before, and the name intrigued me.
I searched online to find out more about it, and thought it was interesting, but not something that I could apply to my day-to-day activities as it was more geared towards developers.
Then this month my manager tells me they want to look at the roles of testers and developers within my teams and blend them, which will involve the devs learning more about testing. I then remembered mutation testing, and thought it would be great.
The idea for this post came from talking at my local Meetup during the break about how I showed my developers a video on mutation testing to teach them what it is, and see if the idea appeals to them as a way of getting them to do more testing. The other testers had never heard of it, and so I explained what it is. But that made me wonder, if I didn’t know what it was, and they didn’t, how many other testers don’t know what it is either?
Now, you may be thinking “That is a great story, but you still haven’t told us what it is!”
Mutation testing is a way of doing more with a developers unit tests, by confirming what they are testing, and if more tests could be done.
The way it does this is once the developer has written their code and unit tests, they put their code into a mutation framework and has it mutated.
It does this by looking for variables within the code and changing them. For example, if your code only allowed someone to buy alcohol if their age is >= 18 years old, it would see the >= and change it into a > to check the threshold, < to see what happens if your’re below the threshold instead of above, removing lines of code to see if you are looking for it, and much more.
Then, once it has mutated the code, the unit tests are re-ran against the code. If a unit test fails, it successfully found the mutant and the unit test works as expected. But if the unit test still passes, it failed to find the mutant and extra work may be needed with the code, the unit test, or both.
Because it involves manipulating the raw code and running the unit tests, it is why I see this geared more towards developers than testers, but depending on where you work it could be useful for both roles.
The presentation at TestBash Utrecht was recorded, and was made available to free members of the site. As such, if you wanted to watch it, or present it to your colleagues, it would be a great way to share what mutation testing is and if it is something that could be done at your workplace.
You can find the video at https://www.ministryoftesting.com/dojo/lessons/kill-the-mutants-nico-jansen-simon-de-lang
In the video they talk about the difference between code coverage and mutations, and how they achieved 100% code coverage with 16 unit tests, yet when they put the code into the mutation framework it created over 100 mutations, showing that code coverage is not the ultimate arbiter on if the unit tests are sufficient.
Hopefully this post has given you an insight into what mutation testing is, how it could help identify defects in the code or unit tests before it leaves development, and aren’t we all wanting to make software that works?