Demo app that shows how can we effectively use mutation testing.
This is a very basic example of test that does not do anything.
Because we have missed that our assert is commented out by accident!
⠙ 2/2 🎉 0 ⏰ 0 🤔 0 🙁 2
To solve this, just uncomment the assert in test_missing_assert.py
This example is used to illustrate the simplest use-case possible. We only have one function with a single statement. This statement will be mutated, our mutation test will catch that.
⠼ 1/1 🎉 0 ⏰ 0 🤔 0 🙁 1
The solution to this case is available here: test_simple.py
Using the same constants for testing and production does not actually test anything.
⠇ 6/6 🎉 2 ⏰ 0 🤔 1 🙁 3
Since we will miss cases like this one:
--- heisenbug/constants.py
+++ heisenbug/constants.py
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
-BAD_FUNCTION_NAMES = ['dir', 'vars', 'locals', 'globals']
+BAD_FUNCTION_NAMES = ['dir', 'vars', 'locals', 'XXglobalsXX']The tests will move in sync with the production code, following the bugs introduced there!
Solution: duplicate your constants in tests, see test_constants.py
This example represents a regular flask application.
It is configured to swallow exceptions and log them somewhere.
It seems to be fully tested, but has one big mistake inside.
⠋ 10/10 🎉 2 ⏰ 0 🤔 0 🙁 8
This is the case we are looking for:
--- heisenbug/flask_app.py
+++ heisenbug/flask_app.py
@@ -18,7 +18,7 @@
@app.route('/{int:index}')
def hello(index):
"""View that will fail in production."""
- return 'Hello, world! {0} faith in you.'.format(1 * index)
+ return 'Hello, world! {0} faith in you.'.format(1 / index)Since we swallow exceptions and test only partial of our output, we will never be able to find the problem of when input is zero and causes a divide by zero.
Main take-away: high-level integrations tests are not good enough on their own.
Solution: do not use 200 code where something goes wrong,
make sure that you write enough unit-tests for your view logic.
This is a simple bubble_sort algorithm.
I have taken an example algorithm as is from TheAlgorithms/Python,
which is a very popular python library.
This is the result (please, take a not that it has several false positives):
⠼ 20/20 🎉 15 ⏰ 0 🤔 0 🙁 5
You can see that we even use property-based tests here.
But it still does not save us from the false-positives of mutation testing.
Main take-away: it is really hard to use mutation testing to test algorithms built with performance and optimisations in mind.
This is a real-world example about
my open-source project called docker-image-size-limit.
It is a perfectly working app with just 70 lines for simple python code.
Fully tested with E2E and unit tests, typed, 100% covered.
And it has ~10 surviving mutants.
⠇ 29/29 🎉 19 ⏰ 0 🤔 2 🙁 8
This article shows you exactly what happened and how to solve this case.
Here I have collected different resource that you might also be interested in.
- https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78
- https://nedbatchelder.com/blog/201903/mutmut.html
Content is licensed under CC BY-NC
