Bart and James do... Mutation Testing

Outline

First half: We'll spend the first half of the workshop getting into Mutation Testing. You'll get hands-on with a tool, mutmut, running against several bits of Python code.

Break: You're welcome to keep working over the break, to take a moment and return as the sessions start – or to leave. You're also welcome to drop in for the second half; we'll help you set up, and you should find plenty of information on this page.

Second half: In the second half, we'll look at how we're learning together, and share ideas and strategies as we dig a little deeper into mutation testing.

End: You'll have tried mutation testing, you'll have tried learning together. Maybe you'll carry on with one or both!

Collaboration tools

We have a shared Miro board for this workshop. You can edit it now, and we'll lock it after the workshop so that you can refer to it.

We'll work in repl.it, which allows us to give you your own environment, pre-configured with working code, working tests, Python, Pytest (with code coverage), and Mutmut. Here's :more on getting set up with repl.it.

:x Repl stuff

You can set up a repl.it account with email or with an existing login from GitHub, Google, Apple or Facebook. You're not siging up for a free trial – you can do everything from the perpetual free plan.

Repl.it runs in your browser. You can download an app if you prefer. Be aware that everyone downloading the app at the same time may trash the wifi. Be early, be late or be patient.

Subjects to test

Each subject is in Python, and has source and test code. Each is contained in a repl.it environment, set up with the mutmut mutation tool and an html report of test coverage metrics. Each has a readme.md markdown document with information and notes. The table below lists the subjects and links to template environments.

To get hands-on, at least one person in a group will need :a (free) replit account. You'll need to :fork whichever of these you're playing with to your own account, so that you can run mutmmut on your own replit server.

:x forkit

Environments

Command-line Stuff to run Mutation and Tests

To work with mutation testing, you'll need to run shell commands. Now you've got your own environment, you can run those commands without changing anyone else's situation.

When you've got your own copy of this repl in your account, look for a tab marked shell. A shell tab appears as a termina emulator and is typically black with white monospaced text. You'll type your command into a prompt line, and hit return to execute. Look for cyan prompt lines starting with ~/something-or-other$. This is a linux environment; try typing ls to list files, or pwd to show your current position in the file system.

If you can't see a shell tab in your own environment, you can make one with the '+' option to the right of the labels in the top bar of the pane to the right of the file list.

Mutation Testing

To run mutation testing – and to run it again...

python3 -m mutmut run

This mutates code in src, and runs the tests in tests for each mutant. Open a file in src while mutating to see the mutation happening!

To run again, run the same command: python3 -m mutmut run

However, this uses a cache, which can be deceptive. If you've changed the tests or the source, and especially if you see weird results when re-running mutmut, clear the cache by deleting the file .mutmut-cache.

Seeing which mutants testing missed

It's easier to browse all survivors using the HTML reports.

Produce an HTML report with python3 -m mutmut html , then go look in the html/src directory to see all the survivors. The files look like html, but the mutated code is easy to pick out.

Example: Surviving Mutant 205  used alertOnCreation(None) instead of  alertOnCreation(player).

</pre><h3>Mutant 205</h3><pre>--- src/round.py
+++ src/round.py
@@ -6,7 +6,7 @@
 		self.score = 0
 		self.currentDice = availableDice
 		self.endedGreedy = False
-		alertOnCreation(player)
+		alertOnCreation(None)
 	def processRoll(self, thisRoll, mentionAvailableDice):
 		self.currentDice = self.currentDice - thisRoll.diceUsed
 		self.score = self.score + thisRoll.score
</pre>

To list survivors one by one

Mutmut's instructions direct you to get the results and go through one by one.
python3 -m mutmut result-ids survived

You then need to pick a specific survivor:

python3 -m mutmut show 2 shows survivor 2.

You can switch the code to a numbered mutant with apply – best not to do this unless you're happy changing the source to a broken version.

Testing

Every mutation generated by mutmut is tested. Testing involves running pytest on /src, using tests from /tests. You can just run pytest to do what it does.

It's worthwhile examining the coverage of those tests; mutations to code that isn't tested will typically survive.

To run tests with coverage metrics, showing branch coverage on stuff in src:

python3 -m pytest tests --cov src --cov-branch --cov-report html:cov_html

To browse coverage

Hit the :"run" button.
Look at 'webview' panel, or (better) open the coverage metrics in a new tab using the :arrow-pointing-diagonally-out-of-a-box button.

:x runbutton

:x arrowbutton

To run pytest on src with the tests in tests:

pytest or
python3 -m pytest or
python3 -m pytest tests

To run tests with coverage metrics, showing statement-only coverage on stuff in src:

python3 -m pytest tests --cov src --cov-report html:cov_html

Activities and Exercises

Here are repl.it basics

Mutating with mutmut

We're proposing three similar exercises with different code bases. The complexity of the system is indicated as S, M or L.

Fork the repl.it environment to make your own to play with.

You don't have to stay with the exercise you pick first. Indeed, as you move on in the workshop, you might find it interesting to step up, or fun to step down. And you're welcome to substitute for your own code...

Learning Strategies

Background: About Mutation Testing

Mutation testing - Wikipedia

Wikimedia Foundation, Inc.Contributors to Wikimedia projects

Mutation Testing with Python

Test the tests — automatically, by applying common mistakes

Analytics VidhyaMartin Thoma

About MutMut

Mutmut is a python library. It's well-established and open-source. Lead contributor is boxed / Anders Hovmöller.

Here are documents, source, more docs on whitelisting.

Mutmut mutation types

In Mutmut's `__init__.py`, you'll find:

mutations_by_type = { 
'operator': dict(value=operator_mutation), 
'keyword': dict(value=keyword_mutation), 
'number': dict(value=number_mutation), 
'name': dict(value=name_mutation), 
'string': dict(value=string_mutation), 
'argument': dict(children=argument_mutation), 
'or_test': dict(children=and_or_test_mutation), 
'and_test': dict(children=and_or_test_mutation), 
'lambdef': dict(children=lambda_mutation), 
'expr_stmt': dict(children=expression_mutation), 
'decorator': dict(children=decorator_mutation), 
'annassign': dict(children=expression_mutation),
}

These do the following

'operator': switches '+' for '-', '/' for '*', '++' for "!=' and more ,
'keyword': switches "'ins' for 'is not', 'break' for 'continue', 'True' for 'False' and more,
'number': change number by +1, -1, with specials for floats and different bases,
'name': switches 'True' for 'False' and more,
'string': adds XX to start and end,
'argument': changes arguments names with XX,
'or_test': switches 'and' for 'or' (?),
'and_test': switches 'and' for 'or' (?),
'lambdef': ? changes order,
'expr_stmt': switches ?something? for 'None',
'decorator': ?switches children for newline,
'annassign': as experssion?,

There's nothing valuable under here – just no need to show you half-thought-through bits that didn't make this page.

ONLY SPRUE, FROM HERE ON

Mutating by hand

• ‌

‌

Your Stories – 10 min collective

How do you work together

• ‌

‌

Meet your peers – 10 min small groups

Needs a purpose – perhaps combo with your stories

• ‌

‌

Mutation basics - 20 min small groups

Get online. Open a repl.

• Play with the subject.
• Open the tests, run the tests, inspect the tests
• Run mutation
• Assess mutations – show survivors
• Improve tests
• Mutate again
• ? Does teh mutatio testing affect the code, or the tests?
• ? Comments on mutation – costs and uses
• ‌

‌

Mutation 2 – 30 mins focussed groups

Decide on level. Get into groups with same level.

Get online. Open the repl for the level.

• Play with the subject.
• Open the tests, run the tests, inspect the tests
• Run mutation
• Assess mutations – show survivors
• Improve tests
• Mutate again
• ‌

‌

Discussion – costs and uses

• ‌

Our story

Narrative, to follow from "your story"

How we worked together

Overview and insight

What we learned about Mutation Testing

What we learned about learning together

Environments for Testing

All environments are repl.it environments.

To get hands-on., you'll need to sign up for repl.it. You'll need to fork the repls you're working with.

If you don't want to sign up, you can watch or pair with someone – even James or Bart.

All are Python 3 coding environments, and have the tool `MutMut` installed.

You can run MutMut in all with the commandline `mutmut run`. Here's a video as a demo.

Env 1 – basics

Source and tests are for a few lines of code which find the maximum value of a numeric list.

Env 2 – moving on

Source and tests are for ? a dice game., Greed.

Env 3 – knock yourself out

Source and tests are for the Gilded Rose Kata.

Learning strategies (and tactics...)

• Authority-first – follow the book, ask the expert
• Promise-driven – commit to do something you don't know how to do,
• Confusion-driven – try to understand the part you recognise as a part, yet understand the least
• Foundation-driven – work from what you already know, and expand outwaerds
• Literature survey – what are the key words? Go search, building a collectioon of core vocabulary (words and concepts). Do they mean different thigns to different groups? What are the core articles / sites / authors / groups / magazines / books / exercises / metaphors?
• Ask publicly for help –
• Aim to teach / write
• Value-driven; find and deliver something of value to someone

Working in repl

• open one of the workshop repls
• To run the tests, go to the command line and enter the command `python -m pytest tests`
• To check the tests with `mutmut`, go to the command line and `python -m mutmut run`. You may want to delete the mutmut cache first –
• To
• go to the command line:
• delete the mutmut cache:
• browse a file
• change control – github

About mutating Gilded Rose

‌             ‌

To do

• get coverage metrics
• - get readable coverage metrics
• Analyse each mutation in the three target environments

¿¿If you want to work on your own, you'll need to make your own account to clone our environments. If you want to work in a group, one of you can have an account, and invite the others.

¿¿ We'll have a few environments that we can invite you to. We'll need you to give us your email address, and we'll need to invite you in the workshop.

¿¿ Set up several envs - perhaps `one of each for 5 ?different areas

¿¿ join link per repl?

¿¿ JL - check 1) learn envs before dup 2) coverage 3) mumut 4) tests