Abstract
Have you ever tried to learn something new, but felt that you didn’t get very far? You can just start and see where you land – we all know that there are other options. You'll find out some of those, and much more, in this workshop.
Bart and James choose, sometimes, to team up to learn; bouncing ideas around, taking surprising paths, and motivating each other. They’ve found that working together helps them not only to start, but to keep going, and to be changed by what they find. And they do most of that remotely.
In this workshop, they’ll show you how they dug into mutation testing, over six months in 2022. You’ll see their strategies and their struggles, and how they (and their goals) changed over time. With short, focussed exercises, you’ll dig into the same subject yourself, sharing ideas and approaches with other workshop participants. By the end of the workshop, you’ll not only have new insights about how to learn new skills, but you’ll also have assembled your own small working example of mutation testing. Expect a double-length session of exploring,wondering, deducing, laughter and fun.
- Mutation testing and how to apply it
- How to apply learning strategies
- Buddying over the internet, how to make it work
Outline
First half: We'll spend the first half of the workshop getting into Mutation Testing. You'll get hands-on with a tool, mutmut
, running against several bits of Python code.
Break: You're welcome to keep working over the break, to take a moment and return as the sessions start – or to leave. You're also welcome to drop in for the second half; we'll help you set up, and you should find plenty of information on this page.
Second half: In the second half, we'll look at how we're learning together, and share ideas and strategies as we dig a little deeper into mutation testing.
End: You'll have tried mutation testing, you'll have tried learning together. Maybe you'll carry on with one or both!
Collaboration tools
We have a shared Miro board for this workshop. You can edit it now, and we'll lock it after the workshop so that you can refer to it.
We'll work in repl.it, which allows us to give you your own environment, pre-configured with working code, working tests, Python
, Pytest
(with code coverage), and Mutmut
. Here's :more on getting set up with repl.it.
:x Repl stuff
You can set up a repl.it account with email or with an existing login from GitHub, Google, Apple or Facebook. You're not siging up for a free trial – you can do everything from the perpetual free plan.
Repl.it runs in your browser. You can download an app if you prefer. Be aware that everyone downloading the app at the same time may trash the wifi. Be early, be late or be patient.
Subjects to test
Each subject is in Python, and has source and test code. Each is contained in a repl.it environment, set up with the mutmut
mutation tool and an html report of test coverage metrics. Each has a readme.md
markdown document with information and notes. The table below lists the subjects and links to template environments.
To get hands-on, at least one person in a group will need :a (free) replit account. You'll need to :fork whichever of these you're playing with to your own account, so that you can run mutmmut
on your own replit server.
:x forkit
Environments
description | link | |
Minimal | Hardly there | replit.com/@workroomprds/PythonMutMutMinimal |
Broad | Watch it work | replit.com/@workroomprds/PythonMutMutBroad |
S | Picks largest value from list | replit.com/@workroomprds/PythonMutMutExerciseS |
M | Gilded Rose Kata | replit.com/@workroomprds/PythonMutMutGildedRose |
L | 'Greed' dice game | replit.com/@workroomprds/PythonMutMutGreed |
Command-line Stuff to run Mutation and Tests
To work with mutation testing, you'll need to run shell commands. Now you've got your own environment, you can run those commands without changing anyone else's situation.
When you've got your own copy of this repl in your account, look for a tab marked shell. A shell tab appears as a termina emulator and is typically black with white monospaced text. You'll type your command into a prompt line, and hit return to execute. Look for cyan prompt lines starting with ~/something-or-other$
. This is a linux environment; try typing ls
to list files, or pwd
to show your current position in the file system.
If you can't see a shell tab in your own environment, you can make one with the '+' option to the right of the labels in the top bar of the pane to the right of the file list.
Mutation Testing
To run mutation testing – and to run it again...
python3 -m mutmut run
This mutates code in src
, and runs the tests in tests
for each mutant. Open a file in src
while mutating to see the mutation happening!
To run again, run the same command: python3 -m mutmut run
However, this uses a cache, which can be deceptive. If you've changed the tests or the source, and especially if you see weird results when re-running mutmut, clear the cache by deleting the file .mutmut-cache
.
Seeing which mutants testing missed
It's easier to browse all survivors using the HTML reports.
Produce an HTML report with python3 -m mutmut html
, then go look in the html/src
directory to see all the survivors. The files look like html, but the mutated code is easy to pick out.
Example: Surviving Mutant 205
used alertOnCreation(None)
instead of alertOnCreation(player)
.
</pre><h3>Mutant 205</h3><pre>--- src/round.py
+++ src/round.py
@@ -6,7 +6,7 @@
self.score = 0
self.currentDice = availableDice
self.endedGreedy = False
- alertOnCreation(player)
+ alertOnCreation(None)
def processRoll(self, thisRoll, mentionAvailableDice):
self.currentDice = self.currentDice - thisRoll.diceUsed
self.score = self.score + thisRoll.score
</pre>
To list survivors one by one
Mutmut's instructions direct you to get the results and go through one by one.python3 -m mutmut result-ids survived
You then need to pick a specific survivor:
python3 -m mutmut show 2
shows survivor 2.
You can switch the code to a numbered mutant with apply
– best not to do this unless you're happy changing the source to a broken version.
Testing
Every mutation generated by mutmut
is tested. Testing involves running pytest
on /src
, using tests from /tests
. You can just run pytest
to do what it does.
It's worthwhile examining the coverage of those tests; mutations to code that isn't tested will typically survive.
To run tests with coverage metrics, showing branch coverage on stuff in src:
python3 -m pytest tests --cov src --cov-branch --cov-report html:cov_html
To browse coverage
Hit the :"run" button.
Look at 'webview' panel, or (better) open the coverage metrics in a new tab using the :arrow-pointing-diagonally-out-of-a-box button.
:x runbutton
:x arrowbutton
To run pytest on src with the tests in tests:
pytest
orpython3 -m pytest
orpython3 -m pytest tests
To run tests with coverage metrics, showing statement-only coverage on stuff in src:
python3 -m pytest tests --cov src --cov-report html:cov_html
Activities and Exercises
Watch the mutations
Do this yourself, or watch with Bart and James.
We'll work with the 'broad' environment: replit.com/@workroomprds/PythonMutMutBroad
- in the file pane, focus on
src/something.py
to see the code - in the shell pane, type
python3 -m mutmut run
and hit enter - watch the code as
mutmut
mutates it, and eventually reverts it back to unmutated.
Extra – what doesn't get mutated?
Mutating by hand
Do this with others, or by yourself
Work with the minimal environment: replit.com/@workroomprds/PythonMutMutMinimal
Make a change to the code in src
.
You're trying to make a change that the existing tests don't notice!
Extra – can you make a change which changes the behaviour, but which the tests don't notice?
Removing tests
This will work best work better in the M and L environments.
- Run
mutmut
to see what mutants survive. - Comment out an
assert
from one of the test files in/tests/
- Remove the cache and re-run mutation. Do the same mutants survive?
- Can you find a test to remove, which does not change the surviving mutants?
If you can remove a test, and the remaining tests still 'kill' mutants, was that test worthwhile, or excess?
Here are repl.it basics
Mutating with mutmut
We're proposing three similar exercises with different code bases. The complexity of the system is indicated as S, M or L.
Fork the repl.it environment to make your own to play with.
You don't have to stay with the exercise you pick first. Indeed, as you move on in the workshop, you might find it interesting to step up, or fun to step down. And you're welcome to substitute for your own code...
Exercise S – largest value from list
Small and readable codebase, to pick the largest value from a list.
- Make a clone of replit.com/@workroomprds/PythonMutMutExerciseS
- Look at the instructions in the environment's
readme.md
. - Open
src
andtests
to see what's what. - Run
Pytest
and look at coverage - Run
mutmut
and look at survivors. - Pick a mutant. Why did it survive?
Can you see a gap in the tests? Does seeing the gap help you improve the tests?
Extra – add to the tests aiming to catch at least one mutant. Run pytest
to check that it works. Re-run mutmut
to see if your new test catches the mutant.
Extra – do you feel like changing the code? Is that a helpful feeling?
Extra – use # pragma: no mutate
to whitelist a line in the code. See whitelisting in the docs for more.
Exercise M – Gilded Rose Kata and tests
One file of code and several tests, to move imagined goods in steps towards expiry.
Make a clone of replit.com/@workroomprds/PythonMutMutGildedRose
This code is an implementatio (with tests) the Gilded Rose Kata, specifically Emily Bache's version, with tests similar to work by MrRKernelPanic and Matthew Morgan.
- Look at the instructions in the environment's
readme.md
. - Open
src
andtests
to see what's what. - Run
Pytest
and look at coverage - Run
mutmut
and look at survivors - Pick a mutant. Why did it survive?
Can you see a gap in the tests? Does seeing the gap help you improve the tests?
Extra – add to the tests aiming to catch at least one mutant, and re-run mutmut
to see if your new test does.
Extra – do you feel like changing the code? Is that a heplful feeling?
Extra – use # pragma: no mutate
to whitelist a line in the code. See whitelisting in the docs for more.
Extra – change __init__.py
or the commandline option --disable-mutation-types
to stop using a whole class of mutants. See advanced whitelisting in the docs for more.
Exercise L – Greed Dice
Several files of soruce code, and incomplete tests, to allow a command-line multi-player game of 'greed' dice. Here are two versions of the rules – grom GroupGames and from HowDoYouPlayIt.
Make a clone of replit.com/@workroomprds/PythonMutMutGreed
In this example, two of the source files have tests and good code coverage. The rest are excluded from mutation with whitelisting in __init__.py
. Mutation works specificially on greed.py
(scoring) and round.py
(handling a player's round of several dice throws)
- Look at the instructions in the environment's
readme.md
. - Play a round of greed with the program.
- Open
src
andtests
to see what's what. - Run
Pytest
and look at coverage - Run
mutmut
and look at survivors - Pick a mutant. Why did it survive?
Can you see a gap in the tests? Does seeing the gap help you improve the tests?
Extra – add to the tests aiming to catch at least one mutant, and re-run mutmut
to see if your new test does.
Extra – do you feel like changing the code? Is that a heplful feeling?
Extra – use # pragma: no mutate
to whitelist a line in the code. See whitelisting in the docs for more.
Extra – change __init__.py
or the commandline option --disable-mutation-types
to stop using a whole class of mutants. See advanced whitelisting in the docs for more.
Learning Strategies
Conversation – how have you been learning?
5-10 minute facilitated chat
Exercise – pick an approach
15 minutes work, group or solo, + debrief
Consciously pick a learning strategy / tactic that you'd like to try here.
Make that choice clear to yourself (and to those around you). Perhaps say whether you're chosing something familar or whether it's a stretch.
Move on with your exploration of mutation testing.
Debrief: Stories and surprises about learning, 5-10 minutes faciltated chat.
Background: About Mutation Testing
About MutMut
Mutmut
is a python library. It's well-established and open-source. Lead contributor is boxed / Anders Hovmöller.
Here are documents, source, more docs on whitelisting.
Mutmut mutation types
In Mutmut's `__init__.py`, you'll find:
mutations_by_type = {
'operator': dict(value=operator_mutation),
'keyword': dict(value=keyword_mutation),
'number': dict(value=number_mutation),
'name': dict(value=name_mutation),
'string': dict(value=string_mutation),
'argument': dict(children=argument_mutation),
'or_test': dict(children=and_or_test_mutation),
'and_test': dict(children=and_or_test_mutation),
'lambdef': dict(children=lambda_mutation),
'expr_stmt': dict(children=expression_mutation),
'decorator': dict(children=decorator_mutation),
'annassign': dict(children=expression_mutation),
}
These do the following
'operator': switches '+' for '-', '/' for '*', '++' for "!=' and more ,
'keyword': switches "'ins' for 'is not', 'break' for 'continue', 'True' for 'False' and more,
'number': change number by +1, -1, with specials for floats and different bases,
'name': switches 'True' for 'False' and more,
'string': adds XX to start and end,
'argument': changes arguments names with XX,
'or_test': switches 'and' for 'or' (?),
'and_test': switches 'and' for 'or' (?),
'lambdef': ? changes order,
'expr_stmt': switches ?something? for 'None',
'decorator': ?switches children for newline,
'annassign': as experssion?,
Background: Learning strategies (and tactics...)
As Bart and James worked together, they identified different ways that they approached learning.
- Authority-first – follow the book, ask the expert
- Promise-driven – commit to do something you don't know how to do,
- Confusion-driven – try to understand the part you recognise as a part, yet understand the least
- Foundation-driven – work from what you already know, and expand outwaerds
- Literature survey – what are the key words? Go search, building a collectioon of core vocabulary (words and concepts). Do they mean different thigns to different groups? What are the core articles / sites / authors / groups / magazines / books / exercises / metaphors?
- Ask publicly for help – get comfortable with your own ignorance and curiosity, attract people who want to help, reward their commitment with your progress.
- Aim to teach / write – teaching and writing both require your mind to enage with the subject in a reflective, more-disciplined way
- Value-driven – find and deliver something of value to someone
- Trial-and-error – thrash about, reflect on what happened, repeat with control, thrash more.
There's nothing valuable under here – just no need to show you half-thought-through bits that didn't make this page.
ONLY SPRUE, FROM HERE ON
Mutating by hand
-
Your Stories – 10 min collective
How do you work together
-
Meet your peers – 10 min small groups
Needs a purpose – perhaps combo with your stories
-
Mutation basics - 20 min small groups
Get online. Open a repl.
- Play with the subject.
- Open the tests, run the tests, inspect the tests
- Run mutation
- Assess mutations – show survivors
- Improve tests
- Mutate again
- ? Does teh mutatio testing affect the code, or the tests?
- ? Comments on mutation – costs and uses
-
Mutation 2 – 30 mins focussed groups
Decide on level. Get into groups with same level.
Get online. Open the repl for the level.
- Play with the subject.
- Open the tests, run the tests, inspect the tests
- Run mutation
- Assess mutations – show survivors
- Improve tests
- Mutate again
-
Discussion – costs and uses
-
Our story
Narrative, to follow from "your story"
How we worked together
Overview and insight
What we learned about Mutation Testing
What we learned about learning together
Environments for Testing
All environments are repl.it environments.
To get hands-on., you'll need to sign up for repl.it. You'll need to fork the repls you're working with.
If you don't want to sign up, you can watch or pair with someone – even James or Bart.
All are Python 3 coding environments, and have the tool `MutMut` installed.
You can run MutMut in all with the commandline `mutmut run`. Here's a video as a demo.
Env 1 – basics
Source and tests are for a few lines of code which find the maximum value of a numeric list.
Env 2 – moving on
Source and tests are for ? a dice game., Greed.
Env 3 – knock yourself out
Source and tests are for the Gilded Rose Kata.
Learning strategies (and tactics...)
- Authority-first – follow the book, ask the expert
- Promise-driven – commit to do something you don't know how to do,
- Confusion-driven – try to understand the part you recognise as a part, yet understand the least
- Foundation-driven – work from what you already know, and expand outwaerds
- Literature survey – what are the key words? Go search, building a collectioon of core vocabulary (words and concepts). Do they mean different thigns to different groups? What are the core articles / sites / authors / groups / magazines / books / exercises / metaphors?
- Ask publicly for help –
- Aim to teach / write
- Value-driven; find and deliver something of value to someone
Working in repl
- open one of the workshop repls
- To run the tests, go to the command line and enter the command `python -m pytest tests`
- To check the tests with `mutmut`, go to the command line and `python -m mutmut run`. You may want to delete the mutmut cache first –
- To
- go to the command line:
- delete the mutmut cache:
- browse a file
- change control – github
About mutating Gilded Rose
To do
- get coverage metrics
- - get readable coverage metrics
- Analyse each mutation in the three target environments
¿¿If you want to work on your own, you'll need to make your own account to clone our environments. If you want to work in a group, one of you can have an account, and invite the others.
¿¿ We'll have a few environments that we can invite you to. We'll need you to give us your email address, and we'll need to invite you in the workshop.
¿¿ Set up several envs - perhaps `one of each for 5 ?different areas
¿¿ join link per repl?
¿¿ JL - check 1) learn envs before dup 2) coverage 3) mumut 4) tests