Exercise – Power of Variety
A 20 minute exploration via a simulation
history
Based on an older exercise, no longer interactive, posts here: An Experiment with Probability, Broken Trucks, Models, lies and approximations, Enumeration hell, Diversity matters, and here's why, Modelling super powers . Go there to read background – I'll bring it here in the next few weeks and update it.
about the simulations
Each of these simulations have a similar core simplifications:
- the simulation knows what can be found, and it knows what's likely to find each thing, but the entities looking don't know either of these things.
- A searchable thing has a limited collection of independent things to be discovered. For testing, think: there are only so many problems.
- Discovery is by chance. Different approaches to searching have different chances. A discoverable thing has a low chance of being discovered by any individual approach – but might have a better chance by another approach. For testing, think: Performance testing will find different problems from usability testing.
- There's a limited budget – the more you spend, the more chances you have to find stuff.
- Something stays found, but only counts the first time it is found.
You, the user, can see and change the number of things that can be discovered, the makeup of the crew that looks for things, and the budget to spend looking. During the simulation, you can see the results of the simulation in terms of the work done, the discoveries found. You can pause and resume the work. You can increase and decrease the budget. Different simulations will throw up different results, and you'll want to keep track of those to see the variation.
Background: a simulation works in chunks. In each chunk, all the explorers have a chance to discover a (random) set of discoverables (of fixed size in the simulation). A discoverable's chance of discovery is set by the explorer's active skill, and the chance of the discoverable being discovered by that skill.
Priming Question
We'll come back to this in the debrief
Get familiar with the simulator
5 mins
Go to https://exercises.workroomprds.com/discoverysim/3/index.html, and set the simulation to One tester, one technique / very sparse, all similar.
Tap the ⎋ button – watch the "things to find" text change. Tap the + and - buttons to see the budget change.
Hit 'start' – see that the budget goes down and that the button now says 'working - pause'. Hit 'pause'. Unpause. Change the budget. Watch the rest of the UI – we'll talk about what it's showing.
Simulation 1: Diminishing returns
5 minutes
Switch to 'Ordinary - A' as the subject and stay with 'one tester, one technique'.
Start the simulation and 'spend' half the budget. At that halfway point, pause, and decide whether it's worth spending the other half. Do this a few times, and see what's influencing your decision.
Change your team to 'six testers, one technique'. You'll see a list of workers.
Run again, several times. What do you see that is different? Has that changed your decision-making?
We'll talk as we go.
Simulation 2: Varied workers
5 minutes
Change your team to "six testers, ten techniques". You'll see your workers have a selection of approaches. Click on the approaches so that they're all different.
Run the simulation, several times and see how the stats change.
What do you see that is different? Has that changed your decision-making?
Simulation 3: One worker, several approaches.
optional
Switch back to a single tester with several skills ("one tester, ten techniques").
Run the simulation, switching their skill as they work.
What do you see that is different? Has that changed your decision-making?
Debrief
5 minutes
How do different approaches affect your decisions about budget and stopping
How do you feel about spending, and stopping, in a situation where you need to discover new things?
Comments
Sign in or become a Workroom Productions member to read and leave comments.