Iterating for Exploratory Testers

still being built – trust nothing

Exploration needs iteration: if you stay in one place you’ll run out of new things to explore. Here are a collection of exercises that might help us think about how we iterate while doing exploratory testing.

This is a growing collection of rough exercises. I'll refine and add over time. For Workroom PlayTime, we'll pick one, and I expect that we'll return to this over time.

Exercise 1 – Telling Stories about Testing

5 minutes
Write down several stories about exploratory testing. Stories should be swift and short: you need three or more.

5 minutes
For those stories, consider where you start, and how you move, how you steer, where you stop, what you cover.

10 minutes
Share some differences between your own stories.

Explore some differences between each other's stories.

Exercise 2 – Iteration Spotting

This is written as an introspective solo exercise. I think it would be better as a pair exercise; one person exploring, the other thinking of iterables that the first is using.

Explore {something}. Watch to see where you have a sense of iterating – you find a set, or a range, or something where you'll try a collection.

As you explore, consider what characteristics the iterable has, and the method of iteration.

Talk, conclude, share.

Exercise 3 – What Iterables?

5 minutes

Think of how you might iterate over the following while exploring. Share with a public sticky note when done. You might need to set out your context to make sense of your answer – you might have a different context for each, too.

• menu choices in app
• buttons available on screen
• regional variations in regulation
• last week's fixes
• error messages

5 minutes

Privately write down at least six more iterables that you might iterate over when exploring. On public sticky notes, share three of those.

10 minutes

Share ways you might iterate over everyone els's new iterables.

10 minutes

Talk and find clusters

Exercise 4 – Edges

10 mins

Pick one of the questions below. Write an answer to help you think

• Why might you go back to some things that you have already used, when iterating?
• What happens if your iteration approach (i.e. how you're stepping through / picking next) changes your results?
• Do you 'steer' in an iteration? How do you steer?
• How can you manage iteration across more than two variables?
• What is the relationship between sampling and iterating in your own exploratory testing?

5 mins

Read everyone else's Q&A.

5 mins

Dot vote two of the Q&As – those with the most dots can volunteer to take / answer questions on their treatment.

Exercise 5 – Internal / External

Split into two groups. The groups do not have to be the same size.

Discuss what makes an internal iterable, and what makes an external iterable.

One group writes down as many internal iterables as possible. The other writes down as many external iterables.

Switch roles. Swap lists, if you like. Do the other one.

Share results.

Exercise 6 – Common Iterables for Exploratory Testers

See tours. Better: try {an artefact} and imagine what you'd do on {tours}

Exercise 7 – Completion and Sampling

Consider something real that you have explored – preferably software that you have tested.

What is a ‘set’ that you can try every piece of?

What is a ‘set’-ish that you can only try some of? How do you choose what to try?

Exercise 8 – tools in a loop

What tools can you use to iterate? What can you iterate over?

Build an iterator (for real if you can, if you can't then do it as a thought experiment)

• how do you read the results?
• in what way is that iteration exploration?

Exercise 9 – Steering

Using {artefact} and at least one thing to iterate over, start exploring – and identify where you need to change the step size / direction. Resist – or give in. Why?

Share your insights into the differences between iterating on a path, and steered iteration.

Exercise 10 – Hold still

Using {artefact}, iterate over {range} with {tool}.

Now use {tool} to hold a value steady while everything else changes.

Share how the different approaches to exploration felt.

Sources and thoughts

An Iterable has...

My vocabulary: An iterable – something one can iterate over – will have individual things. Those things will be related in some way to make a set.

A range is a one-dimensional set. Iterating on a range typically means going from one end to the other. More sophisticating iterations may go exponentially, or hunt by binary ?splitting.

Examples of discontiguous ranges might be integers from 0 to 9, month names from January to December. Those have serially-related individuals. A continuous range might be numbers from 0 to 9 which has a delimited but infinite set of values (note that this is theoretical: when represented in a machine, that representation imposes quantisation).

A range may have behaviour changes: imagine an integer age range for car insurance.

Some sets may be multi-dimensional. See ESH variables in Explore It! for deeper. ?Set of sets?

Some sets may be populations – made of individuals with loose relationships. Iterating involves picking an unused individual. Examples are functions, or navigation.

We can iterate over inputs and outputs, of course – and with forethought we can iterate over states, records and other more internal representations. We can iterate over things that aren't behaviours, but do belong to the system – lines of code / decisions / data representations.

We might also iterate, as exploratory testers, over externals – requirements, examples, tests.

Every iteration has a sense of done-ness or coverage – and the many measures of coverage give a sense of the different ways we can iterate.

I think of exploratory surfaces – a way that the context invites us to iterate. Behaviour is one surface. User inteface another. An API or CLI another. Code another. Bugs another. Change history another. Logs. Data. On we go.

Step back. What exercises?

If a range

• Somewhere to start
• A sense of 'step' – maybe regular, maybe
• Somewhere to stop – a number of steps, a maximum or minimum size.

If a population

• a picker of some sort
• a way to tell which individuals in the population have been tried

If multi-dimensional, then what variables – or perhaps think in a different way

Tours

Just search for it.

Who was the progenitor? Whittaker?

Stopping Heuristics for Exploratory Testing

Let’s distinguish being stuck from stopping. Stuck is when you can’t move despite (something). Stopping is when you’ve run out of resource.

A great way to run out is when you’ve set yourself a small budget – a timebox, a stack of discoveries: running out means it’s time to move on and switch to a different location or approach.

Here’s my non-exhaustive list of situations to stop, and here’s a better list from Michael Bolton and James Bach:

• out of resource: time / money / licenses
• found enough (by number)
• found enough (a big-enough problem)
• found nothing of note after some time (before out of resource?)
• answered all the outstanding questions (you did have questions?)