Iterating for Exploratory Testers

still being built – trust nothing

Exploration needs iteration: if you stay in one place you’ll run out of new things to explore. Here are a collection of exercises that might help us think about how we iterate while doing exploratory testing.

This is a growing collection of rough exercises. I'll refine and add over time. For Workroom PlayTime, we'll pick one, and I expect that we'll return to this over time.

For Workroom PlayTime 055, we'll do...

Exercise 3 – What Iterables?

An iterable, in this exercise, is a set of similar things that you can loop over, doing much the same thing to each, judging against similar obstacles.

5 minutes

Think of how you might iterate over several of the following while exploring.

• menu choices in app
• buttons available on screen
• regional variations in regulation
• last week's fixes
• error messages

Share with a public sticky note when done. You might need to set out your context to make sense of your answer – you might have a different context for each, too.

5 minutes

Privately write down at least six more iterables that you might iterate over when exploring. On public sticky notes, share just three of those.

10 minutes

Share ways you might iterate over everyone els's new iterables. While doing that, note clusters in what y0u're acting on, the action you're taking, the observations you might make.

Exercise 1 – Telling Stories about Testing

5 minutes
Write down several stories about exploratory testing. Stories should be swift and short: you need three or more.

5 minutes
For those stories, consider where you start, and how you move, how you steer, where you stop, what you cover.

10 minutes
Share some differences between your own stories.

Explore some differences between each other's stories.

Exercise 2 – Iteration Spotting

This is written as an introspective solo exercise. I think it would be better as a pair exercise; one person exploring, the other thinking of iterables that the first is using.

Explore {something}. Watch to see where you have a sense of iterating – you find a set, or a range, or something where you'll try a collection.

As you explore, consider what characteristics the iterable has, and the method of iteration.

Talk, conclude, share.

Exercise 4 – Edges

10 mins

Pick one of the questions below. Write an answer to help you think

• Iterating consumes something: starts with a pile of unused stuff, and transfers items over to a pile of used stuff. Why might you go back to some things that you have already used, when iterating?
• What happens if your iteration approach (i.e. how you're stepping through / picking next) changes your results? Think of an example:
• Do you 'steer' in an iteration? How do you steer?
• How can you manage iteration across more than two variables?
• What is the relationship between sampling and iterating in your own exploratory testing?

5 mins

Read everyone else's Q&A.

5 mins

Dot vote two of the Q&As – those with the most dots can volunteer to take / answer questions on their treatment.

Exercise 5 – Internal / External

Split into two groups. The groups do not have to be the same size.

Discuss what makes an internal iterable, and what makes an external iterable.

One group writes down as many internal iterables as possible. The other writes down as many external iterables.

Switch roles. Swap lists, if you like. Do the other one.

Share results.

Exercise 6 – Common Iterables for Exploratory Testers

See tours (below). Better: try {an artefact} and imagine what you'd do on {tours}

Exercise 7 – Completion and Sampling

Consider something real that you have explored – preferably software that you have tested.

What is a ‘set’ that you can try every piece of?

What is a ‘set’-ish that you can only try some of? How do you choose what to try?

Exercise 8 – tools in a loop

What tools can you use to iterate? What can you iterate over?

Build an iterator (for real if you can, if you can't then do it as a thought experiment)

• how do you read the results?
• in what way is that iteration exploration?

Exercise 9 – Steering

Using {artefact} and at least one thing to iterate over, start exploring – and identify where you need to change the step size / direction. Resist – or give in. Why?

Share your insights into the differences between iterating on a path, and steered iteration.

Exercise 10 – Hold still

Using {artefact}, iterate over {range} with {tool}.

Now use {tool} to hold a value steady while everything else changes.

Share how the different approaches to exploration felt.

Exercise 11 – Lost toys

Imagine you're looking for something – you know what it is, you had it in your hand a moment ago. It's got to be here somewhere. Now you can't see it.

• How do you go about looking for it?
• Write a short set of instructions for a child, to help them look for a misplaced toy in a similar circumstance. Doesn't have to b the same, if your approach works.
• Pivot to software testing: in what situation might (one or both) of those approaches work? What are you iterating over?
• Share that situation (and the approach) to the group, with a specific example if you have one.

Sources and thoughts

An Iterable has...

My vocabulary: An iterable – something one can iterate over – will have individual things. Those things will be related in some way to make a set.

A range is a one-dimensional set. Iterating on a range typically means going from one end to the other. More sophisticating iterations may go exponentially, or hunt by binary ?splitting.

Examples of discontiguous ranges might be integers from 0 to 9, month names from January to December. Those have serially-related individuals. A continuous range might be numbers from 0 to 9 which has a delimited but infinite set of values (note that this is theoretical: when represented in a machine, that representation imposes quantisation).

A range may have behaviour changes: imagine an integer age range for car insurance.

Some sets may be multi-dimensional. See ESH variables in Explore It! for deeper. ?Set of sets?

Some sets may be populations – made of individuals with loose relationships. Iterating involves picking an unused individual. Examples are functions, or navigation.

We can iterate over inputs and outputs, of course – and with forethought we can iterate over states, records and other more internal representations. We can iterate over things that aren't behaviours, but do belong to the system – lines of code / decisions / data representations.

We might also iterate, as exploratory testers, over externals – requirements, examples, tests.

Every iteration has a sense of done-ness or coverage – and the many measures of coverage give a sense of the different ways we can iterate.

I think of exploratory surfaces – a way that the context invites us to iterate. Behaviour is one surface. User inteface another. An API or CLI another. Code another. Bugs another. Change history another. Logs. Data. On we go.

Step back. What exercises?

If a range

• Somewhere to start
• A sense of 'step' – maybe regular, maybe
• Somewhere to stop – a number of steps, a maximum or minimum size.

If a population

• a picker of some sort
• a way to tell which individuals in the population have been tried

If multi-dimensional, then what variables – or perhaps think in a different way

Tours

Just search for it. You'll find a stack; they won't cover the gamut of what you can think of, but may help you think of gaps. Make a list whichever ones appeal to you, and a couple that raise your eyebrows. Keep the list somewhere you can see all of them at once when you want to.

Here are links to the ideas in Ch 4 of Exploratory Software Testing (Whittaker, 2009), Ch 3 of Taking Testing Seriously (Bach, Bolton 2025). They're in my materials from 2005 or so, so mush have been in common use among the gang earlier – it's not exactly a big metaphorical leap from "exploratory" to "tourism"...

Stopping Heuristics for Exploratory Testing

Let’s distinguish being stuck from stopping. Stuck is when you can’t move despite (something). Stopping is when you’ve run out of resource.

A great way to run out is when you’ve set yourself a small budget – a timebox, a stack of discoveries: running out means it’s time to move on and switch to a different location or approach.

Here’s my non-exhaustive list of situations to stop, and here’s a better list from Michael Bolton and James Bach:

• out of resource: time / money / licenses
• found enough (by number)
• found enough (a big-enough problem)
• found nothing of note after some time (equivalent to out of resource – but time's the one you can't buy)
• answered all the outstanding questions (you did have questions...)