Exercise: Becoming Coverage

A game to play with colleagues to understand coverage more deeply. You’re going to pretend to be a metric!

Logistics

A handful of people can get useful value from this in an hour.  This exercise can spin on for a long time, and lose people as it goes. Consciously choose to use a timebox to get to a fruitful end together.

Gather a group for a known time around a whiteboard or other shared visual space. I use Miro if we’re online.

Before the workshop, make everyone aware of this text, ask them to read (and extend / correct if interested) the overview, coverage types and questions. Extra kudos if you can

Priming - 5 mins

Briefly exchange examples where someone in the group has seen or heard the term ‘coverage’ in use in their work.

Game - split available time evenly between participants:

Take turns pretending.

• One person pretends to be a specific meaning of coverage (see types below)
• Other people ask questions about that meaning (see questions below)
• When one is done, repeat with a new person and measure

This is a conversation starter. Connection is more important than consensus. This not a test about who knows most: there are no right answers during the game.

So: please bear in mind this important note about answers.

Closing debrief - 10 mins

Pick some from:

• Share surprises
• Share how you might make use of different kinds of coverage.
• Share how you might help people to understand what each other might mean by ‘coverage’.
• Discuss whether you need a shared definition of coverage
• Complain about the misunderstandings you’ve experienced
• Consider what action you could take to decrease confusion / increase clarity around ‘coverage’ as used in docs and conversation.

Resources

Coverage - overview

Coverage is a proxy for Done-ness - people use it with the aim of assessing whether / when they stop.  It is an assessment of the work, not (generally), a measure of the end product or its quality. However, while coverage indicates doneness of work, coverage trends can tell us that the product has changed, or may tell us that we have adjusted our idea of the problem.

Coverage can go down as well as up – this is typically because we realise we need to work more than we had expected. This is good news, because we’ve learned necessary things about out problem. It is also bad news, because we while the shape of the problem is more clear, the solution is further away.

Coverage means different things to different people, especially with different senses of ‘done;. Those differences may not be clear. Look out for situations where ‘coverage’ is passed around as a generally good thing, with no actual measure attached.

A particular test technique may imply a specific kind of coverage (i.e. unit testing w/ lines of code, req testing with requirement coverage). Some authorities suggest that you should measure goodness of your testing by a different measure from the one that you used to drive your design (i.e. see if your req-based tests give you much code coverage).

Sources - add your own!

Code coverage - Wikipedia
Kaner: Software Negligence & Testing Coverage

Types of coverage - add your own!

• code - lines exercised by a set of tests
• code - decisions
• code - paths (combinations of decisions)
• code - files / modules / objects / functions
• data elements - crud(ms)
• functionality
• UI elements
• Pages
• Menu items
• System parts
• Transaction types
• File types
• Record types
• Requirements
• Acceptance criteria
• Service-level agreements
• Fixed bugs
• Edge cases
• Planned tests
• Risks
• Personas
• Account types
• Character sets
• Platforms (Browsers / Operating systems)
• Edge cases
• Known errors
• Error messages
• Default values
• Data combinations

Questions about coverage - add your own!

• How does one measure this metric?
• What does this  measurement tell us?
• What does this metric leave out?
• What situations suit this metric?
• What does “coverage going up” mean for this metric?
• What does “coverage going down” mean for this metric?
• Is there a minimum? What does minimum coverage mean?
• Is there a maximum? What does maximum coverage mean?
• What might it mean if  doing work, over a given period, makes no change to the metric?

See Questions About Metrics for more.