Exploration and Testing (BBC 23Q2)

Open Miro frame. Return to central page.

thinking of exploration as purposeful play...

Exploratory Testing can be thought of as exploration guided by :judgement.

In this section, we're going to start testing.

As you explore, you'll find that some things catch your attention. If something seem wrong, then try to listen for that feeling. Notice when it:

  • sits at the edge of your senses, without pointing at anything specific.
  • shouts so loud that you can't explore anything else
  • arises, yet is immediately flattened by another part of your mind
  • tells you what is wrong, but not why

Exercise: Timer

10 mins exploring, 20 mins debrief and re-exploring

Explore the timer.

Work in groups or alone. Make notes to share. Your purpose is to find things that are wrong. Put a sticky on the board with a short description โ€“ use the iritation box.

Debrief: We'll pick out some of the stickies. You'll tell the group what's wrong, how you recognised it, what you did with the feeling. You'll post your notes, if you like.

James will set up several charters that go further into the problems you found โ€“ we'll put those in the investigation box and use them in the next exercise.

Timer (test subject)

A Framework for Judgement

Judgement requires comparison. You'll have made several comparisons while you worked, judging what you found against what you expected. Consider how those would fit into the following classification:

  • external inconsistency (vs an external authority i.e. a spec)
  • internal inconsistency (vs another similar thing)
  • cultural inconsistency (vs my own expectations)
  • absence: something is surprising by its absence
  • extra: something is surprising by its presence
There's a trap, here, for experienced testers. The trap is set when you notice an inconsistency with something else which is well-described. If you find yourself working through that other thing (a spec, a standard, a competitor), rather than the thing you're exploring, you've been caught!

Learning more

If you don't have a hunch about what's going on, you'll build experiments to find out more information. Experiments designed to find out more will typically go in several directions to give you a wider set of clues; if you've seen a crash, you'll play more โ€“ and you'll turn on CPU monitors and loging, inspect and reply what you did the first time, keep close track of sequences, data, dependencies and constraints. If you've got a specific behaviour that you're chasing, you'll try to catch it several times with as much sensing as you can muster, and you'll see what variations in situation can lead to the same behaviour, and what similar situations show different behaviours.

If you think you might know what's going on, that's a model. You'll look for evidence to support your model, and should consider looking for evidence that refutes it.

Supporting your model: If you think it's a code bug, looking at the code can give you strong evidence. If you think it's to to with an external dependency, you'll find evidence in changing that dependency. If you think that your experience give you a special perspective, seeking the opinion others who share that perspective gives extra evidence.

Refute your model: You'll consider other plausible models, and build experiments to see whether they might be true. You'll work towards edge-cases in your model, seeing if the behaviour persists when your model indicates that it should not.

Exercise: Where Next?

10 minutes exploring, 10 minutes debrief

Pick a charter โ€“ or make your own (record it on the board if you do, in the investigations box)

Work to the charter โ€“ explore within the time limit, seting your exploration to fit the time, and allowing it to be explicitly guided by your judgement. Post details of your findings on the board, in the Information box.

Debrief: We'll talk about what you found, how you learnt more, what you might do next.

A bug?
Michael Bolton puts it neatly: ยซA bug is anything that bugs youยป.
Your local vocabulary and standards may vary.

Judgement, Discipline and Aesthetics

We need to develop our judgement โ€“ to recognise when something is worth our attention, to make sense of it in context, and to highlight the most important elements.

:Discipline in test design helps us discover information more swiftly or more truthfully.

:Discipline in test recording enables analysis, learning and trustable evidence.

As we develop that judgement, we're building an :aesthetic. Without an aesthetic, it's hard to judge (as you're always seeking external validation of your judgement) and hard to explore well (because you're senses haven't been sharpened).

If your testing has been user-action led, you'll already have a sense of what is right and wrong about (for instance) UIs and user journeys. If you've been involved primarily in code / unit test reviews, you may already have great judgement around what seems good and bad about code, tests and data schemas. Subject-matter experts are used, by testers, for their sense of what's good and bad.

The more novel the thing under test, the more you'll have to work towards a reliable aesthetic.




    function doTests() {
        let verbose = false
        log = function () {
            if (verbose) { console.log(...arguments) }
        assert = function () {
        assertTrue = function () {
            assert(arguments[0], ...arguments)
        assertFalse = function () {
            assert(!arguments[0], ...arguments)

        function tryTestReporter() { // unused on purpose
            assert(true, "should not show this message");
            assertTrue(true, "should not show this message");
            assertFalse(false, "should not show this message")
            //assert(false, "should show this message", "...concatenated with this");

        function testValidTime() {
            log("testing validTime")
            var subject = validTime;
            var examplesOfValidTimes = [
                "00:10", "10:20"

            var examplesOfInvalidTimes = [
                "", "a", "-1", "0:111", "111:00", "00:70"

                (myTime) => {
                    assertTrue((validTime(myTime)), "this should be invalid", myTime)

                (myTime) => {
                    assertFalse((validTime(myTime)), "this should be invalid", myTime)
        function testGetCircleDashArray() {
            log("testing getCircleDashArray")
            let subject = getCircleDashArray;
            assertTrue(subject(1) === "283 283", "full-scale should report 283 283, and reports: ", subject(1))
            assertTrue(subject(0) === "0 283", "full-scale should report 0 283, and reports: ", subject(0))
        function testCalculateTimeFraction() {
            log("testing calculateTimeFraction")
            let subject = calculateTimeFraction
            //assertTrue(subject(0) === 0, "check 0 at end. reported: ", subject(0)) //FAILS
            assertTrue(subject(TIME_LIMIT) === 1, "check 1 at start. reported: ", subject(TIME_LIMIT))
        function testFormatTime() {
            log("testing formatTime");
            let subject = formatTime;
            assertTrue(subject(63) === "1:03", "check just over a min in secs returns 0-padded. Expected 63 to be 01:03, reported ", subject(63)) // broken test left in...
        log("doing tests")

:x Discipline in test design

Building experiments purposefully, and taking action whose purpose is clear.

:x Discipline in test recording

Making the intent of one's actions clear, whether that is action taken in the moment, action captured in a script or automation, or an action that can only be taken with automated assistance.

Recording observations in a way that is reliable and broad, and analysing system behaviours in the light of that evidence.

Making hypotheses and working models clear, and staying alert to the possibility of disconfirming evidence, and whether to find time to seek it.

:x Aesthetic in testing

A personal sense of what is right, and what is wrong. Typically built over time, by exposure.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.