Open Miro frame. Return to central page.
We missed the last exercise (pre-filled list).
Exploratory testing without tools is weak and slow.
In this session, we'll look at those tools which help you to explore a system, and concetrate specifically on generation tools.
Exercise: Tools In Use
10 - 15 mins
Everyone: type the name of a tool you use into the chat.
Do unmute – I may invite you to describe what you use that tool for. As people talk, add the tools which come to mind (one at a time works better than a big chunk).
Tool Types
There are lots of ways to group tools. Here's one – I use it to think of the tools I might use. I also use it to consider what I might repurpose.
Observation tools
- database contents, environments, match/compare
- object browser, text stripper, mapper
- screen capture, keylogger, transaction logs, system logs
Action / setup tools
- test harness, replay tool, macro tool, data loader
- generators for data, transactions, load, system messages and events
- emulator / simulator
Analysis tools
- code analysis, data verification
- unit tests, coverage
- data analysis, patterns and clusters, graphing
Bulk Generators
Generators are powereful tools for the exploratory tester, allowing you to build a test from thousands of similar interactions. Working in bulk means we need ot design differently, and opens a door to a more experimental approach, one which is less deterministic and sometimes less mathematical. Working in bulk requires us to take in the results of thousands of tests, which means we need to observe differently – and gives us more opportunities to get lucky.
We'll use Converter_v3 to explore bulk testing. When we wiggle a slider, we're using a generator which gives us immediate feedback (this is unusual in real-world systems, and is to help you feel how bulk testing works).
Slider I
In this exercise, you'll switch to changing the input swiftly and perhaps carelessly.
Exercise: Input with a Slider
5 minutes, mostly debrief, working as a whole group
Choose the radio button labelled "slider"
Work with the slider to reveal more about the system under test. Do you need more than a minute?
Debrief – What's different about your interactions with the system under test, when you're using the slider?
Slider II
In this exercise, you'll use a slider whcih lets you set the ends. You typically woudn't find this on a UI, but it will push you to think about test design.
Exercise: Input with a Parametric slider
10 minutes, mostly debrief, working as a whole group
- Choose the radio button labelled "parametric slider"
- Choose the start and ends of the slider. Play with the slider. What have you found?
- Pick a target, share it with the group in the chat.
Debrief: Share your slider ends (and perhaps why you chose them). Share what you learned about the system under test.
Linear generation
In this, you'll generate values with regular steps. You can pick the start and the step size; the generator will make 1000 regularly-spaced values. This generates and converts all the values at once.
Exercise: Generated Input I
5 mins, small groups, 5 mins group debrief
- Have a look at generating numbers from 1, with a step size of 1. Or from 0, with a step size of 0.01. Or from 0, with a step size of 3600.
- Pick some more ranges. Why those ranges. What do they tell you? What do you want to do next?
Debrief: how did you react, as a tester, to the information in these bulk tests?
Exponential generation
Different units are interesting when converting time – but they're not evenly-distrubuted. The smaller the number, the more closely-spaced the units.
When you're generating, you need to think about how you're picking the values. This generates numbers with a distribution which generates more tiny numbers than huge numbers.
This generator is good for looking at unit boundaries. Pick a number to start at – you'll find that number in the middle of the range.
Exercise: Generated Input II
10 mins, small groups, 5 mins debrief
- Try 1 and 1.001, then 1 and 2, then 1 and 10
- Pick some more ranges. Why those ranges. What do they tell you? What do you want to do next?
Debrief: how did you react, as a tester, to the information in these bulk tests?
Pre-filled bulk input
In this exercise, you'll work with a list of inputs.
Exercise: pre-filled input
Use the pre-filled list to see how several values are converted. Feel free to change those values.
Debrief questions:
- Do you see any surprises – reasons to change the model of the underlying system which you build in the previous exercise?
- Do you judge any of the outputs to be wrong? What are you basing that judgement on?
- How did you change this collection of tests? Why?
Extras
A few things we won't have time for.
Pre-filled bulk input
In this exercise, you'll work with a list of inputs.
Exercise: pre-filled input
Use the pre-filled list to see how several values are converted. Feel free to change those values.
Debrief questions:
- Do you see any surprises – reasons to change the model of the underlying system which you build in the previous exercise?
- Do you judge any of the outputs to be wrong? What are you basing that judgement on?
- How did you change this collection of tests? Why?
Every experiment must enable us to make a maximum number of predictions having the highest possible degree of probability. The problem is, so to speak, to increase the output of the scientific machine. I may be permitted to compare science to a library which must go on increasing indefinitely; the librarian has limited funds for his purchases, and he must, therefore, strain every nerve not to waste them. Experimental physics has to make the purchases, and experimental physics alone can enrich the library. As for mathematical physics, her duty is to draw up the catalogue.
Henri Poincaré. "Hypotheses in Physics". Chapter 9 in Science and Hypothesis . London: Walter Scott Publishing (1905)
What to automate
If you're testing a UI, use a UI testing tool. If the thing you're testing has a UI, but you're testing through the UI, find another way. Concentrate instead on automating: environments, APIs, database access, file access, permissions, data creation / comparison, test scenario generation, set up, day-to-day admin.
If you find youself automating a terminal, a green screen, or a graphical tool to make a machine easy to use for a human, stop. Consider if that's the only way, or the best way, to do it – you may simply be automating the actions of a tester.
JL – Approval Tests, Property Tests