Shaping Testing With Charters
placeholder – needs the exercise extracted, and needs to be properly integrated.
Charters are one way of giving purpose to your exploration and exploratory testing. They tend to be talked about by testers as an integral part of Session-Based Testing.
Charters are Work, Sessions are Time
A charter is a unit of work. It has a purpose. A charter may be done over and over again, by different people. When planned, it's often given a duration – the duration indicates how much time the team is prepared to spend not how long it should take.
A session is a unit of time. It is typically 10-120 minutes long; long enough to be useful, short enough to be done in one piece. A session is done once.
The team may plan to run the same session several times during a testing period, with different people, or as the target changes.
Ownership
Anyone can add a new charter – and new charters are often added. When tester needs to continue an investigation, but wants to respect the priorities decided earlier, they will add a charter to the big pile of charters, then add that charter somewhere in the list of prioritised charters – which may bump another charter out.
Making Charters
In Explore It, Elisabeth Hendrickson suggests
A simple three part template
Target: Where are you exploring? It could be a feature, a requirement, or a module.
Resources: What resources will you bring with you? Resources can be anything: a tool, a data set, a technique, a configuration, or perhaps an interdependent feature.
Information: What kind of information are you hoping to find? Are you characterizing the security, performance, reliability, capability, usability, or some other aspect of the system? Are you looking for consistency of design or violations of a standard
I've found it helpful to consider a charter with a starting point, a way to generate or iterate, and a limit (which may work with the generator), and to explicily set out my framework of judgement.
Where to Start
Writing charters takes practice. A single charter often gives a scope, a purpose and a method (though you may see limits, goals, pathologies and design outlines). You could approach it by considering...
- Existing bugs – diagnosis
- Known attacks / suspected exploits / typical pathologies
- Demonstrations – just off the edges
- Questions from training and user assessment
A collection of charters works together, but should always be regarded as incomplete. You're investing resources as you work on whatever you choose to be best, not trying to complete a necesary set. A collection for a given purpose (to guide testing for the next week, say) is selected for that purpose and is designed to be incomplete.
Here's a foldup collection of charter starters
Charter Starters
Use these to give shape to early exploratory test efforts.
These are unlikely to be useful charters on their own – but they may help to provoke ideas, clarify priority, and broaden or refine context.
- Note behaviours, events and dependencies from switch-on to fully-available
- Map possible actions from whatever reasonably-steady state the system stabilises at after switch-on. Are you mapping user actions, actions of the system, or actions of a supporting system?
- How many different ways can the system, or a component within that system, stop working (i.e. move from a steady, sustainable state to one unresponsive ton all but switch-on)? Try each when the system is working hard - use logs and other tools to observe behaviours.
- Pre-design a complex real-world scenario, then try to get through it. Keep track of the lines of enquiry / blocked routes / potential problems, and chase them down.
- What data items can be added (i.e. consumable data)? Which details of those items are mandatory, and which are optional? Can any be changed afterwards? Is it possible to delete the item? Does adding an item allow other actions? Does adding an item allow different items to be added? What relationships can be set up between different items, and what exist by default? Can items be linked to others of the same type? Can items be linked to others of different types? Are relationships one-to-one, many-to-one, one-to-many, many-to-many? What restrictions and constraints be found?
- Try none-, one-, two-, many- with a given entity relationship
- Explore existing histories of existing data entities (that keep historical information). Look for bad/dirty data, ways that history could be distorted, and the different ways that history can be used (basic retrieval against time, summary, time-slice, lifecycle).
- Identify data which is changed automatically, or actions which change based on a change in time, and explore activity around those changes.
- Respond to error X by pressing ahead with action.
- Identify potential nouns and verbs - i.e. what actions can you take, and what can you act upon? Are there other entities that can take action? What would their nouns and verbs be? Are there tests here? Are there tools to allow them?
- Identify scope and some answers to the following: In what ways can input or stimulus be introduced to the system under test? What can be input at each of those points? What inputs are accepted or rejected? Can the conditions of acceptance or rejection change? Are some points of inputs unavailable because they're closed? Are some points of input unavailable because the test team cannot reach them? Which points of input are open to the user, and which to non-users? Are some users restricted in their access?
- Identify scope and some answers to the following: In what ways can the system produce output or stimulate another system? What kinds of information is made available, and to what sort of audience? Is an output a push, a pull, a dialogue? Can points of output also accept input?
- Explore configuration, or administration interfaces. Identify environmental and configuration data, and potential error/rejection conditions.
- Consider multiple-use scenarios. With two or more simultaneous users, try to identify potential multiple-use problems. Which of these can be triggered with existing kit? If triggered, what problems could be seen with existing kit, and what might need extra kit? Try to trigger the problems that are reachable with existing kit.
- Explore contents of help text (and similar) to identify unexpected functionality
- Assess for usability from various points of view - expert, novice, low-tech kit, high-tech kit, various special needs
- Take activity outside normal use to observe potential for unexpected failure; fast/slow action, repetition, fullness, emptiness, corruption/abuse, unexpected/inadequate environment.
- Identify ways in which user-configurable technology could affect communication with the user. Identify ways in which user-configurable technology could affect communication with the system.
- Pass code, data or output through an automated process to gain a new perspective (i.e. HTML through a validation tool, a website through a link mapper, strip text from files, dump data to Excel, job control language through a search tool)
- Go through Edgren’s “Little Black Book on Test Design”, Whittaker's "How to Break..." series, Bach's heuristics, Hendrickson's cheat sheet, Beizer-Vinter's bug taxonomy, your old bug reports, novel tests from past projects to trigger new ideas.
Stuff to do
Exercise: Writing Charters
20 minutes, groups or individually
- Pick a subject – preferably your own system. If you want a new one, I suggest https://flowchart.fun/
- Write at least four charters – one to explore in a new way, one to investigate a known bug, one to search for surprises, one to work through a list
- We'll talk about those charters
Further work – prioritise and run the sessions.
Comments
Sign in or become a Workroom Productions member to read and leave comments.