Unix Tools for Testers

Tools Jan 6, 2025 (Jun 21, 2025) Loading...

A series on how I use these as a tester – probably to be used in Workroom PlayTime, and to support workshops on toolsmithery.

Published info and exercises

diff – info – exercises
grep – info – exercises
sort – exercises
cat, head, tail – info – exercises

It's good to know how these simple tools can be used together.

Example 1

It may be that you've got a log from a test system and you want to extract all lines that refer to a particular transaction that seemed problematic. You'd grep.
Maybe you've got several logs, and you need to look for every reference. You could merge them while paying attention to the timestamps with a sort, then grep, and you'd join those two with a pipe |.
Maybe those logs are remote, but you can get them with cURL. You | them into sort. Maybe you want to add a suffix to each line, so that you can see the source. You curl | sed | sort | grep
Maybe you can't read the pages of scrolling weirdness. You | the grep into tail, to see the last 10 lines.
At the end, you've got a single hard-to-read but repeatable command curl | sed | sort | grep | tail – you use it once and throw it away. Or keep it in a handy spot.

Example 2

You've got a huge json record, containing some slice of a data state of a whole system – you want summaries for some test accounts, but it's got sooo much more. The json's invalid: it's got bad escapes in several places, but always in the same way and not anywhere you're interested in. You use cat to send the contents to sed and knock out the offending bits with a regular expression, send the now-valid json to jq and pick out the repeating parts you need. You want parts of all of them in a table – you tell jq to pick out what you need, transform the dates to human-readable, and output as a csv . Because this is a command, you make it part of the start and end of every big test run and drop the manageable files into... I dunno, Excel? so that you can see the context of each account as you explore.

Lots of utilities do text processing of some sort – filtering, sorting, aggregating. To get that text, other utilities pull information from files, the file system or processes. Information is passed from one utility to another with a handful of operators, and the end result is typically dumped into an editor or a file for someone to play with – or for another process to pick up.

As a tester, I've used these tools to check changes to my environment, to pick out specific data, to gather context for my exploration, to watch what's going on as I work, to filter the firehose.

Testers don't (typically) write stuff for permanent deployment: their tools are ephemeral. Those tools can still be shared, reused and tuned. However, capabilities built with these general tools are often so specific – and so hard to interpret – that they don't get shared, and get remade rather than reused.

By knowing what's possible, we can build things that are practical, and I hope that we will share and learn from each other.

It's a great time for testers to get better at using tiny tools: The rise of StackOverflow and blogging has led directly to a greater exchange of tricks, and LLMs, trained on these freely-available sources, are extraordinary at helping me to swiftly get to the tooling I need. I can ask, iterate and learn about what can be done, and options around how to do it, without getting too stuck on syntax and cluelessness.

Some sources:

The key permanent / formal collection that these are drawn from is probably the IEEE / Open Group's 'shell and utilities' page, and it's a beast.
Most of these are found in GNU core utilities (here's WP's page on core utilities), and there's a substantial crossover with Portable Operating System Interface Commands (so that's what POSIX means...).
Here's a GNU's page of coreutils's faqs, which outlines some weirdnesses, edge cases, and unexpected expected behaviours. And here's a pair which are interesting not only because of their content but taken a pair because they're written by the same person (GNU coreutils maintainer Pádraig Brady): How the GNU coreutils are tested, Coreutils Gotchas.