Parts for Tools
A page setting out a framework I use when thinking about making tools.
Why a framework? Because it's handy. I don't know what tools I'll have to hand. A framework for how I can use (and misuse) the things I find around the place can help me to be more resourceful.
A tool, for here, is something that acts for us. We use a tool because we work better with the tool. Tools can be pretty-much anything, by these lights: a checklist, a process, thirty years experience, a pocketknife. For this framework, we're mostly considering executable software. And I have a
Why make tools?
As anyone who has seen a specialist's collection of hammers knows, the right tool for the job can differ from the wrong tool by what might seem to be small details. If you can't configure an existing tool just right, you need to make your tool. You'll need to be purposeful, to experiment, and to use the tools you have around you.
You configure your tools, combine them, and use your new resulting tool for just the purpose you intended.
And then you pick it up again for the wrong thing, and stick the sharp end through your thumb. Careful of what you build.
Framework
I get to the right custom tool more easily if I choose to think. This helps me think.
A whole class of tools transforms data. Let's start with those, because we recognise and can reuse their parts.
- There's usually some part that gathers or produces information. That information has a source, typically a list, which may be in a file from some person, or be extracted from the file system, the running system, the network, or generated.
- Then that data is transformed – and tools that work on lines are different from those that work on columns, because we've got a typical pattern where each line is a similar and countable item, and each column tells us how those items differ. Transformations stack: you might filter for the lines you want, cut out only those characteristics that matter, sort by size, and filter again to highlight some extreme.
- The transformed data is aggregated or compared, taking you to some set of information that is comprehensibly small and directly relevant. And your tiny tool lets you repeat the action, so it can be tested, tuned and replayed when the initial data has changed.
- These parts need to be linked together. That linkage may be very environment-specific and dependent on where you are in a systems interconnection model. Unix systems have the redirection operators, corporates have middleware, the internet has https and (much) more, Macs have whatever restrictive / enabling flow diagram tool the OS is currently favouring...
Another class of tools takes action – again, on the file system, the system, something remote – and also perhaps on the tool and its environment. Perhaps that action is scheduled.
Another class measures something – time, space (storage or memory), CPU taken, files locked, permission.
If I start to recognise that something needs careful judgement or a particularly twitchy algorithm or process, then that's an indication that I probably need to split my end point: I need two tools and at least one of them might involve tools which get a person involved.
My parts list – make your own
This is rather unix-focussed. You'll make your own for the tools and technologies you have at your fingertips.
About this list
While the framework above has been kicking around for ages, I made this tools list with assistance from an LLM, with reference to the POSIX commands and GNU core utilities. Then I fiddled with it. It's a work in progress.
My prompt to Claude 4 Sonnet was:
> We'll be making several table of posix commands (list attached). I want the following tables: commands that produce information (like ls and ps and cat and curl), commands that change information by filtering or rearranging rows (like grep, sort, uniq, head and tail), commands that change information by filtering columns (like cut), commands that change characters (like tr), commands that aggregate or combine (like wc, paste, sum), commands which take action (like curl, mv, sh), commands that set or change context (like cd), redirection commands (like >, | and – for me – tee), commands to start a human interface (like less and top), commands that persist (like cron) or which measure (like time, du). Commands can be in more than one table. If a command is not i any table, put it in a ‘miscellaneous’ table.
> Your tables will have columns. Each command in a table should have all columns filled in if possible (if empty, make a note below the table). Columns include: description in 10 words or less | whether the command is for files (like ls and lsof), processes (like ps and lsof), networks (like netstat), the system (like ps, stat, which) or something else (categorise if I’ve missed something), a column to indicate whether the command is (notably popular, or deprecated, or superseded by something), a column to indicate if the POSIX command is in the GNU coreutils (also attached) and whether (file, text or shell), a column to indicate the other tables a command can be found in.
Info – gather / produce
files and text
| Command | Description |
|---|---|
ls |
List directory contents |
find |
Return files to match criteria |
cat |
Concatenate and print files |
df |
Report free storage space |
du |
Estimate file space usage |
file |
Report type of files |
seq |
Generate number sequences |
echo |
Display text |
system
| Command | Description |
|---|---|
ps |
Report process status |
date |
Report system date and time |
env |
Report environment variables |
newtwork / remote
| Command | Description |
|---|---|
curl |
get info from an internet source |
wget |
get a file from an internet source |
ping |
contact a network address |
ifconfig netstat ss iproute |
network information |
Excel / sheets / notepad / VSCode
| Command | Description |
|---|---|
| excel: paste csv / tsv | import rows, splitting into colums |
| copy from somewhere, paste somewhere else |
Filter and Change
filter / rearrange rows
| Command | Description |
|---|---|
grep |
Search text for pattern |
sort |
Sort lines of text files |
uniq |
Report or filter repeated lines |
head |
Copy first part of files |
tail |
Copy last part of files |
join |
Merge files on common field |
diff |
Compare two files |
| excel: use filters / sort |
filter columns
| Command | Description |
|---|---|
cut |
Cut selected fields from lines |
paste |
Merge corresponding lines of files |
awk |
Pattern scanning and processing |
basename dirname |
Extract filename / directory from path |
| excel: drag + drop / cut or copy + paste |
change characters
| Command | Description |
|---|---|
tr |
Translate characters |
sed |
Stream editor |
expand / unexpand |
Convert tabs to / from spaces |
printf |
Format and print text |
| excel: global find / replace | |
| excel: replace on import |
Aggregate and Combine
| Command | Description |
|---|---|
wc |
Count lines, words, bytes |
paste |
Merge lines of files |
join |
Join files on common field |
sort |
Sort and merge files |
sum cksum |
Checksum |
test |
Evaluate expressions |
| excel: pivot |
Take action
on files
| Command | Description |
|---|---|
> >> |
write to / append to a file |
mv cp cp |
Move or rename / copy / remove files |
mkdir rmdir |
Make / remove directories |
chmod chown chgrp |
Change file permissions / ownership / group |
ln |
Link files |
touch |
Change file timestamps |
trigger and act on processes
| Command | Description |
|---|---|
kill |
Terminate processes |
sh |
Shell command interpreter |
nohup |
Run immune to hangups |
xargs |
do a command on all items |
remote action
| Command | Description |
|---|---|
curl |
send info to an internet endpoint |
ping |
check response |
ssh |
log into a remote |
Set or change context
| Command | Description |
|---|---|
cd |
Change working directory |
env |
Set environment for commands |
Linking and redirecting
| Command | Description |
|---|---|
tee |
Duplicate standard output |
| (pipe) |
pipe output to input |
> |
Output redirection - overwrite |
> |
Output redirection – append |
< |
Input redirection |
Measuring
| Command | Description |
|---|---|
time |
Measure command execution time |
du |
Measure disk usage |
df |
Measure filesystem space |
ps + options |
various measures of processes i.e. memory / CPU / uptime |
Scheduling / repeating
| Command | Description |
|---|---|
cron |
Schedule periodic tasks |
at |
Execute commands later |
watch |
execute a command regularly |
Start a human interface
| Command | Description |
|---|---|
less |
Page through files |
more |
Display files page by page |
top |
Display running processes |
vi |
Visual text editor |
ed |
Line text editor |
