Exercise: Custom Asserts

For these exercises, we'll use Github Codespaces. You will need a github account. You will need to run this on your own account, which has ?60 hours a month free codespaces.

In this exercise. we'll use LLMs to help us build custom asserts. You'll use your own access to an LLM, and we'll work in environments with a JavaScript test harness, a Python test harness, and code which could do with a custom assert or two.

We'll look at the tiny systems, identify what would help, ask LLMs to build working asserts, see what they give back, and compare notes.

Here's more on custom asserts.

Environment

I've built something that has all the code, test harnesses and plumbing for you to try this in JavaScript and Python. You'll access your own copy of that environment via VisualStudio Code in your browser.

Exercise 0 – your own environment

Go to the Custom Asserts repo , and you should see a readme page – click on the "open in Github Codespaces" button, wait and respond reasonably to Github's questions.

Further instructions are in the readme – once you're in, you'll need to run something on the commandline to install all the infrastructure dependencies, and more to run tests. You can browse the software under test while the dependencies are being installed.

The environment has two exercises, each in both Javascript and in Python. I'm still building the examples, so if you set up early, you won't have them.

Infrastructure

The tests for my JavaScript examples here use Chai (assertion library) and Mocha (unit test framework). My Python examples use bare Python and pyTest (unit test framework). 

You'll need to go to your own LLM to ask for code – you'll find examples below.

Pick a codebase, get the tests running

Different tech has different tests. I've tried to make this exercise work in Python and in Javascript / Node. Both are readable: you might want to work in your less-familiar technology. Or you might want to compare.

You'll find Python in ./python_project and JavaScript / Node in ./javascript_project.

I've tried to give the files and functions similar names.

Exercise 1 – get the tests running

Get the tests running.

Have a look at the calculator code and tests to see how they work.

Exercise 2 – checking XML tags

We'll build a custom assert to check XML tags.

• MT103_gen.js and MT103_gen.py have a function generateMT103 which returns a simple XML example record.
• ./test/test_MT103_gen.js and ./tests/test_MT103_gen.py both test that function.

We're going to use this as a dummy to build a custom assert. We want the custom assert to check that all the XML tags are present, and that there are no extras.

For the sake of plumbing build the custom assert in a file called customAsserts.js/.py, and call the method containXmlTags.

You'll need to register it before using it. Using the names above, you could

// Register the custom assertion
const containXmlTags = require('./customAsserts');
chai.use(containXmlTags);

from customAsserts import containXmlTags

And, of course, you'll need to add the test...

Once you've added it, you can test it by changing the generation code MT103_gen.

We'll compare and contrast our experiences.

Examples of generated test code

My approach – for greatest transparency, I'm copying the code and chucking it straight into Claude 4.

Given the following code, and the following tests, make me a shareable custom assert called containXMLtags in a file customAsserts.js. The custom assert should check that all tags for this example MT103 are present, and none are extra.
[I pasted the code in here]

which returned:

Sure, here's the customAsserts.js file with the containXMLtags custom assert:

const { DOMParser } = require('@xmldom/xmldom');

const containXMLtags = (actual, expected) => {
  const parser = new DOMParser();
  const doc = parser.parseFromString(actual, 'text/xml');

  // Check for parse errors
  const parseError = doc.getElementsByTagName('parsererror');
  if (parseError.length > 0) {
    return {
      message: () => 'The provided XML string is not valid.',
      pass: false,
    };
  }

  // Check root element exists
  if (!doc.documentElement) {
    return {
      message: () => 'The provided XML string does not have a root element.',
      pass: false,
    };
  }

  // Check for expected tags
  const expectedTags = [
    'MessageType',
    'SenderBIC',
    'ReceiverBIC',
    'TransactionReference',
    'ValueDate',
    'Amount',
    'Currency',
    'BeneficiaryName',
    'BeneficiaryAccount',
    'BeneficiaryAddress',
    'RemittanceInformation',
  ];

  for (const tag of expectedTags) {
    if (!doc.getElementsByTagName(tag).length) {
      return {
        message: () => `The expected tag '${tag}' is not present in the XML.`,
        pass: false,
      };
    }
  }

  // Check for extra tags
  const extraTags = Array.from(doc.getElementsByTagName('*'))
    .map((node) => node.nodeName)
    .filter((tag) => !expectedTags.includes(tag));

  if (extraTags.length > 0) {
    return {
      message: () => `The following unexpected tags were found in the XML: ${extraTags.join(', ')}`,
      pass: false,
    };
  }

  return {
    message: () => 'The provided XML string contains all expected tags and no extra tags.',
    pass: true,
  };
};

module.exports = { containXMLtags };

To use this custom assert in your tests, you can import it from the customAsserts.js file:

const { expect } = require('chai');
const { containXMLtags } = require('./customAsserts');

describe('MT103_gen', () => {
  let generateMT103;

  before(() => {
    try {
      ({ generateMT103 } = require('../MT103_gen'));
    } catch (err) {
      generateMT103 = null;
    }
  });

  it('should generate valid XML', () => {
    const result = generateMT103();
    expect(result).to.satisfy(containXMLtags);
  });

  it('XML should be labelled a SwiftPayment record', () => {
    const result = generateMT103();
    expect(result).to.satisfy(containXMLtags);
  });
});

This does both more (more checks) and less (it's not generic) than we'd asked for. I might re-prompt, asking for the test to contain the list of tags.

Note: these are examples – you'll get different stuff, and it might be better. I've not yet used this example.

Exercise 2 – checking a distribution

We'll build a custom assert to check that a certain type of record is in the 'right' proportion to other records.

• candidate_gen generates candidates
• test_candidate_gen does very little

We're going to use this as a dummy to build a custom assert. We want the custom assert to check that 8-12% of the candidates are of type high, 0.5 to 1.5 % are mighty and the rest (and no more than 91%) are base.

Again, you'll need to register your custom assert, and use it in a test.

For the sake of plumbing build the custom assert in a file called customAsserts.js/.py, and call the method checkDistribution.

Adjust the candidate generation to test it – note that the dummy should fail. It has too few records to have a mighty. So you'll need to make different distributions to properly test the assert. Tested test code? Whatever next!?

Examples of generated test code

I want to build a custom assert to check candidate proportions. The assert should take parameters. In this case, 8-12% of the candidates should be of type high, 0.5 to 1.5 % are mighty and the rest (and no more than 91%) are base. Example candidate generation and existing test code follows. The custom assert should be called `checkDistribution` and will be used in a file called customAsserts.py

we'll build examples in the workshop.