Developing a Unit Test Framework (Part 1)

I always knew this was bound to happen, I have dreaded its inevitability but alas it finally happened…and not for the first time. All programmers have been down this road, you perusing that aging lump of cobwebs you call code. You decide to do a little dust busting and before you know it, everything is broken and you do not have the slightest clue what just happened!

I guess this was the proverbial straw that broke the camel’s back, because the sheer exasperation I felt in that moment shredded through my tolerance barrier and I could feel myself sinking into a code rage. It was time to find a better way of coding, a better way of sniping down those pesky bugs. I had been procrastinating getting into unit testing, but now was as good a time as ever.

Unit Testing

Unit tests are arguably one of the best ways to future proof your code against bugs. In Test Driven Design (TDD) [1], units of code to test the behaviour of functions and modules are written before implemenation [2]. These units of code known as test cases [3], are used to continuously test your code as you implement functionality.

You repeat short cycles of incrementally developing tests and code until your code passes all the tests. A good unit testing framework will help you to not just find bugs quickly, but also locate were they are occurring.

Whenever you are tempted to type something into a print statement or a debugger expression, write it as a test instead.

Martin Fowler [4]

Whenever you encounter a bug, write a test for it. That should be the last time a human has to manually check for that bug.


Finding a Common Lisp Unit Test Framework

Finding a good unit testing framework for my Common Lisp libraries turned out to be a bit harder than I expected due to the bewildering number of options. Thankfully, Phil Gold evaluated most of the unit testing frameworks and wrote this article [5].

Having read the article, I came to the same conclusion as the author, each framework lacked something that was found in another framework.  What was needed was the sum total of the unit testing frameworks.

As fate would have it, the very same evening I stumbled onto a blog [6] by François-René on starting a movement to consolidate Common Lisp libraries. Inspired by the tremendous work done by Zach Beane on Quicklisp [7], I decided to take up the challenge and write a consolidated unit testing framework that would hopefully become the standard in the Common Lisp community.

What makes a good framework?

The EmacsLisp Unit Testing wiki page [8] gives a nice concise description of the core features of a unit testing framework. Based on what I have read elsewhere, I have listed and described the key features of what I believe constitutes a good unit testing framework.

Ease of use

By far one of the most important requirements in my books is ease of use. The complexity of most unit testing frameworks is a barrier to entry and prevents most people from getting into the habit of writing unit tests.

The more time and effort you need to spend setting up a framework instead of actually using it, the less likely you are to use it frequently.


Assertions are statements that verify some condition. The code being tested by an assertion either passes or fails the assertion.

Test Cases

A test case is a collection of assertions grouped together to test some core functionality. The general practice is to implement a test case as a function, though this is not necessarily an ironclad rule.

Test Suites

A test suite allows you to group your tests into collections, different frameworks offer varying levels of grouping. What is the point of this you might ask? If you have only about 10 test cases, then invoking them directly would not be too burdensome.

No Hierarchy

Now consider the number of tests that are required for the linux kernel, the sum total for all the tests could be anything from the 10′s of thousands into the 100′s of thousands. Invoking each one directly is clearly infeasible, some way of invoking them indirectly in terms of suites would be needed.

Flat Hierarchy

Most frameworks generally provide a flat heirarchy. Each test suite contains a group of tests, but does not contain other test suites. In these frameworks you still have to invoke each test suite directly. You might have the option to call another test suite from within a different test suite but thats not a real tree hierarchy.

Multiple Inheritance

My ideal unit testing framework would be one that provides not just a tree heirarchy, but more of a directed acyclic graph i.e. multiple inheritance. Test cases would be independent instances, that can be a member of zero or more test suites. Using this approach, one could conceivably define a root suite which all other suites inherit from, by invoking the root suite you can execute all your tests. It also gives you the option to either just execute one test case or a sub-collection of test suites.

Composable Results

A unit testing framework should be able to return an aggregated result report instead of spitting out a report for each test case or suite. It should always return one aggregated report regardless of whether a single test case, a single test suite or multiple test suites were executed.

Configurable Reporting

A useful feature would be the ability to configure the reporting or at least provide a list of different reporting formats to choose from e.g. Test Anything Protocol (TAP) output [9].

Composable Fixtures

A unit testing framework should make it easy to use test fixtures [10]. A fixture is used to setup a consistent context/enviroment in which tests can be executed so that results are reproducible. The fixture is also responsible for cleaning up that context once the test is done. In most frameworks, you are able to define a fixture for a test suite and the fixture will then be applied to each test in the suite.

Since, my ideal unit testing framework should support multiple inheritance. The framework should be able to compose the fixtures of multiple test suites into a single fixture for a test that belongs to more than one suite.

Test Environment Access

In Common Lisp I use the debugger often to inspect the call stack and calling arguments when something goes wrong. In languages that provide access to the debugger, a unit test framework should allow you to access the debugger in the test context if an assertion fails. Using the debugger you can then trace the bug back to its point of origin.

Looking Ahead

Part 2 [11], documents the step by step design process of a Common Lisp unit testing framework.


  1. Test Driven Development
  2. Unit Testing
  3. Unit Tests – Lessons Learnt
  4. JUnit Test Infected: Programmers Love Writing Tests
  5. Notes on Lisp testing frameworks
  6. Consolidating Common Lisp Libraries
  7. Quicklisp
  8. Emacs Unit Testing
  9. Test Anything Protocol
  10. Test Fixture
  11. Developing a Unit Test Framework (Part 2)








  1. 1
    Sabra Crolleton on Monday 12 November, 15:52 PM #

    Glad to see you have taken up the challenge. We certainly do not need 20 unit testing frameworks.

    I would suggest that you also look at the NST framework (which is relatively recent).

    I think there are also some discussions buried in irc or elsewhere on technical issues with the FiveAm/eos implementation that you may want to make sure you do not fall into.



    • 2
      Tapiwa on Tuesday 13 November, 07:41 AM #

      Thanks for the link, just gave it a quick browse. Its a pretty through design document. I will have to save it for a weekend read if I am gonna do it justice.


  2. 3
    Jean-Philippe Paradis on Wednesday 14 November, 00:30 AM #

    Good article. I liked the “What makes a good framework?” part most, and it’s too bad I can’t link to that subsection directly because it doesn’t have an id attribute. :(

    By the way, it’s “Zach Beane” (well, except in copyright statements), not “Xach Beane”.

Leave a comment

Leave a Reply