Modern C++ Testing

Various C++ testing framework exist. Phil Nash compares CATCH with the competition.

As many readers may know, Catch is a test framework that I originally wrote (and still largely maintain) for C++ [ Catch ]. It’s been growing steadily in popularity and has found its way to the dubious spotlight of HackerNews on more than one occasion.

The most recent of such events was late last August. I was holidaying with my family at the time and didn’t get to follow the thread as it was unfolding so had to catch up with the comments when I got back. One of them [ HackerNews ] stuck out because it called me out on describing Catch as a ‘Modern C++’ framework (the commenter recommended another framework, Bandit, as being ‘more modern’).

I first released Catch back in 2010. At that time C++11 was still referred to as C++1x (or even C++0x!) and the final release date was uncertain. So Catch was written to target C++03. When I described it as being ‘modern’ it was in that context and I was emphasising that it was a break from the past. Most other C++ frameworks were just reimplementations of JUnit in C++ and did not really embrace the language as it was then. The use of expression templates for decomposing the expressions under test was also a factor.

Of course since then C++11 has not only been standardised but is fully, or nearly fully, implemented by many leading, mainstream, compilers. I think adoption is still not high enough, at this point, that I’d be willing to drop support for C++03 in Catch (there is even an actively maintained fork for VC6! [ Moene12 ]. But it is enough that the baseline for what constitutes ‘modern C++’ has definitely moved on. And now C++14 is here too [ Sutter14 ] – pushing it even further forward.

‘Modern’ is not what it used to be

What does it mean to be a ‘Modern C++ Test Framework’ these days anyway? Well the most obvious thing for the user is probably the use of lambdas. Along with a few other features, lambdas allow for a lot of what previously required macros to be done in pure C++. I’m usually the first to hold this up as A Good Thing. In a moment I’ll get to why I don’t think it’s necessarily as good a step as you might think.

But before I get to that; one other thing: For me, as a framework author, the biggest difference C++11/14 would make to something like Catch would be in the internals. Large chunks of code could be removed, reduced or at least cleaned up. The ‘no dependencies’ policy means that Catch has complete implementations of things like shared pointers, optional types and function objects – as well as many things that must be done the long way round (such as iterating collections – I long for range for loops – or at least BOOST_FOREACH ).

The competition

I’ve come across three frameworks that I’d say qualify as truly trying to be ‘modern C++ test frameworks’. I’m sure there are others – and I’ve not really even used these ones extensively – but these are the ones I’ll reference in this discussion. The three frameworks are:

Lest – by Martin Moene, an active contributor to Catch – and partly based on some Catch ideas – re-imagined for a C++11 world [ Moene ].
Bandit – this is the one mentioned in the Hacker News comment I kicked off with [ Karlsson ].
Mettle – Seeing this mentioned in a tweet from @MeetingCpp is what kicked off the train of thought that led me to this article [ Porter ].

The case for test case macros

But why did I say that the use of lambdas is not such a good idea? Actually I didn’t quite say that. I think lambdas are a very good idea – and in many ways they would certainly clean up at least the mechanics of defining and registering test cases and sections.

Before lambdas C++ had only one place you could write a block of imperative code: in a function (or method). That means that, in Catch, test cases are really just functions – which must have a function signature –including a name (which we hide – because in Catch the test name is a string). Those functions must be captured somehow. This is done by passing a pointer to the function to the constructor of a small class – whose sole purposes is to forward the function pointer onto a global registry. Later, when the tests are being run, the registry is iterated and the function pointers invoked.

So a test case like this:

  TEST_CASE( "test name", "[tags]" )
  {
      /* ... */
  }

...written out in full (after macro expansion) looks something Listing 1.

static void generatedFunctionName();
namespace{ 
  ::Catch::AutoReg generatedNameAutoRegistrar
    (   &generatedFunctionName, 
        ::Catch::SourceLineInfo( __FILE__,
           static_cast<std::size_t>( __LINE__ ) ), 
        ::Catch::NameAndDesc( "test name",
          "[tags]") ); 
  }
  static void generatedFunctionName()
  {
    /* .... */
  }

Listing 1

( generatedFunctionName is generated by yet another macro, which combines root with the current line number. Because the function is declared static the identifier is only visible in the current translation unit (cpp file), so this should be unique enough)

So there’s a lot of boilerplate here – you wouldn’t want to write this all by hand every time you start a new test case!

With lambdas, though, blocks of code are now first class entities, and you can introduce them anonymously. So you could write them like:

  Catch11::TestCase( "test name", "[tags]", []()
  {
    /* ... */
  } );

This is clearly far better than the expanded macro. But it’s still noisier than the version that uses the macro. Most of the C++11/14 test frameworks I’ve looked at tend to group tests together at a higher level. The individual tests are more like Catch’s sections – but the pattern is still the same – you get noise from the lambda syntax in the form of the []() or [&]() to introduce the lambda and an extra ); at the end.

Is that really worth worrying about?

Personally I find it’s enough extra noise that I think I’d prefer to continue to use a macro – even if it used lambdas under the hood. But it’s also small enough that I can certainly see the case for going macro free here. Since the first version of this article was published on my blog the author of Lest commented that he now uses a macro for test case registration too. He also reported that, at least with present compilers, the lambda-based version has a significant compile-time overhead

Assert yourself

But that’s just test cases (and sections). Assertions have traditionally been written using macros too. In this case the main reasons are twofold:

It allows the expression evaluation to be wrapped in an exception handler.
It allows us the capture the file and line number to report on.

(1) can arguably be handled in whatever is holding the current lambda (e.g. it or describe in Bandit, suite, subsuite or expect in Mettle). If these blocks are small enough we should get sufficient locality of exception handling – but it’s not as tight as the per-expression handling with the macro approach.

(2) simply cannot be done without involving the preprocessor in some way (whether it’s to pass __FILE__ and __LINE__ manually, or to encapsulate that with a macro). How much does that matter? Again it’s a matter of taste but you get several benefits from having that information. Whether you use it to manually locate the failing assertion or if you’re running the reporter in an IDE window that automatically allows you to double-click the failure message to take you to the line – it’s really useful to be able to go straight to it. Do you want to give that up in order to go macro free? Perhaps. Perhaps not.

Interestingly Lest still uses a macro for assertions and (again, as the author commented on my blog) Mettle now uses a macro for expect() in order to capture that information.

Weighing up

So we’ve seen that a truly modern C++ test framework, using lambdas in particular, can allow you to write tests without the use of macros – but at a cost!

So the other side of the equation must be: what benefit do you get from eschewing the macros?

Personally I’ve always striven to minimise or eliminate the use of macros in C++. In the early days that was mostly about using const , inline and templates. Now lambdas allow us to address some of the remaining cases and I’m all for that.

But I also tend to associate a much higher ‘cost’ to macro usage when it generates imperative code. This is code that you’re likely to find yourself needing to step through in a debugger at runtime – and macros really obfuscate this process. When I use macros it tends to be in declarative code. Code that generates purely declarative statements, or effectively declarative statements (such as the test case function registration code). It tends to always generate the exact same machinery – so should not be sensitive to its inputs in ways that will require debugging.

How do Catch’s macros play out in that regard? Well the test case registration macros get a pass. Sections are a grey area – they are on the path of code that needs to be stepped over – and, worse, hide a conditional (a section is really just an if statement on a global variable!). So score a few points down there. Assertions are also very much runtime executable – and are frequently on the debugging path! In fact stepping into expressions being asserted on in Catch tests can be quite a pain as you end up stepping into some of the ‘hidden’ calls before you get to the expression you supplied (in Visual Studio, at least, this can be mitigated by excluding the Catch namespace using the StepOver registry key [ Pennell04 ]).

Now, interestingly, the use of macros for the assertions was never really about C++03 vs C++11. It was about capturing extra information (file/ line) and wrapping in a try-catch. So if you’re willing to make that trade-off there’s no reason you can’t have non-macro assertions even in C++03! That said, the future may hold a non-macro solution even for that, in the form of proposal N4129 for a source_context type to be provided by the language [ N4129 ].

Back to the future

One of my longer arcs of development on Catch (that I edge towards on each refactoring) is to decouple the assertion mechanism from the guts of the test runner. You should be able to provide your own assertions that work with Catch. Many other test frameworks work this way and it allows them to be much more flexible. In particular it will allow me to decouple the matcher framework (and maybe allow third-party matchers to work with Catch).

Of course this would also allow macro-less assertions to be used (as it happens the assertions in bandit and mettle are both matcher-like already).

So, while I think Catch is committed to supporting C++03 for some time yet, that doesn’t mean there is no scope for modernising it and keeping it relevant. And, modern or not, I still believe it is the simplest C++ test framework to get up and running with, and the least noisy to work with.