Peter Sommerlad presents a lightweight framework for C++ unit testing.
This article describes my attempt to leverage more modern C++ libraries and features to make the writing of unit tests easier in C++. For example, one disadvantage of CPPUnit is that you have to write a subclass to have your own test case. This is a lot of programmer overhead, especially when you want to start small.
I was inspired by Kevlin Henney's Java testing framework called JUTLAND (Java Unit Testing: Light, Adaptable `n' Discreet), and the corresponding presentation he gave at JAOO 2005. In addition, I wondered if I could come up with a design that is similarly orthogonal, easily extendable and also much simpler to use than current C++ unit testing approaches.
You will learn how to use CUTE in your projects and also some of the more modern C++ coding techniques employed for its implementation. I also ask you to give me feedback to further simplify and improve CUTE. Even during the writing of this article I recognized simplification potential and refactored CUTE accordingly. Porting and testing onto other compilers not available to me is also an appreciated contribution.
My problems with CPPUnit
Inheritance is a very strong coupling between classes. Requiring a test-case class to inherit from a CPPUnit-framework class couples both closely together. The CPPUnit tutorial [ CppUnitCookbook ] lists at least six classes you have to deal with to get things up and running. You even have to decide if you want to inherit from TestCase or TestFixture for your simple test case you intend to write. I do not want to go into more details, but I show how I would like to write tests.
#include "cute.h"
int lifeTheUniverseAndEverything = 42;
void mysimpletest(){
t_assert(lifeTheUniverseAndEverything == 6*7);
}
That's it. A simple test is a simple void function. Done (almost).
In addition CPPUnit stems from a time of non-standard C++ compilers, where more modern features of the language just have not been available broadly. This limits its design from a modern perspective.
Today, all relevant compilers (with the exception of the still in use MSVC6) are able to compile most of standard C++ and the proposed std::tr1 libraries from Boost.
Running a single test
Since we lack some of the reflection mechanisms available in Java, you have to write your own main function for testing with CUTE. For simple cases this is straightforward. As with CPPUnit you have to instantiate a runner object and pass your test for running it. The simplest possible way, producing some output is shown in Listing 1.
#include <iostream>
#include "cute_runner.h"
int main(){
using namespace std;
if (cute::runner<>()(mysimpletest)){
cout << "OK" << endl;
} else {
cout << "failed" << endl;
}
}
|
| Listing 1 |
OK, not very impressing yet, but simple. Reporting test outcome is so common, that CUTE provides a means to configure the runner with a so-called `listener'. You already might have wondered what these template-brackets with cute::runner were about. So we can change that to the following:
int main(){
using namespace std;
cute::runner<cute::ostream_listener>()(
mysimpletest);
}
"void () () OK"
is what we get from that one. We know that a void function without any arguments succeeded to run. This shows C++ introspection limitation, which only provides type information, not function names. But the preprocessor can help us. So making our test cuter by applying the macro CUTE() , helps us:
cute::runner<cute::ostream_listener>()( CUTE(mysimpletest));
This achieves the output "mysimpletest OK" . Not too interesting. However, if we make our test case fail by setting deep thought's answer to 41 instead of 42, we get:
../simpletest.cpp:6: testcase failed: lifeTheUniverseAndEverything == 6*7 in mysimpletest
That is everything we need to track down the failure's origin and even some context helping with a first guess. You already guessed that the preprocessor macro t_assert() from cute.h contains the magic for collecting this interesting information.
Note that all classes presented in the following are in namespace cute , that I omit from the definitions shown for brevity.
How things work
I tried to create a simple orthogonal and thus easy to extend and adapt testing framework that stays easy to use. Exploiting C++ library features that are modern (boost) and will be part of the future standard ( std::tr1 ) I could avoid some complexity for CUTE's users.
CUTE test
The core class representing tests uses boost::function , which will be std::tr1::function , when compilers support this designated extension, to store test functions. So any parameterless function or functor can be a test. In addition each cute::test has a name, so that it can be easier identified. That name is either given during construction or derived from a functor's typeid. The GNU g++ compiler requires to demangle that name given by the type_info object, where VC++ already provides a human readable type_info::name() result.
struct test{
template <typename VoidFunctor>
test(VoidFunctor const &t, std::string name =
demangle(typeid(VoidFunctor).name()))
:theTest(t),name_(name){}
void operator()()const{ theTest(); }
std::string name()const{ return name_;}
static std::string demangle(char const *name);
private:
boost::function<void()> theTest;
std::string name_;
};
As you can see, there is no need to inherit from class test . The only thing I do not yet like, is the static function demangle , that needs to be implemented per compiler. I haven't found a better place for it in the framework yet.
For simple functions, or if you want to name your tests differently from the functor's type, you can use the following CUTE() macro.
#define CUTE(name) cute::test((name),#name)
The using of a template constructor allows you to use any kind of functor, that can be stored in a boost::function<void()> , that means a functor that doesn't take parameters. With boost::bind() you are able to construct those, even from functions, functors or member functions with parameters as shown below.
Sweet suites
Running a single test with cute::runner is not very interesting - you might call that function directly and check results. But having a larger collection of test cases and running them after every compile and on a build server after every check in, is what makes unit testing so powerful. So there is a need for running many tests at once.
But in contrast to other unit testing frameworks (including JUnit) I refrained from applying the Composite Design Pattern ! [ GoF ] for implementing the container for these many test cases your project requires. I love Composite and it is handy in many situations for tree structures, but it comes at a price of strong coupling by inheritance and lower cohesion in the base class, because of the need to support the composite class interface. The simplest solution I came up with is just using a std::vector<cute::test> as my representation for test suites. Instead of a hierarchy of suites, you just run a sequence of tests. When the tests run, the hierarchy doesn't play a role. You still can arrange your many tests in separate suites, but before you run them, you either concatenate the vectors or you run the suites in your main function separately through the runner.
For those of you who really want your test suites to be tests, CUTE provides a suite_test functor that will take a suite and run it through its call operator. But if any test of the suite in such a suite_test fails, the remaining tests won't be run.
To make it easy to fill your suite with your tests CUTE provides an overloaded operator+= that will append a test object to a suite. This idea is blatantly stolen from boost::assign , which I didn't use, because to my knowledge it didn't make it to std::tr1 (yet?).
So this is all it takes to have test suites:
typedef std::vector<test> suite; suite &operator+=(suite &left, suite const &right); suite &operator+=(suite &left, test const &right);
Assertions and failures
A unit testing framework wouldn't be complete without a means to actually check something in a convenient way. One principle of testing is to fail fast, so any failed test assertion will abort the current test and signal that failure to the top-level runner. You might already have guessed that throwing an exception is the corresponding mechanism. Since we want to know later on, where that test failed, I introduced an exception class cute_exception that takes the filename and line number of the source position. Java can do that automatically for exceptions, but as C++ programmers we have to carry that information ourselves and we have to rely on the preprocessor to actually know where we are in the code. Another std::string allows sending additional information from the test programmer to the debugger of a failing test.
This is how cute.h looks without the necessary #include guards and #include of <string> :
namespace cute{
struct cute_exception {
std::string reason;
std::string filename;
int lineno;
cute_exception(std::string const &r,
char const *f, int line)
:reason(r),filename(f),lineno(line)
{ }
std::string what() const ;
};
}
For actually writing test assertions I provided macros that will throw, it a test fails. I deliberately used lower case spelling for these macros to make them easier to use.
#define t_assertm(msg,cond) if (!(cond)) \
throw cute::cute_exception((msg),__FILE__,__LINE__)
#define t_assert(cond) t_assertm(#cond,cond)
#define t_fail() t_assertm("fail()",false)
#define t_failm(msg) t_assertm(msg,false)
This is all for you to get started. However, some convenience is popular in testing frameworks. But convenience often tends to be over-engineered and I am not yet sure if the convenience functionality I provided is yet simple enough. Therefore I ask for your feedback on how to make things simpler or encouragement that it is already simple enough.
Equality - overengineered?
Testing two values for equality is may be one of the most popular tests. Therefore, all testing frameworks provide a means to test for equality. JUnit, for example, provides a complete amount of overloaded equality tests. C++ templates can do that as well with less code. For more complex data types, such as strings, it might be hard to see the difference between two values given, when they are simply printed in the error message.
void anothertest(){
assertEquals(42,lifeTheUniverseAndEverything);
}
One means to implement assertEquals would be to just #define it to map to t_assert((expected)==(actual)) . However, from my personal experience of C++ unit testing since 1998, this is too simplistic in cases where the comparison fails. Especially for strings or domain objects seeing the difference between two values is often important for correcting a programming mistake. In my former life, we had custom error messages for a failed string compare to spot the difference easily. Therefore, CUTE provides a template implementation of assert_equal that again is called by a macro, to enable file position gathering.
I speculated (maybe wrongly) it would be useful to be able to specify your own mechanism to create the message if two values differ, which also is implemented as a to-be-overloaded template function. (See Listing 2.)
template <typename EXPECTED, typename ACTUAL>
std::string diff_values(EXPECTED const &expected
,ACTUAL const & actual){
// construct a simple message...
std::ostringstream os;
os << "(" << expected<<","<<actual<<")";
return os.str();
}
// special overloaded cases for strings
std::string diff_values(
std::string const &,std::string const &);
std::string diff_values(
char const * const &exp,std::string const &act);
template <typename EXPECTED, typename ACTUAL>
void assert_equal(EXPECTED const &expected ,
ACTUAL const &actual
,char const *msg,
char const *file,int line) {
if (expected == actual) return;
throw cute_exception(
msg + diff_values(expected,actual),file,line);
}
#define assertEqualsm(msg,expected,actual) \
cute::assert_equal(\
(expected),(actual),msg,__FILE__,__LINE__)
#define assertEquals(expected,actual) \
assertEqualsm(\
#expected " expected but was " #actual,\
expected,actual)
|
| Listing 2 |
I encourage readers to criticize this design to help me to come up with something simpler.
Listening customization
You've already seen, that the runner class template can be specialized by providing a listener. The runner class is an inverted application of the Template Method Design Pattern ![ GoF ] . Instead of implementing the methods called dynamically in a subclass, you provide a template parameter that acts as a base class to the class runner , which holds the Template Methods runit() and operator() . (See Listing 3).
template <typename Listener=null_listener>
struct runner : Listener{
runner():Listener(){}
runner(Listener &s):Listener(s){}
bool operator()(test const &t){
return runit(t);
}
bool operator()(suite const &s,
char const *info=""){
Listener::begin(s,info);
bool result=true;
for(suite::const_iterator it=s.begin();
it != s.end();++it){
result = this->runit(*it) && result;
}
Listener::end(s,info);
return result;
}
private:
bool runit(test const &t){
try {
Listener::start(t);
t();
Listener::success(t,"OK");
return true;
} catch (cute_exception const &e){
Listener::failure(t,e);
} catch(...) {
Listener::error(
t,"unknown exception thrown");
}
return false;
}
};
|
| Listing 3 |
If you look back to runner::runit , you will recognize that if any reasonable exception is thrown, it would be hard to diagnose, what the reason for an error is. Therefore, I included catch clauses for std::exception , string and char pointers to get information required for diagnosis. The demangling is required for GNU g++ to get a human-readable information from the exception's class name.
} catch (std::exception const &exc){
Listener::error(t,test::demangle(
exc.what()).c_str());
} catch (std::string &s){
Listener::error(t,s.c_str());
} catch (char const *&cs) {
Listener::error(t,cs);
Again I ask you for feedback if doing so seems over-engineered. Are you throwing strings as error indicators?
As you can see, there are a bunch of methods delegated to the base class given as runner 's template parameter ( begin , end , start , success , failure , error ). The default template parameter null_listener applies the Null Object Design Pattern and provides the concept all fitting Listener base classes. (Listing 3).
struct null_listener{
// defines Contract of runner parameter
void begin(suite const &s, char const *info){}
void end(suite const &s, char const *info){}
void start(test const &t){}
void success(test const &t,char const *msg){}
void failure(test const &t,
cute_exception const &e){}
void error(test const &t,char const *what){}
};
So whenever you need to collect the test results or you want to have a nice GUI showing progress with the tests, you can create your own specific listener.
Again you can employ an inverted version of a GoF Design Pattern, to stack listeners. This is application of an inverted Decorator using C++ templates, for example to count the number of tests regarding their category, see Listing 4.From the schema in Listing 4, you can derive your own stackable listener classes, e.g. one showing the progress of running the tests and their results in a GUI. If you do so, share your solution.
template <typename Listener=null_listener>
struct counting_listener:Listener{
counting_listener()
:Listener()
,numberOfTests(0),successfulTests(0)
,failedTests(0),errors(0),numberOfSuites(0){}
counting_listener(Listener const &s)
:Listener(s)
,numberOfTests(0),successfulTests(0)
,failedTests(0),errors(0),numberOfSuites(0){}
void begin(suite const &s, char const *info){
++numberOfSuites;
Listener::begin(s,info);
}
void start(test const &t){
++numberOfTests;
Listener::start(t);
}
void success(test const &t,char const *msg){
++successfulTests;
Listener::success(t,msg);
}
void failure(test const &t,
cute_exception const &e){
++failedTests;
Listener::failure(t,e);
}
void error(test const &t,char const *what){
++errors;
Listener::error(t,what);
}
int numberOfTests;
int successfulTests;
int failedTests;
int errors;
int numberOfSuites;
};
|
| Listing 4 |
Test extensions
With the idea of allowing all non-parameter Functors to be eligible as tests, it is relatively simple to provide test-wrappers for different kind of functionality.
Exception testing
Good practice of unit testing is also to check if things go wrong as intended. So you want to expect a specific exception from a test functor. The code to do that can easily be canned for reuse in a template and with CUTE it is activated by a macro call like:
suite s; s += CUTE_EXPECT( functor_that_throws(),std::exception);
The implementation of that mechanism is as you have expected (see Listing 5).
template <typename EXCEPTION>
struct cute_expect{
test theTest;
std::string filename;
int lineno;
cute_expect(test const &t,char const *file,
int line)
:theTest(t), filename(file), lineno(line){}
void operator()(){
try{
theTest();
throw cute_exception(
what(),filename.c_str(),lineno);
} catch(EXCEPTION &e) {
}
}
std::string what() const{
return theTest.name() + " expecting "
+
test::demangle(typeid(EXCEPTION).name());
}
};
|
| Listing 5 |
No need to implement the try-catch again by hand for testing error conditions. What is missing, is ability to expect a runtime error recognized by the operating system such as an invalid memory access. Those are usually signalled instead of thrown as a nice C++ exception.
With a similar wrapper you can implement a class for repeatedly running a test. There isn't even a template required.
Member functions as tests
Having boost::bind() at your disposal, it is easy to construct a functor object from a class and its member function. Again this is canned in a macro that can be used like:
CUTE_MEMFUN(testobject,TestClass,test1); CUTE_SMEMFUN(TestClass,test2); CUTE_CONTEXT_MEMFUN(contextobject,TestClass,test3);
The first version uses object testobject , an instance of TestClass , as the target for the member function test1 . The second version creates a new instance of TestClass to then call its member function test2 when the test is executed. The last macro provides a means to pass an additional object, to TestClass ' constructor, when it is incarnated. The idea of incarnating the test object and thus have its constructor and destructor run as part of the test comes from Kevlin Henney and is implemented in Paul Grenyer's testing framework Aeryn.
The macro CUTE_MEMFUN delegates its work to a template function as shown in Listing 6.
template <typename TestClass>
test makeMemberFunctionTest(TestClass &t,
void (TestClass::*fun)(),char const *name){
return test(boost::bind(fun,boost::ref(t)),
test::demangle(typeid(TestClass).name())+
"::"+name);
}
#define CUTE_MEMFUN(testobject,TestClass,\
MemberFunctionName)\
cute::makeMemberFunctionTest(testobject,\
&TestClass::MemberFunctionName,\
#MemberFunctionName)
|
| Listing 6 |
The template function makeMemberFunctionTest employs boost::bind to create a functor object that will call the member function fun on object t , when called. Again we can employ C++ reflection using typeid to derive part of the test object's name. We need to derive the member function name again using the preprocessor with a macro. To allow to use also const member functions, the template function comes in two incarnations, one using a reference as shown and the other one using a const reference for the testing object.
Test object incarnation
I will spare you all details, but give you the mechanism of object incarnation and then calling a member function for the case, where you can supply a context object. (See Listing 7).
#define CUTE_EXPECT(tt,exc) \
cute::test(cute::cute_expect<exc>(\
tt,__FILE__,__LINE__),tt.name())
template <typename TestClass,typename MemFun,
typename Context>
struct
incarnate_for_member_function_with_context_object
{
MemFun memfun;
Context context;
incarnate_for_member_function_with_context_object(
MemFun f,Context c)
:memfun(f),context(c){}
void operator()(){
TestClass t(context);
(t.*memfun)();
}
};
template <typename TestClass, typename MemFun,
typename Context>
test makeMemberFunctionTestWithContext(
Context c,MemFun fun,char const *name){
return test(
incarnate_for_member_function_with_context_object
<TestClass,MemFun,Context>(fun,c),
test::demangle(typeid(TestClass).name())
+"::"+name);
}
|
| Listing 7 |
This will allow you to use test classes with a constructor setting up a test fixture and a destructor clearing it again. So there is no longer a need for writing Java like setUp() and tearDown() methods.
Limitations and outlook
One big difference between C++ and other languages is the lack of method-level introspection. The only means for getting a list of tests to execute is having a programmer specifying it, i.e., by registering test objects somewhere. If anybody is aware of how to get rid of that and have tests registered automatically please let me know. CppUnit compensates this by the ability to automatically load shared libraries with test classes. On the other hand, this makes writing tests, using CppUnit and its implementation more complex.
CUTE is still in a nascent state at the time of this writing. It comes with a small test suite for itself, but especially with all the templates it might still suffer from problems in its use, not yet encountered by me. If you haven't yet written unit tests for your code, try starting now using CUTE and tell me how it feels and works. You can download the currently released version of CUTE in source form from my wiki web at http://wiki.hsr.ch/PeterSommerlad/ .
There are many ideas for extending CUTE to make it a more convenient environment to live in. For example, better IDE integration to directly navigate from a failed test is a must for professional use. Tell me your ideas, or just implement them. Thank you in advance.
References
[ Aeryn] Paul Grenyer http://www.aeryn.co.uk
[ CppUnit] http://cppunit.sourceforge.net/cppunit-wiki
[ CppUnitCookbook] http://cppunit.sourceforge.net/doc/lastest/cppunit_cookbook.html
[ GoF] Gang of Four, E. Gamma, R. Helm, R. Johnson, J. Vlissided: Design Patterns - Elements of Reusable Object-Oriented Design
[ JUTLAND] Kevlin Henney, Java Unit Testing Light Adaptable `N' Discreet, presentation at JAOO 2005 and private communication
