Executive Summary
Unit testing is the lowest level of testing performed during software development. Conscientious unit testing will detect many problems at a stage of the development where they can be corrected economically. For software developed in C++, the class is the smallest unit which it is practical to unit test.
A well designed class will provide an abstraction, with implementation details hidden within the class. Objects of such classes can be difficult to thoroughly unit test. This paper discusses the problems involved in unit testing C++ classes, presenting strategies which solve these problems. A detailed example of the unit test for a C++ class, using IPL's testing tool Cantata, is included on the disk.
What is a Unit Test?
The unit test is the lowest level of testing performed during software development. Individual units of software are tested in isolation from other parts of the program. Unit testing typically aims to achieve 100% decision coverage of the code within a unit (although other coverage measures can also be used). Throughout this paper the general term "coverage" is used to refer to whatever coverage measure (decision coverage or other) is adopted.
Experience has shown that a conscientious approach to unit testing will detect many problems at a stage of the software development where they can be corrected economically. In later stages of software development, detection and correction of problems is much more difficult, time consuming and costly.
In a conventional structured programming language, such as C, the unit to be tested is traditionally the function or sub-routine. To test such a unit in isolation, external program units called by the unit under test and external data used by the unit under test have to be simulated.
What Makes C++ Different?
The class is the major new feature of the C++ language over C. Its primary purpose is to provide encapsulation within an object. In general a C++ class is more complex then a C function and has much more scope for variety. Consequently, the testing strategy adopted should be more flexible. It should aim to establish guide-lines and not rules.
A class contains both member functions and data, exhibiting a strong degree of internal coupling. Members which are only relevant to the internal implementation of the class are termed private. Such members are encapsulated entirely within the class and are not visible to external program parts. The high degree of coupling exhibited by members of a class makes it impractical to separate member functions from a class in order to test each member function in isolation. Consequently, the class is the basic component to be tested when unit testing object based software written in C++.
The resulting problems fall into two groups. Firstly, the testing of a class must be thorough, but the lack of visibility of private part of a class means that it may be difficult to achieve full coverage of some member functions, particularly private member functions. Section 2 of this paper discusses this problem and presents a solution.
The second problem area stems from the aim to unit test a class in isolation from other program parts. In order to isolate a class for testing, other classes have to be simulated. This problem, and solutions, are discussed in section 3.
Unit testing of a class is not a one-off activity. Unit tests have to be repeated when the class in modified or used in a new environment.
Section 4 discusses unit test maintenance. Once unit tested, classes have to be integrated to provide the overall software system.
Integration of classes is briefly described in section 5.
Section 6 presents conclusions. A detailed example of the unit test for a C++ object, using Cantata, is given on the disk.
Achieving Visibility for Testing
The structure of a class and inter-relationships between members of a class result in a class being a much more complex unit than a single function or sub-routine. Achieving full test coverage of some class members may be difficult. It is especially difficult to achieve full test coverage of the private member functions of a class because they are not directly visible or accessible outside of the class.
One solution is to structurally modify a class for testing by making all of its members externally accessible, but any temporary modification for testing purposes is undesirable. A modification may obscure problems which should have been detected by the unit test and may also introduce new problems. Temporary (or permanent) modification of a class to facilitate testing would effectively alter the entire context of the class, defeating the initial design aim of encapsulation of detail within the class.
Test Methods
In common with other design methodologies, when using object oriented design techniques, a software engineer should account for testing of a class in its design. When using C++, this means that the definition of a class must encompass the means by which the class is to be tested. In practice, the declaration of an externally visible member function within a class definition can be used to facilitate unit testing of the class.
Example 2a shows how a test method can be incorporated into a C++ class.
The test method could be a public member function of the class. However, it is better declared as a friend function of the class (as in example 2a), because the method actually instantiates objects of the class rather than acting upon a particular object. All classes can have the same test method as a friend function, so access to the private members of any class is available from the test method. This can prove useful during higher levels of testing and integration.
Example 2a - Private members and the Test function.
class example_class
{
// Public member functions public:
int add (); // sums x and y
int subtract(); // subtracts y from x
void PrintLargest(); // Prints greatest of x or y
// Test Method
friend void test();
// Private member functions
private:
int largest(); // Returns largest of x or y
// Private member data
int x;
int y;
};
Test Scripts
To test a class, a file called a test script is created, containing code for the test method declared in the class declaration. Test cases within the test method should instantiate objects of the class type, provide data in order to achieve a desired path through the member functions, and check data against expected values at the end of each test case execution.
Test cases should be designed to achieve full coverage of all functions within the class. Initial test cases should use the public member functions, including constructors. The objective is to achieve as much test coverage (both structural and functional) of all members of the class as possible, without recourse to direct access of the private member functions. If necessary, further test cases can then directly access the private member functions to enable test coverage to be completed.
Each test case should be designed to work in isolation from the other test cases. A test case will normally start by instantiating an object of the class type and finish by deleting it. The method to be tested will be called after the class member data has been set up. This may be done directly from the test script or indirectly by calling previously tested methods. All class data should be checked before deleting the object.
This will entail slightly more work, and will require longer test scripts, than the alternative of allowing test cases to rely on data from previous test cases. However, it does offer a number of advantages.
(a) Individual test cases are more robust. Errors in one
test case will not result in secondary errors in subsequent cases.
(b) Test scripts are more maintainable. Test cases can easily be inserted, moved, changed or deleted without affecting surrounding test cases.
(c) Each member function is effectively tested in isolation, but
within the overall environment of the class.
(d) Achieving complete structural coverage is simpler.
Functional testing of a class could be achieved with less work by designing test cases to rely on the results of preceding test cases. However, all of the above advantages would soon be lost.
It may be more practical to achieve functional and structural test objectives by making calls to more than one method in a single test case. However, member data should still be checked following each call.
Heap Leakage
An all too common problem in object oriented C++ systems is that of heap leakage. This phenomenon occurs when memory is allocated for use, but is not released when it is no longer required. Gradually more and more heap space is consumed as a result of this 'leakage' until catastrophic system failure results.
Good design practice should obviously protect against heap leakage, and unit testing can be used to reinforce this. The unit test for a class should ensure that no heap space remains allocated at the end of each test.
Testing a Class in Isolation
To test a unit in isolation, surrounding units have to be simulated by the test. For C++ classes, this means that the test environment will have to simulate entire classes.
An obvious question to ask is "Why unit test a class in isolation, when it will have to be integrated with other units anyway? Why not use the real classes?". A well designed object oriented system should have a well structured class hierarchy. Consequently, it should be possible to plan unit testing to test the most abstract classes first, then progress through the hierarchy, ensuring that each new class tested only requires classes that have already been fully tested. This simple form of unit testing obviates any need to simulate (stub out) classes external to the class under test.
Hierarchical Class Testing
To explore the possibility of hierarchical class testing further, consider the typical class inheritance hierarchy shown in figure 3 a. Suppose these classes were to be tested using hierarchical class testing. A test plan would have to consider the following:
(a) In order to test class b202, classes bl0l and b0 need to have already been tested.
(b) In order to test class a301 we need to take into account the fact that class al02, from which a301 is derived, contains an object of class b201.
(c) Hence in order to test class a301, classes a201, al0l, a0, b201, bl0l and b0 need to have already been tested.
Containment occurs where a member of one class is an object of another class. In example 3a, class a102 contains an object of class b201. The use or mis-use of containment, along with inheritance, is a design issue and beyond the scope of this report. However, legitimate use of containment results in a class from one branch of an inheritance hierarchy containing an object from a different branch. Containment frequently complicates class inter-relationships.
Example 3a - Typical Class Inheritance Hierarchy
Rumbaugh Model. Triangles represent inheritance, the Diamond containment. The top of the diagram represents the most abstract classes.
i.e.
class b101: public b0
{
| // class
| // definition
};
i.e.
class a102: public a0
{
public:
// public members
|
private:
|
b201 a102_b201;
// class a102
// contains an
// object of type
// b201
};
An example of a real situation where containment occurs, is when a class wishes to store/retrieve data from a particular file, which is often achieved by declaring a member of the class to be an object of the file-handling class.
A further complicating factor is global objects These occur where a member function of a class uses a globally instantiated object of another class. Good design should seek to minimise the use of global objects. However, in practice, the use of global objects is often unavoidable to implement objects such as semaphores, queues and common data areas.
In practice, hierarchical class testing will often result in having to wait for the successful completion of other unit tests before a class can be unit tested. In example 3a, unit testing of class a102 and all of those deriving from it would have to wait until class b201 had been tested. This problem would be exacerbated further on a project where the two hierarchies shown might well be designed and implemented by different software teams, working to different timescales.
A major consequence of hierarchical class testing is thus to lengthen the overall timescale of a project. In reality, a result of the complex inter-relationships that exist in any object oriented design, is that hierarchical class testing is impractical in any object oriented development which is not trivial.
In summary, hierarchical class testing presents unacceptable obstacles to practical object based software development, for any but a small project. The original premise of an isolation approach therefore has to be pursued.
Testing Classes in Isolation
In isolation testing, classes are tested in isolation from one another. This means that all classes required by a class under test, either by virtue of inheritance, containment or as a result of a global object used in a member function, must be simulated.
The practical way to manage the testing of classes in isolation is to create and maintain a suite of stub classes alongside their real implementations. Each class stub specifies a stub for each of the class member functions, allowing member functions to be simulated from a test script:
(a) To determine that the member function has been called at an expected point.
(b) To enable parameters to the member function to be checked against expected values.
(c) To return values required by each test case.
This suite of class stubs is called the "stubs library".
Prior to testing a class, class stubs for each class requiring simulation are copied from the stubs library into a test stubs file. For a highly derived class, many classes will require simulation and the test stubs file will consequently be large. However, because each test stubs file is created by copying stubs from the stubs library, its creation should not be a particularly onerous or time consuming task.
In practice, the dividing line between isolation testing of classes and hierarchical class testing is occasionally crossed as a matter of practical convenience. Some basic classes are relatively simple and stand-alone ( for example, a class that implements strings). Such base classes are predictable in that their functionality is easily discerned and is unlikely to change (these classes can be considered is analogous to the pre-defined C++ types). Once unit tested, it may be advantageous to use such classes directly in the unit testing of other classes, instead of simulating them.
A similar approach can be applied to library routines, simulating or using previously tested routines as applicable.
Unit Test Maintenance
Unit level testing is not just intended for one-off development use, to aid bug free coding. Unit tests should be repeated whenever a class is modified or used in a new environment. Consequently, all unit tests must be maintained throughout the software life cycle. It should be possible to decompose a system into its fundamental units and repeat all of the unit tests without error at any time in the system's life.
The maintenance of tests is especially important for object oriented software, which is specifically intended to encourage the re-use of software between different projects. In order for a new project to re-use a class, it must have confidence that the class will work in the new project's environment. This confidence can be gained by re-running the test for the class.
The commitment to maintenance does carry an overhead. When developing software in C, if an interface to a unit is changed, then that unit and all units that call that unit must have their tests repeated. However, in an object based C++ development the unit for testing purposes is the class, and if the interface to a class is changed then all tests on classes that require the modified class must be repeated. This often represents a considerably larger volume of re-testing.
This paper has already observed that the relationship between the classes may be somewhat indirect. If a change is made to the interface of a base class, then the amount of re-testing required may well be considerable. (Nevertheless, it is unacceptable to 'bodge' a class in order to avoid altering its interface).
Consequently, it is very important when developing an object based system to get the interfacing between classes right. Minimisation of the need to change interfaces will result from good design, which should enforce the maximum amount of encapsulation, limiting the externally visible members of a class to a minimum.
Object oriented design is a somewhat more iterative process than conventional techniques. Selecting the right objects will result in stable interfaces. Selecting inappropriate objects will result in volatile interfaces and excessive maintenance effort. This is a major cause of concern for software developers when considering object orientation, supporting the belief that the design stage of an object oriented development should be a greater proportion of the total development effort than is conventionally the case.
Integration Testing of Object Oriented Software
This paper has defined a unit level testing strategy for object based C++ developments based on the C++ class as the fundamental unit of test. However, testing at higher levels than units (classes in isolation) is also essential.
With object oriented software there is no reason why this should be substantially different from higher level testing practices adopted when using other software development methodologies. Such testing is basically in the form of proving the relationships between objects and overall system functionality.
In a large object oriented system the software is typically divided into sets of functionally related classes, commonly termed class categories. Class categories may be functionally tested in isolation from one another, using the class stubs created for unit level testing when simulation is required.
For a system consisting of only a few class categories system integration can be performed in a single 'big-bang' stage -integrating all of the class categories in one go, once they are all fully tested.
However, for more complex systems, a gradual approach to integration is more practical.
An initial integration build of the system might consist of a few fully tested class categories, with the rest of the system being simulated. Functional tests on this part of the system can then be performed and problems ironed out. The integration progresses with subsequent builds replacing more and more of the simulated classes with their real implementations.
This approach to integration is very flexible with regard to timescales, allowing integration to begin before all of the class categories (or even classes) are fully tested.
Conclusion
Unit testing is an essential component of software development. For traditional software development methodologies the unit tested is the function or sub-routine. For an object based development using C++ the natural unit for testing is the class.
When contemplating an object based development, the project manager must not only consider the design methodology to be employed, but also how testing is to be performed and verified. An overall testing strategy for the project should be defined before the actual design of the software begins. This will aid good design, since software engineers have to consider how software will be tested as part of the design process.
Each C++ class should be provided with a test method declared as a friend of the class. This will provide the access to the hidden parts of a class needed to achieve full coverage of a class during a rigorous unit test.
For all but small systems, hierarchical unit testing is impractical as a consequence of the complexity of the inter-relationships between classes which results from inheritance, containment and the use of global objects. The solution is to test classes in isolation from one another.
To facilitate unit testing classes in isolation, a suite of class stubs should be created and maintained alongside the real software. These stubs can then be used during unit testing to enable a test script to simulate classes other than the class actually being tested. Class stubs can also prove useful for simulation during higher levels of testing and integration of a system.
Good object oriented design followed by rigorous unit testing of classes should ease system integration. Containment of complexity within objects will also provide containment of problems, allowing straight-forward identification of the objects responsible for faults.
The benefits of an extended design phase, and rigorous unit testing, will be observed in efficient system integration and lower software maintenance costs.
The test example and results can be found on the disk in the directory labelled CANTATA.
