Test fixtures - global, specific for test suite or test case, conditional

Modules
	C++ Example for simple test fixtures
	C++ Example for conditional test fixtures
	C++ header for conditional test fixtures
	C++ implementation for conditional test fixtures

Detailed Description

Overview:

• What are test fixtures?
• Test case fixtures - execute before each test case
• Test suite fixtures - execute before each test case (not suite!)
• Global fixtures - execute once at start of test application
• The problem of test execution time - idea of 'conditional' fixtures
• Conditional test fixtures - execute only if required
• Conditional test case fixtures - execute only if required
• Conditional test suite fixtures - execute once for each suite
• Technical hints: dependencies

What are test fixtures?

Within unit testing a test fixture is used to prepare a system under test (SUT) and its test environment in such a way that a specific test case can be repeatedly executed under well known conditions. Different test cases may have different requirements and may use different test fixtures.

As a typical implementation a fixture is represented by a class (e.g. "MyFixture"). Relevant code to prepare for testing is within class constructor.

Within Constructor do one or more of the following to prepare for testing:

• create simulated objects within test environment
• startup the system with a specific configuration
• remove / create files
• prepare test data
• ensure that SUT and test environment are in a well known state, e.g. perform all needed state changes via available (test) interfaces

Within destructor do one or more of the following to cleanup after testing

• shutdown SUT
• tear down the test environment
• cleanup file system (remove generated files)
• reset state of SUT and test environment
• all steps from above may be skipped as the fixture of the next test case knows best what to prepare within its constructor

Example of a test fixture:

class MySimpleFixture
{
public:
    explicit MySimpleFixture(std::string const& in_configFile = "Default")
        : m_configFile(in_configFile)
    {
        // writes info to test protocol
        TTB_INFO_S("Constructor MySimpleFixture " << m_configFile);
        // start up test environment, set needed state, ...
    }

    ~MySimpleFixture()
    {
        TTB_INFO_S("Destructor MySimpleFixture " << m_configFile);
        // tear down test environment or reset state changes, ...
    }

private:
    std::string m_configFile;
    // ... (other data needed within destructor)
};

Test case fixtures - execute before each test case

In the simplest case the fixture is created within the test case itself. As the fixture is created on the stack its destructor will be called automatically when the code block of the test case is left. This principle works within all types of test frameworks (BOOST, TTB or anything else).

TTB_BOOST_TEST_CASE(TestSomething)
{
    MySimpleFixture myFixture; // Constructor prepares for testing

    // test instructions follow...

    // automatic destruction of MyFixture
}}

In order to parametrize the specific preparations you can also pass arguments to your test fixture:

TTB_BOOST_TEST_CASE(TestSomethingElse)
{
    MySimpleFixture myFixture("ConfigFile_X");

    // test instructions follow...

    // automatic destruction of MyFixture
}}

Test suite fixtures - execute before each test case (not suite!)

Within BOOST test environments test cases can be grouped within test suites. By using macro BOOST_FIXTURE_TEST_SUITE you can specify a test suite with a specific fixture.

You may think that analogous to test case fixtures which are instantiated once for each test case a test suite fixture is instantiated once for the test suite. But this is NOT true: Test suite fixtures are instantiated and destroyed for each test case within the suite. A test suite fixture of a suite containing 10 test cases will therefore be executed 10 times when the whole suite with all test cases is executed.

In the following example the test suite fixture will be called three times before each of the test cases A, B, C:

// Start tag of test suite. Fixture MySimpleFixture is constructed BEFORE EACH test case!
BOOST_FIXTURE_TEST_SUITE(TestSuiteWithFixtureCalledBeforeEachTestCase, MySimpleFixture);

// Test cases within test suite:

TTB_BOOST_TEST_CASE(TestCaseA)
{
}}

TTB_BOOST_TEST_CASE(TestCaseB)
{
}}

TTB_BOOST_TEST_CASE(TestCaseC)
{
}}

// End tag of test suite
BOOST_AUTO_TEST_SUITE_END();

In more detail: the constructor of the test fixture will be executed before a test case function gets called. The destructor of the fixture will be called after a test case function has returned.

A test suite fixture is simply a sort of code simplification: Instead of writing the needed fixture at the begin of each test case you write it only once within the test suite macro. Runtime behaviour of test case specific and test suite specific fixtures is identical.

Global fixture - execute once at start of test application

Within BOOST environments you can define a global fixture which is executed ony once when the test application is started:

BOOST_GLOBAL_FIXTURE(MySimpleFixture);

In more detail: the constructor of the test fixture will be executed before the first test case function gets called. The destructor of the fixture will be called after the last test case function has returned.

The problem of test execution time - idea of 'conditional' fixtures

Test fixtures are a useful means to guarantee that each test case has the chance to run exactly under the same conditions independent from the execution of preceding test cases. A simple method to avoid any leftovers from preceding test cases is to startup again the whole system. This often results in copying/removing files, reading configuration files, loading/unloading DLLs. Compared to the execution time needed for a single test case, the time needed for preparation may be greater by a factor of 100.

As a unit test (hopefully) consists of a greater number of small test cases, the total time needed for execution may reach from some minutes up to some hours and 95% of the time is used only for preparation code within test fixtures.

On the ohter side testing should give a quick response to the developper. It is preferable to get a result from unit test e.g. in less than 30 seconds. Otherwise test execution cannot be an element of the iterative cycle

• change/add some code
• compile
• run unit test
• fix errors
• change/add some code
• etc.

Idea to improve execution time

• Execute time consuming preparations only when they have to be executed
• Avoid repeatedly starting up and shutting down identical configurations of the whole system for subsequent test cases
• Only shutdown and startup the system if a changed configuration is needed and cannnot be reached by quicker state changes via (test) interfaces

Conditional test fixtures - execute only if required

Perform time consuming test fixture operations (e.g. startup and shutdown of a specific system configuration) only when it is really needed. If two subsequent test cases need the same system configuration reuse the already existing system from the preceding test case.

Principle

Each test case requests its startup conditions individually by using a "conditional" fixture (e.g. TTB_COND_FIXTURE(MyFirstStartupFixture) or TTB_COND_FIXTURE_1(MyStartupFixture,"FirstConfig.xml")); At runtime the required fixture will only be created if it is not already existing from preceding test cases.

Behind the scenes of TTB_COND_FIXTURE

The TTB environment will check if an instance with the same class name and parameter list already exists. If there is no such instance a new fixture instance will be created on the heap. TestToolBox::ConditionalFixture stores a pointer to a single fixture instance. The instance is identified by an ID string built from its class name and any number of (streamable) parameters passed to the constructor. As long as the test cases request for a fixture with the same ID string nothing will change. When a test case has new requirements then he will use an other fixture class or at least change some parameter. As a consequence the resulting ID string will change. TTB framework will then destroy the existing fixture and create a new one.

How to implement and use a conditional fixture

The only difference between a regular fixture class and a conditional fixture class is the use of the base class ConditionalFixtureBase:

class MyCondFixture : public TTB::ConditionalFixtureBase
{
public:
    explicit MyCondFixture(std::string const& in_configFile = "Default")
        : m_configFile(in_configFile)
    {
        // writes info to test protocol
        TTB_INFO_S("Constructor MyFixture " << m_configFile);
        // start up test environment, set needed state, ...
    }

    ~MyCondFixture()
    {
        TTB_INFO_S("Destructor MyFixture " << m_configFile);
        // tear doown test environment or reset state changes, ...
    }

private:
    std::string m_configFile;
    // ... (other data needed within destructor)
};

Call your fixture at any place where needed:

// without params
TTB_COND_FIXTURE(MyCondFixture);

You can only use fixtures with streamable parameters:

// with (streamable) params
TTB_COND_FIXTURE_1(MyCondFixture, "FirstConfig");

Final cleanup

As conditional fixture instances are created on the heap a final cleanup step is necessary. When using TTB framework (possibly in combination with BOOST framework) the framework cares for final cleanup automatically. There is no need for special client code.

If you are using TTB::ConditionalFixture alone you have to care for cleanup by the following call:

TTB_COND_FIXTURE_CLEANUP();

Within BOOST test environments a good place for this instruction would be the destructor of the global fixture.

Conditional test case fixtures - execute only if required

The following 5 test cases are using 2 different fixtures. With use of conditional fixtures only 2 fixture instances will be created:

TTB_BOOST_TEST_CASE(TestTopicA)
{
    TTB_COND_FIXTURE(MyCondFixture); // -> create fixture on heap

    // test instructions follow...
}}

TTB_BOOST_TEST_CASE(TestTopicB)
{
    TTB_COND_FIXTURE(MyCondFixture); // no action!
}}

TTB_BOOST_TEST_CASE(TestTopicC)
{
    TTB_COND_FIXTURE(MyCondFixture); // no action!
}}

TTB_BOOST_TEST_CASE(TestTopicD)
{
    TTB_COND_FIXTURE_1(MyCondFixture, "FirstConfig");
    // -> delete last fixture and create new fixture instance
}}

TTB_BOOST_TEST_CASE(TestTopicE)
{
    TTB_COND_FIXTURE_1(MyCondFixture, "FirstConfig"); // no action
}}

With more detail: all test cases are using the same fixture class, but the first 3 test cases are using class MyCondFixture with default constructor argument and the last 2 test cases have an different constructor argument.

Conditional test suite fixtures - execute once for each suite

Combining a BOOST test suite fixture with an embedded conditional fixture allows executing of fixture code only once for the test suite.

Example for a test sute fixture:

// Possibility: formulate a test fixture which is only executed at the begin of the testsuite
class MyBoostTestSuiteFixture
{
public:
    MyBoostTestSuiteFixture()
    {
        // The boost test suite fixture "MyBoostTestSuiteFixture" will be constructed each
        // time a BOOST test case within the suite is executed.

        // The following line will be called for each test case,
        // BUT only for the first test case the test fixture "MyFixture" will be constructed!
        TTB_COND_FIXTURE_1(MyCondFixture, "ConfigXY");

        // You may add initializations which will execute before each test case
        // ...
    }
};

When running the following test suite the embedded conditional fixture will be executed only once:

BOOST_FIXTURE_TEST_SUITE(TestSuiteWithOneTimeInitialization, MyBoostTestSuiteFixture);

TTB_BOOST_TEST_CASE(TestCaseA) // -> creation of MyCondFixture
{
}}

TTB_BOOST_TEST_CASE(TestCaseB) // no action
{
}}

TTB_BOOST_TEST_CASE(TestCaseC) // no action
{
}}

BOOST_AUTO_TEST_SUITE_END();

This mechanism will also work when arbitrary test cases are selected for execution by command line parameters. There is no need of selecting special test cases responsible for startup and shutdown if you provide appropriate conditional fixtures.

Technical hints: dependencies

Use TTB::ConditionalFixture together with other TTB features

The simplest way of using conditional fixtures is to use the TTB test framework or a combination of TTB and BOOST test framework:

• include of TestTolBox.h/TestToolBoxImpl.h is enough
• cleanup will be automatically provided
• as default behaviour TTB::ConditionalFixture will write informational messages to TTB test protocol

Minimize dependencies - Use TTB::ConditionalFixture as standalone feature

• include ConditionalFixture.h where needed
• specify define "TTB_COND_FIXTURE_STANDALONE" and include ConditionalFixtureImpl.h in a single cpp file
• at end of test care for destroying a possibly existing conditional fixture instance via a call to TTB::ConditionalFixture::Cleanup() or TTB_COND_FIXTURE_CLEANUP().