Boost.Test combined with TTB

Modules
	C++ Example for main source file
	C++ Example for test case source file

Detailed Description

Overview:

• Motivation
• Structure of a typical test project
• Required project settings
• Main source file MyTestApp.cpp
• Test case file TestCaseX.cpp
• Command line params to control runtime behaviour
  • Useful BoostTest command line params
  • TestToolBox command line params
    • Control recording of test events to prepare for copy to source code (-prefixEvents TTB_EXP)
    • Activate/deactivate sections of your test code (-suppressTopicA)
    • Retrieve numeric values from command line (-valX 17.2)
• Writing info messages to stdout and protocol files (TTB_OUT_S)
• References

Motivation

Boost.Test is a proven open source testing library which provides a robust runtime framework. With Boost.Test you can organize your test cases within suites and select the test cases or suites to execute from the command line. Many more command line options and other possibilities are available.

The TestToolBox (TTB) is a collection of test orientated classes which can run on their own but which can also be used together with Boost. By adding TTB to a Boost.Test application you can get the following possibilities:

• generation of a detailed test protocol (may be a starting point for error analysis, e.g. for tests failed during last nightly build)
• comfortably collecting of TestEvents possibly generated from several objects of your test environment, possibly originating from different threads
• automatic check of recorded events, allowing random order of events where needed
• synchronization within multithreaded environments: synchronize your test script (usually running within your main thread) with worker threads of test environment and test object
• support for automated checking of XML data (e.g. possibility to ignore uninteresting nodes and attributes, check double values with given precision, select subnodes with XPath)
• more features ...

The following chapters describe how you can add TTB functionality to a Boost.Test application and what are the necessary steps to setup a project.

Structure of a typical test project

A typical test project consists at least of the following parts:

• project settings:
  contains library path definitions and preprocessor defines to configure code generation
• MyTestApp.cpp:
  main source file containing
  • instructions to parametrize, include and link boost library and TestToolBox,
  • a "global fixture" containing initialization / cleanup routines to be executed before / after execution of the set of all test cases, furthermore here the connection to TestToolBox is established
• TestCasesA.cpp, TestCasesB.cpp,... :
  
  • multiple files containing specific test cases possibly organized within test suites
  • optional "test suite fixtures" with initialization / cleanup routines executed before / after the execution of each test case within the fixture

Required project settings

• C/C++ settings
  • General / Additional Include Directories: c:/boost/boost_actual;
  • Preprocessor Definitions
    • TTB_USE_BOOST_TEST:
      TestToolBox will activate code to support Boost test framework
    • TTB_NOT_INLINE:
      prepare source code to be directly included via TestToolBoxImpl.h within a single source file of your test project
    • TTB_USE_STANDARD_ERROR:
      activate default error format to store catched and returned errors
    • TTB_USE_PUGI_XML:
      optional switch, needed to add support for checking XML data with pugi xml; assumption: you have already added pugixml lib to your project.
    • TRACE_IS_ON:
      optional switch to activate trace output from TestToolBox sources
• Linker
  • General / Additional Library Directories: c:/boost/boost_actual/lib;

Main source file MyTestApp.cpp

Connect with Boost.Test library:

// To write boost test cases within multiple source files
// define the name of your test module within exactly one cpp file
// and before including any boost test header.
// Do not define BOOST_TEST_MAIN. Otherwise you may get link errors
// like "init_unit_test_suite already defined"
#define BOOST_TEST_MODULE "Testing XmlCheck with PugiXml"

#include <boost/bind.hpp>
#include <boost/test/auto_unit_test.hpp>

// Use the boost auto link feature which automatically chooses the correct library version
#define BOOST_LIB_NAME boost_unit_test_framework
#include <boost/config/auto_link.hpp>

Connect with implementation of TestToolBox:

// directly include implementation code,
// possible configuration via preprocessor definitions within project settings
#include "TestToolBox\TestToolBoxImpl.h"

// Define name of this component for simple trace utility and set initial trace level
TTB_TRC_DEF_GLOBALS("TestXmlCheckWithPugiXmlc", TL_VAL_DEBUG);

namespace TTB = TestToolBox;

Define your global test fixture (typical example):

// Global Fixture for onetime initialization and tear down.
// Base class DefaultSettingsForBoostAndTTB cares for proper initialization
// and cleanup of TestToolBox (e.g. test protocol files, recorded TestEvents)
struct MyGlobalSettingsForTestToolBox : public TTB::DefaultSettingsForBoostAndTTB
{
    MyGlobalSettingsForTestToolBox(void)
    {
        // In case of using TTB::TestEvents you can specify your desired
        // syntax for writing test events to protocol file.
        // This may be useful when writing / correcting test cases because
        // you can directly copy recorded test sequences from protocol file
        // to your source code.
        // When defining via command line the needed check macro as prefix
        // (e.g. "-prefixEvents TTB_EXP") you get the ready to copy format
        // in C++ syntax (TTB_EXP("Some event generated");).
        // If prefix string is not set the simple and more readable prefix
        // ">" is used.
        std::string prefixForTestEventsWithinOutFile =
            TTB::TheEnvironment()->GetCommandLineOption("-prefixEvents");
        bool detailedLog = true;
        TTB::TestEvents::Get()->SetDetailedLog(
            detailedLog, prefixForTestEventsWithinOutFile);

        // Specific preparations
        // e.g. generate test data, startup system under test,
        // execute initial test cases to be performed directly after startup
        InitialPreparation();
    }

    // Define your specific destructor if you have to perform special cleanup
    // at end of testing (e.g. shutdown test environment)
};

Install global fixture:

// Install your specific class MyGlobalSettingsForTestToolBox
// as a global fixture for boost
BOOST_GLOBAL_FIXTURE(MyGlobalSettingsForTestToolBox);

For a complete example see C++ Example for main source file

Test case file TestCaseX.cpp

Includes needed to use both Boost and TTB functionality:

#include <TestToolBox\TestToolBox.h>
namespace TTB = TestToolBox;

Define your test suite fixture as needed:

class MyTestSuiteFixture
{
public:
    MyTestSuiteFixture()
    {
        // Log to test protocol
        TTB_INFO("\nMyTestSuiteFixture-Constructor");

        // Do some preparations needed before each test case
        // within this file / test suite
        // ...
    }
    ~MyTestSuiteFixture()
    {
        // Log to test protocol
        TTB_INFO("\nMyTestSuiteFixture-Destructor");

        // Reset some settings possibly changed by a preceding test.
        // For example reset XmlCheck to default settings:
        TTB::TheXmlCheck()->SetDefaults();

        // Other reset actions
        // ...
    }
};

Define your test cases:

// Define fixture which is executed for each test case within this file
BOOST_FIXTURE_TEST_SUITE(SuiteTestingSomeSubject, MyTestSuiteFixture)

//-----------------------------------------------------------------------------

// Represents test case visible for both Boost and TestToolBox
TTB_BOOST_TEST_CASE(DemoTestCase)
{
    int a = 4;
    int b = 2;

    // You can use Boost check macros
    BOOST_CHECK(a == 2 * b);

    // You can use TTB check macros
    TTB_CHECK_EQUAL(a, 2 * b);
    TTB_ACT_S("a=" << a << ", b=" << b);
    TTB_EXP("a=4, b=2");

    // more meaningful checks
    // ...

}}; // yes, there are double paranthesis!

//-----------------------------------------------------------------------------

// continue with other test cases
// ...

//-----------------------------------------------------------------------------

// closing tag for test suite
BOOST_AUTO_TEST_SUITE_END()

// end of cpp file

For a complete example see C++ Example for test case source file

Command line params to control runtime behaviour

Both Boost Test and TestToolBox can be controlled via command line params. Because of a slightly differing syntax you can mix params for both tools. Each tool will only take into account those params matching with the expected syntax. All params with other syntax will be ignored without any complaints.

General syntax:

Tool set	Prefix for params	Example
Boost Test	two dashes	–option_a=some_value
TTB	one dash	-optionA or -optionA someValueA

Useful BoostTest command line params

Table of useful parameter setttings:

Boost Test Command	Purpose
–log_level=test_suite	Boost Test will write more output, useful for better understanding of control flow and error conditions
–run_test=SuiteA	run all test cases of your test suite with name "SuiteA"
–run_test=SuiteA/TestCase3	run only the single test case "TestCase3" within test suite with name "SuiteA"
–run_test=SuiteDo*,*MyTop*	run all test suites with matching names, e.g. "SuiteDoSomething" and "SuiteForMyTopic"
–detect_memory_leak=47	for finding memory leaks specify the allocation number you have received from the memory leak dump after program termination, only to be used within debugger!

Full list of command line params at Boost.org

TestToolBox command line params

When the TestToolBox is used in combination with Boost Test (i.e. TTB is running without its runtime framework TTB::TestRunner) there are no predefined command line params. But you can easily add params of your choice by using the possibilities within TestToolBox::Environment to access the command line:

• IsExistingCommandLineOption("-optionA")
  Checks whether the given commmand line option is set
• GetCommandLineOptionVal<int>("-optionA")
  converts the token following to -optionA to int type. You can convert to any basic type by setting a proper template type.

Table of possible / recommended parameter settings:

Recommended TTB Command	Purpose
-prefixEvents TTB_EXP	If set different from empty string, then C++ syntax will be used for recording test events. Thus direct copying of text lines from test protocol file to your C++ test specification is posssible
-suppressTopicA	You can skip code sections within your test script via checking TTB::TheEnvironment()->IsExistingCommandLineOption("-suppressTopicA")
-valX 17.2	Read some value needed for testing (e.g. number of executions, values to use for some input to your test object,...) directly from command line using: double valX = TTB::TheEnvironment()->GetCommandLineOptionVal<double>("-valX")

Control recording of test events to prepare for copy to source code (-prefixEvents TTB_EXP)

With setting command line param "-prefixEvents TTB_EXP" you can set the output format of recorded test events to allow direct copying of selected lines from test protocol to the test cases within your C++ source code.

• The exact syntax for the name of the option ("-prefixEvents") has to be defined within your global test fixture, see Example for global test fixture.
• The needed option value ("TTB_EXP") depends on the check macro which you are using within the test case you are working on. You can vary the option to other values (e.g. "TTB_EXP_VAR") as you need it.
• Resulting entry within test protocol is: TTB_EXP("someVal=3.14");
• (more readable) protocol entry without this option: >someVal=3.14

Activate/deactivate sections of your test code (-suppressTopicA)

Depending on the runtime environment (e.g. development computer, cruise control, nightly build) you may want to deactivate portions of your test cases. Possible reasons could be:

• skip time consuming actions
• skip actions for which there is no runtime environment available at the current system
• skip experimental code you want to execute only sometimes, but for which you do not want to supply automatic verification

Example how to skip parts of a test case via option "-suppressSomething":

    // Optionaly skip execution
    if (!TTB::TheEnvironment()->IsExistingCommandLineOption("-suppressDemoOutput"))
    {
        DemonstrateUsageOfPugiXml();
    }

Retrieve numeric values from command line (-valX 17.2)

In a similar way you can use the command line to retrieve values for test execution (e.g. number of executions, values to use for some input to your test object,...).

Example how to read some value from command line:

    // For demonstration purpose read some value from command line.
    // A not existing value will be default constructed (here 0.0)
    double valX = TTB::TheEnvironment()->GetCommandLineOptionVal<double>("-valX");

    // Write value as informational message to test protocol
    TTB_INFO_S("command line param valX=" << valX);

Writing info messages to stdout and protocol files (TTB_OUT_S)

Sometimes it is useful to write informations to stdout while the test cases are executing. Instead of using "std::out" or BOOST_TEST_MESSAGE directly there is the more generic way of using macro TTB_OUT_S. This macro allows the selection of output targets (stdout, .out/.rpt files) and the messages can be switched off without changing any other outputs:

    // Illustrate usage of macros TTB_OUT_S and TTTB_T_OUT_S

    bool someCondition = false;
    double pi = 3.14159265359;

    // Macro TTB_OUT_S writes informational messages to one or more of the
    // targets .out, .rpt, stdout. The calls do not affect test results!
    TTB_OUT_S("This macro supports stream syntax x=" << std::setprecision(4) << pi
        << " condition=" << std::boolalpha << someCondition);
    TTB_OUT_S("With default settings the output goes to .out + .rpt + stdout");

    // Change target for TTB_OUT_S
    TTB::TheProtocol()->SelectTargetsForOutS(TTB::OutputMode::M_STDOUT);
    TTB_OUT_S("After change of default settings this message goes only to stdout");

    // You can select multiple targets
    TTB::TheProtocol()->SelectTargetsForOutS(TTB::OutputMode::M_STDOUT | TTB::OutputMode::M_OUTFILE);
    TTB_OUT_S("After another change of settings this message goes to .out + stdout");

    // Explicitly specify target for a single message
    TTB_T_OUT_S(TTB::OutputMode::M_STDOUT, "This message with explicit target goes to stdout");
    TTB_T_OUT_S(TTB::OutputMode::M_OUTFILE| TTB::OutputMode::M_RPTFILE,
        "This message with explicit targets goes to .out + .rpt");

    // Switching off output of TTB_OUT_S:
    // As default class Protocol produces output for messages up to verbosity eREGULAR.
    // By changing verbosity of macro TTB_OUT_S from its default eREGULAR to eVERBOUS
    // the output through TTB_OUT_S can be switched off completely.
    TTB::TheProtocol()->SetVerbosityForOutS(TTB::OutputLevel::eVERBOUS);
    TTB::TheProtocol()->SelectTargetsForOutS(TTB::OutputMode::M_STDOUT | TTB::OutputMode::M_OUTFILE | TTB::OutputMode::M_RPTFILE);
    TTB_OUT_S("This verbous message goes to nowhere");

    TTB::TheProtocol()->SetVerbosityForOutS(TTB::OutputLevel::eREGULAR);
    TTB_OUT_S("And this regular message goes to out + rpt + stdout");

References

• Boost.Test online documentation at Boost.org
• Full list of command line params at Boost.org
• C++ Example for main source file
• C++ Example for test case source file