Using Runner
[Using instructions]

Class Runner executes a user defined set of test functions containing arbitrary test cases. More...

Collaboration diagram for Using Runner:

Defines
#define	TTB_TEST_FUNC(param_name)
	Macro to define a test function to be executed under control of TestToolBox::TestRunner().
#define	TTB_TEST_FUNC_DESC(param_name, param_description)
	Same as TTB_TEST_FUNC but with possibility to assign an arbitrary descriptive text to the test function.
#define	TTB_TEST_CASE(DESC)
	Macro to define a test case.
#define	TTB_USE_DEFAULT_TEST_RUNNER()
	Macro to define the default implementation of main function executing all test cases with TestToolBox::TestRunner().
#define	TTB_USE_TEST_RUNNER_WITH_EXTENSIONS(YOUR_EXTENSION_CLASS_TYPE)
	Defines main(argc,argv) which executes all test functions via TestToolBox::TestRunner. Specific initializations can be embedded in the test sequence by providing your own extension class.

Detailed Description

Class Runner executes a user defined set of test functions containing arbitrary test cases.

The test functions have to be declared within macros TTB_TEST_FUNC or TTB_TEST_FUNC_DESC.

Supported features

provides a controlling main function, users have only to define the test functions (similar to automatic test cases within Boost::Test)

possibility to select and sort the test functions for execution

relevant test events and results are written to logfiles
- Report YourTestApp.rpt: overview of test cases and results
- Protocol YourTestApp.out: more detailed log file, usable for analysis of test errors

optional time controlled repeated execution of test functions for long duration tests with check of performance counters (e.g. consumption of memory)

check for memory leaks

any error detected during testing (e.g. memory leak) will result in an exit code different from 0

Example for a simple test application

 #include "TestToolBox/RunnerImpl.h" // includes also implementation code

 //include headers of your specific test object
 #include ...

 // Define name of this component for simple trace utility and set initial trace level
 TTB_TRC_DEF_GLOBALS ("TestRunner", TL_VAL_DEBUG);

 // automatic execution of main with execution of all test functions
 // via TestToolBox::TestRunner
 TTB_USE_DEFAULT_TEST_RUNNER();

 // here follows your first test function
 TTB_TEST_FUNC (YourFirstTestFunc)
 {
     // The name of the test function implicitly is the name of the first test case.
 
     // Instantiate the object to test and prepare for testing
     YourObjectUnderTest yourObject;
     ...

     // Call some interface of your object under test
     yourObject.DoSomething();
     ...

     // Here we assume, that the reaction somehow arrives within
     // TestToolBox::TestEvents (e.g. through synchronous or
     // asynchronous callback; the callback object may simply generate
     // one or more entries within TestEvents which can be immediately
     // verified).

    // Check the received result
    TTB_EXP("CallBack-Result: 14");

    // optionally proceed with further interactions with your object under test
    ...
 }

 // here follows your second test function
 TTB_TEST_FUNC (YourSecondTestFunc)
 {
     ...
 }

Hint: It may be a good advise to always use a new test function when you need a new test case.

Example of a test report file

 Generic test protocol template / TestToolBox (TTB) 2010
 
 
                     T E S T   P R O T O C O L
 
 Date              : 30.05.2010 16:36:33
 Test application  : c:\My Target\Debug\TestRunner.exe
 Test protocol     : c:\My Target\Debug\TestRunner.out
 Test report       : c:\My Target\Debug\TestRunner.rpt
 Source path       : c:\userdata\gerald\sw\c\testtoolbox
 Command line args : "-checkForMemLeaks" "AT_SHUTDOWN"
 Run on computer   : SIRIUS
 Run by            : Gerald
 ________________________________________________________________________________
 
 Executing 9 of 9 specified test functions
 
 Test: YourFirstTestFunc
 Test: YourSecondTestFunc
      ...
 ________________________________________________________________________________
 
 Test started    : 30.05.2010 16:36:33
 Test ended      : 30.05.2010 16:36:33
 Duration        :            00:00:00,016

 Tests performed : 9
 Tests failed    : 0

 No error detected!

Snippet from a test protocol file

 ...
 ================================================================================
 Test: WaitForSingleAsyncAnswer
 ================================================================================

 Calling CalcAsync(7)
 (Waiting for asynchronous result via callback)

 >CallBack-Result: 14
 
 ================================================================================
 Test: Waiting for multiple sync answers
 ================================================================================
 

 <so.CalcSync          2
 >CallBack-Result: 4

 <so.CalcSync          33
 >CallBack-Result: 66

 <so.CalcSync          99
 >CallBack-Result: 198
 ...

Recommendations for organizing and running your tests

initialization and cleanup for your test environment or for single test functions is possible by providing an extension class derived from TestToolBox::RunnerExtensionPoints and using macro TTB_USE_TEST_RUNNER_WITH_EXTENSIONS

you should organize your test functions in several source files, e.g.:
- YourTestApp.cpp: setup your test environment (e.g. define your extension class, define global data used within your tests)
- YourTestTopicA.cpp, YourTestTopicB.cpp: implement test functions/test cases and group them
- you can use command line option -selectTestFile to execute tests from only one or more source files

you can define additional test cases within your test functions using macro TTB_TEST_CASE

to improve the information written to test protocol file you can use macros TTB_INFO, TTB_SECTION, TTB_CMD, TTB_CMD1,...
For more information see TestEvents.h and Using TestEvents

for handling asynchronous reactions you should use macros TTB_INIT_SYNC, TTB_WAIT_SYNC, TTB_SYNC, TTB_SET_SYNC_TIMEOUT
For more information see TestToolBox::Synchronizer and Using Synchronizer

for checking the observed results within your test use macros TTB_EXP, TTB_SEQ and TTB_EXP_VAR
For more information see TestToolBox::TestEvents and TestEvents.h

Command line arguments

Help about possible arguments is available through param "-?":

 >MyTestApp.exe -?

 TestRunner / TestToolBox (TTB) 2010
 
 
 Commandline Syntax
 
 -outputLevel SILENT | FATAL | ERROR | REGULAR | VERBOUS
 
   adjust amount of output written to stdout
 
       SILENT            : no output at all
       FATAL             : display only fatal execution errors
       ERROR             : display also regular errors detected
                           within test cases
       REGULAR (default) : medium level of output, show progress
                           of test cases just executing
       VERBOUS           : maximum level of output, shows more
                           details about execution, also rises amount
                           of output written to protocol file
 
 -selectTestFunc <expr1>,<expr2>,...
 
   allows filtering of the test functions to execute,
   you can combine several filter conditions by separating them
   with ",". The filter expression will be compared with the
   name of the test function as defined within TTB_TEST_FUNC and
   TTB_TEST_FUNC_DESC (in the last case the long name is used)
 
   Example: Assume a test func has the name "MySpecialTest"
   Then the expressions "My*", "*ialTest", "*eci*" and
   "MySpecialTest" will match and "My", "Test" and "*Z*"
   will not match
 
   Default: no filter set, i.e. all test functions are selected
   for execution

 -selectTestFile <expr1>,<expr2>,...
 
   allows filtering of the test functions by selecting the file,
   name where the test is implemented.
   For more details of defining filter expressions see description
   of option "-selectTestFunc"
   The filter conditions of both options are combined via logical
   AND
 
 -sort ALPHABETICAL
 
   execute the test functions in alphabetical order of their names
 
 -loopTimeSec <numSeconds>
 
   repeat the execution of test cases until the specified amount
   of time has elapsed
   performance data will be recorded within logfile and written to stdout
 
 -perfDataIntervalSec <numSeconds>
 
   record performance data every time the given period
   has elapsed
 
 -perfCounterConfig XpEnglish | XpGerman
 
   the name of the performance counters is set according to the
   windows system
   Default: XpGerman
 
 -prefixForReceivedTestEvents <MyPrefix>
 
   generates modified output of test events to support direct copying
   to test script. E.g. setting prefix 'TTB_EXP' generates output
   'TTB_EXP("some result: 42");' instead of '>some result: 42'
 
 -checkForMemLeaks OFF | AT_MAIN_EXIT | AT_SHUTDOWN
                       | AT_MAIN_EXIT+AT_SHUTDOWN
 
     OFF          : no check for memory leaks
     AT_MAIN_EXIT : check if all data on heap has been released
                    when main() has finished. A detected memory
                    leak will cause an exit code of 1
     AT_SHUTDOWN  : check if all data allocated since program
                    startup has been released at time of shutting down.
                    Startup and shutdown is defined by creation and
                    destruction of an internal static object residing within a
                    compiler segment of type 'lib'. A detected memory
                    leak will cause an exit code of 3 and a fatal system
                    message (which can be ignored)
 
   Hint: static and global data of your specific test environment may
   also have an effect on memory allocation and deallocation both before
   and after execution of main function.
   Default: AT_MAIN_EXIT+AT_SHUTDOWN
 
 -breakAtMemAlloc <N>
 
   break program execution when the given memory allocation is executed.
   the number to specify can be derived from memory leak reports.
   Works only when running within debugger.
 
 -? or -help
 
   show these infos and exit test application

See also:: Runner.h (File documentation); TestToolBox::Runner (class documentation)

Define Documentation

#define TTB_TEST_FUNC ( param_name )

Value:

static void TestFunc_##param_name();                                \
static void ExecuteTestFunc_##param_name()                          \
{                                                                   \
    TestToolBox::TestEvents::Get()->CheckForUnexpectedEvents (      \
        "next test func", __FILE__,__LINE__);                       \
    TestToolBox::TestEvents::Get()->StartNewTestCase(#param_name);  \
    TestFunc_##param_name();                                        \
};                                                                  \
TestToolBox::Runner::Register regTestFunc_##param_name(#param_name, \
    __FILE__,__LINE__, &ExecuteTestFunc_##param_name);              \
static void TestFunc_##param_name()

Macro to define a test function to be executed under control of TestToolBox::TestRunner().

The trick for automatic registration is a simplified variant of methods used within BOOST::Test (http://www.boost.org/doc/libs/1_43_0/libs/test/doc/html/index.html)

First there will be created an additional execution function which checks for test errors (of preceding activities), defines a test case with the name of your test function and finally calls your test function.

Second a special (file global) register object is instantiated for each test function. The register object is constructed before main() will be called. Within the constructor the test function is registered within the TestToolBox::Runner singleton instance. When main starts the Runner instance can call all registered test functions.

#define TTB_TEST_FUNC_DESC	(	param_name,
		param_description		)

Value:

static void TestFunc_##param_name();                                        \
static void ExecuteTestFunc_##param_name()                                  \
{                                                                           \
    TestToolBox::TestEvents::Get()->CheckForUnexpectedEvents (              \
        "next test func", __FILE__,__LINE__);                               \
    TestToolBox::TestEvents::Get()->StartNewTestCase(param_description);    \
    TestFunc_##param_name();                                                \
};                                                                          \
TestToolBox::Runner::Register regTestFunc_##param_name(##param_description, \
    __FILE__,__LINE__, &ExecuteTestFunc_##param_name);                      \
static void TestFunc_##param_name()

Same as TTB_TEST_FUNC but with possibility to assign an arbitrary descriptive text to the test function.

The descriptive text is used to identify the test function e.g. when filtering or sorting test functions via command line options -selectTestFunc and -sort.

#define TTB_TEST_CASE ( DESC )

Value:

TestToolBox::TestEvents::Get()->CheckForUnexpectedEvents  ("next test case", __FILE__,__LINE__); \
TestToolBox::TestEvents::Get()->StartNewTestCase(DESC);

Macro to define a test case.

By default TTB_TEST_FUNC and TTB_TEST_FUNC_DESC define a test function which automatically defines a test case. If you need further partitioning you can define additional test cases within your test function with use of macro TTB_TEST_CASE.

#define TTB_USE_DEFAULT_TEST_RUNNER ( )

Value:

int _tmain(int argc, _TCHAR* argv[])                                        \
{                                                                           \
    return TestToolBox::Runner::RunMain<TestToolBox::RunnerExtensionPoints> \
        ("main",__FILE__,__LINE__);                                         \
}

Macro to define the default implementation of main function executing all test cases with TestToolBox::TestRunner().

The test application will execute the following steps:

TestRunner::Execute (execution of all test functions you have defined)
final check for test results
optional check for memory leaks
main will return an exit code of 1 if an error is detected within the tests or a memory leak was detected otherwise 0

#define TTB_USE_TEST_RUNNER_WITH_EXTENSIONS ( YOUR_EXTENSION_CLASS_TYPE )

Value:

int _tmain(int argc, _TCHAR* argv[])                                   \
{                                                                      \
    return  TestToolBox::Runner::RunMain<YOUR_EXTENSION_CLASS_TYPE>    \
        ("main",__FILE__,__LINE__);                                    \
}

Defines main(argc,argv) which executes all test functions via TestToolBox::TestRunner. Specific initializations can be embedded in the test sequence by providing your own extension class.

The test application will execute the following steps:

constructor of YOUR_EXTENSION_CLASS_TYPE
TestRunner::Execute (execution of all test functions you have defined)
- before each test function the extension method BeforeCallOfTestFunction() of your extension class is called
- after each test function the extension method AfterCallOfTestFunction() of your extension class is called
destructor of YOUR_EXTENSION_CLASS_TYPE
final check for test results
optional check for memory leaks
main will return an exit code of 1 if an error is detected within the tests or a memory leak was detected otherwise 0

Hint: it is allowed to define test cases (e.g. by using TTB_TEST_CASE) also within constructor and destructor of YOUR_EXTENSION_CLASS_TYPE

Using Runner [Using instructions]

Defines

Detailed Description

Define Documentation

Using Runner
[Using instructions]