Simulated Objects

Inhalt

Top1 Motivation and concept

Top1.1 Embedding simulated objects into test environment

Simulated objects (sometimes also called fake objects or mock objects) are an important part of a test environment. They form a connection point between the "real" system under test and the test environment:

Simulated object 1

From the system under test (SUT) they are needed as regular communication partners fulfilling some services. For the test environment they give information about the communication control flows in which the system under test is involved and they offer the possibility to send some trigger to the SUT.

Top1.2 Identifying common behaviour

There are many types of applications and use cases SW systems have to deal with. Nevertheless there are common tasks a simulated object has to be prepared for:
  • record function calls ("test events") which can be automatically verified to proof correct control flows for the system under test
  • allow proper simulation of function output parameters and (error) return values
  • especially in multithreaded environments: allow control of execution (synchronization, simulation of blocking calls)
  • depending on the test case only specific parts of a control flow are important for verification, the not interesting parts should be switched off to simplify testing and sustaining
Simple idea:
Provide as much as possible of the common behaviour (e.g. test control and verification) within a base class. Then your specific simulated object's implementation can be focussed on application specific code.
For implementing a specific simulated object you can derive from a base class SimulatedObjectBase which is provided by a general testing framework:

Simulated object base

Top2 Quick start guide for using SimulatedObjectBase

This section gives instructions how to use the functionalities of the testing framework. For a deeper look into some design decisions and internal mechanisms see section Internal design.

Top2.1 Implementing a specific simulated object / fake object

Lets assume your specific test environment requires the implementation of an interface ISomeService with two methods CalculateOne() and CalculateTwo(). Then the header file of your simulated object will look similar to: 
#include "ISomeService.h" // regular interface to be implemented
#include "TestToolBox/SimulatedObjectBase.h"

class MySimulatedObject
    : public ISomeService
    , public TestToolBox::SimulatedObjectBase
{
public:
    // Constructor
    MySimulatedObject (std::string const & in_objectName);

    // Interface ISomeService
    void CalculateOne (int in_val, int& out_result);
    void CalculateTwo (int in_val, int& out_result);
};
To connect to the base functionality for simulated objects as given by the test framework you have simply to derive your class from SimulatedObjectBase. Within constructor you have to pass the instance name and class type of your object to the base class: 
MySimulatedObject::MySimulatedObject(
    std::string const & in_objectName)
    : SimulatedObjectBase (in_objectName, TTB_CLASS_NAME)
{
    ...
}

The macro TTB_CLASS_NAME will automatically deduce the name of your specific class. The names of enclosing namespaces are removed. In the Example above the result will be "MySimulatedObject".

Default constructor
Instead of having to pass the object name to the constructor of your specific class you could also enable a default constructor: 
MySimulatedObject::MySimulatedObject(void)
    : SimulatedObjectBase ("noSpecificName", TTB_CLASS_NAME)
{
    // Here or in any other method you can change the object name
    SimulatedObjectBase::SetObjectName (SomeSpecificFunctionToBuildName());
}
Simply pass a dummy name to the base class within initializer list. If needed build the final instance name by calling base method SetObjectName within constructor body or in any other method.

Probably you will have to choose specific instance names only for the case when there are multiple instances of the same class type and you want to address a specific instance.

Top2.2 Tracking method calls - TTB_METHOD_CALL

In many cases the interface methods can be implemented in one or two lines: 
void MySimulatedObject::CalculateOne (int in_val, int& out_result)
{
    out_result = 17; // if result is needed within your test

    TTB_METHOD_CALL();
}
Within macro TTB_METHOD_CALL() the test framework automatically deduces the name of your function. When during testing your simulated object gets called the following output will be recorded as test event: 
CalculateOne

For automatic verification of the occured control flow you now can write within your test case: 

SendSomeTriggerToTheSystemUnderTest();

// Now expecting:
TTB_EXP("CalculateOne");

Top2.3 Tracking method calls with parameters - TTB_METHOD_CALL1

Besides the name of the called function often the input (and sometimes also the output) parameters are of interest during testing: 
void MySimulatedObject::CalculateTwo (int in_val, int& out_result)
{
    out_result = in_val * 2;

    TTB_METHOD_CALL1("in_value=" << in_value << " out_result=" << out_result);
}
Macro TTB_METHOD_CALL1() allows to specify any output in stream syntax. When during testing your simulated object gets called the following output will be recorded as test event: 
CalculateTwo in_value=13 out_result=26

For automatic verification of the occured control flow you now can simply write within your test case: 

SendSomeTriggersToTheSystemUnderTest();

// In this test case expecting several calls:
TTB_EXP("CalculateTwo in_value=13 out_result=26");
TTB_EXP("CalculateTwo in_value=10 out_result=20");
TTB_EXP("CalculateTwo in_value=-7 out_result=-14");

Top2.4 Tracking the names of your fake objects - Option::oOBJECT_NAME

Frequently multiple instances of fake objects of the same or different class types are used within a test environment. To be able to recognize whether the right instance was called within your test sequence you should activate the tracking of object names. 
TTB::TheGlobalOptions()->Set(Option::oOBJECT_NAME);
An appropriate location where to add this instruction could be the initialization routine where you setup your environment.

When during testing your simulated object gets called the following output will be recorded as test event: 

CalculateOne (MyObjectName)
CalculateTwo in_value=5 out_result=10 (MyObjectName)
It is possible to switch on tracking of object names only for a specific class type or even only for a specific method: 
// activate object name for class type
TTB::TheGlobalOptions()->Set(Option::oOBJECT_NAME,"MySimulatedObject");

// activate object name for specific method
TTB::TheGlobalOptions()->Set(Option::oOBJECT_NAME,"MySimulatedObject::CalculateTwo");

Top2.5 Configuring verbosity of output - Option::oSILENT

Within testing it depends on the specific test case which parts of a system's communication flows are of interest. To reduce complexity of testing the not interesting parts should be left off.

Transferred to our simulated objects this means: in one test case we want to track a specific subset of method calls, in another test case we want to track a different subset of method calls. To solve this requirement the test framework offers the possibility to set the not interesting methods to "silent" mode: 

TTB::TheGlobalOptions()->Set(Option::oSILENT, MySimulatedObject::CalculateTwo);
An appropriate location where to add this instruction could be the initialization code where you prepare the environment for your test case.

When during testing both methods (e.g. CalculateOne(), CalculateTwo()) of your simulated object get called then the following output will be recorded as test event: 

CalculateOne (MyObjectName)
The call to CalculateTwo is not visible here but it will be performed regularly and from the point of view of the system under test all works fine. With setting silent mode for a method only the tracking as test event and the optional generation of a sync event is switched off.

Tracking only selected methods
It is possible to set silent mode for all methods of a specific class type and then optionally activate only selected methods to be of interest for testing: 
// activate silent mode for all methods of given class type
TTB::TheGlobalOptions()->Set(Option::oSILENT,"MySimulatedObject");

// deactivate silent mode for a selected method, method will be tracked within test
TTB::TheGlobalOptions()->Set(Option::oSILENT,
    "MySimulatedObject::CalculateTwo", OptionType::NOT_ACTIVE);

Top2.6 Synchronizing asynchronous calls - Option::oSYNC

Within multithreaded environments your fake object may get called asynchronously at some point later in time. This leads to the following problem you have to solve within your test application:

Your test sequence sends some trigger to the system under test (e.g. calls some interface function) and then has to wait until the expected delayed call to your fake object occurs.

For an easy solution to this problem the test framework offers the possibility of automatically rising a synchronization event when a method of a simulated object gets called.

To activate the automatic generation of sync events you could address a single method or a whole class (i.e. all methods of the class): 

// activate sync for a specific class method
TTB::TheGlobalOptions()->Set(Option::oSYNC,"MySimulatedObject::CalculateTwo");

// activate sync for all methods of a class
TTB::TheGlobalOptions()->Set(Option::oSYNC,"MySimulatedObject");
When the synchronization is activated, a test sequence waiting for a delayed call looks like this: 
TTB_INIT_SYNC(1);
DoSomeTriggerCallToTheSystemUnderTest()
TTB_WAIT_SYNC(); // wait here until the sync event occurs

// now the call has happened and the expected
// call data can be verified
TTB_EXP("CalculateTwo in_value=13 out_result=26 (MyObjectName)")
For a detailed description of test control and automatic verification for asynchronous control flows see Multithreading, Synchronisation paralleler Abläufe and Zentralisierte Erfassung der "Testereignisse" (TestEvents).

Top2.7 Supporting (error) return values - TTB_METHOD_CALL_RET

Function signatures often have a return value signalling the execution status of the method. The return type may be a simple numeric value or a specific data struct giving more information about the failure and perhaps storing a complete error chain.

From the aspect of testing it is important to stress the SUT also with error reactions. Therefore when implementing a simulated object it always should support a failing of its methods.

When deriving your specific object class from SimulatedObjectBase you will get optional error return values (nearly) for free. In most cases you only have to

  • use macro TTB_METHOD_CALL_RET1 or TTB:METHOD_CALL_RET2 within your method implementations
  • provide a function overload TestToolBox::MakeReturnValue(MyReturnType,..) for each of your return types.

The following two examples illustrate how simple it is to simulate error behaviour:

Top2.7.1 Example: Integer return value

Assume you have two functions returning int values then you can implement: 
int MySimulatedObject::DoSomething ()
{
    return TTB_METHOD_CALL_RET1(int);
}

int MySimulatedObject::DoSomethingElse (std::string in_info)
{
    // adding some info about input params
    return TTB_METHOD_CALL_RET2(int, "in_info=" << in_info);
}
To support your return type "int" you have to implement an appropriate overload of function MakeReturnValue somewhere in your link unit: 
namespace TestToolBox
{
    std::string MakeReturnValue (
        int&                        out_retVal,
        bool                        in_simulateFailure,
        std::string const &         in_methodName, // not used here
        SimulatedObjectBase const * in_pObject)    // not used here
    {
        if (in_simulateFailure) out_retVal = -1;
        return ""; // optionally give info for test output
    };
}
When during testing the simulated methods get called they will return value 0 and write the following output to test events: 
DoSomething (MyObjectName)
DoSomething in_info=someInfo (MyObjectName)
You can activate an error return value specific for each method by writing within test script: 
TTB::TheGlobalOptions()->Set(Option::oERROR,"MySimulatedObject::DoSomething");
TTB::TheGlobalOptions()->Set(Option::oERROR,"MySimulatedObject::DoSomethingElse");
When the methods are called again they will return value -1 and write the following output to test events: 
DoSomething return error (MyObjectName)
DoSomethingElse in_info=someInfo return error (MyObjectName)

Top2.7.2 Example: Error struct return value

Assume you have a complex error struct, e.g.: 
struct SomeErrorResult
{
    SomeErrorResult () : m_errorCode (0) {}

    SomeErrorResult (
        int                 in_errorCode,
        std::string const & in_errorInfo)
        : m_errorCode (in_errorCode)
        , m_errorInfo (in_errorInfo)
    {}

    int         m_errorCode;
    std::string m_errorInfo;
};
and a method returning this type, then you can implement: 
SomeErrorResult MySimulatedObject::DoSomethingImportant ()
{
    return TTB_METHOD_CALL_RET1(SomeErrorResult);
}
To support your return type "SomeErrorResult" you have to implement an appropriate overload of function MakeReturnValue somewhere in your link unit: 
namespace TestToolBox
{
    std::string MakeReturnValue(
        SomeErrorResult&            out_retVal,
        bool                        in_simulateFailure,
        std::string const &         in_methodName,
        SimulatedObjectBase const * in_pObject)
    {
        if (in_simulateFailure)
        {
            out_retVal.m_errorCode = -1;
            out_retVal.m_errorInfo = "Simulated error in "
                // add some context info about method and object
                + in_pObject->GetClassName() + "::" + in_methodName;
        }
        else // if not yet done within default construction
        {
            out_retVal.m_errorCode = 0;
            out_retVal.m_errorInfo = "";
        }

        // to override the default error info "return error"
        std::string info;
        if (in_simulateFailure)
        {
            std::ostringstream oss;
            oss << "return error code " << out_retVal.m_errorCode;
            info = oss.str();
        }
        return info;
    };
}
When during testing the simulated method gets called it will return SomeErrorResult() and write the following output to test events: 
DoSomethingImportant (MyObjectName)
You can activate an error return value specific for your method by writing within test script: 
TTB::TheGlobalOptions()->Set(Option::oERROR,"MySimulatedObject::DoSomethingImportant");
When the method is called again it will return an error struct containing m_errorCode = -1, m_errorInfo = "Simulated error in MySimulatedObject::DoSomethingImportant " and write the following output to test events: 
DoSomethingImportant return error code -1 (MyObjectName)

Top2.7.3 Project specific macro variants

Often in SW projects only a limited number of return types is used within all SW units. To simplify testing even a bit more you can
  • create specific macro variants TTB_METHOD_CALL_RET_MYERROR, which release you from specifying the explicit return type (e.g. with additional namespaces) as macro parameter and
  • provide implementations of MakeReturnValue(MyError) e.g. within header files available for all test projects

Top2.8 Easy access to registered simulated objects

Sometimes you need access to one or more of your simulated objects, e.g. to generate a specific trigger or to make changes to some specific behaviour. Of course you can store all your simulated objects within global variables. But things may get complicated when simulated (sub)objects are created within your test run e.g. when the SUT requests one or more handling objects from one of your simulated objects. Instead of implementing a more or less complex and specific object hierarchy think of a better solution:

Idea:
Register your object by its name at a global registry. Each time you need access you can ask for the object simply by knowing its name.
The registration functionality is provided by the testing framework in form of the base class RegisteredSimulatedObject:

Registered simulated object 1

Top2.8.1 Implement a self registering class

To connect to the registration functionality for simulated objects as given by the test framework you simply have to derive your class from RegisteredSimulatedObject:

Header file: 

#include "ISomeService.h" // regular interface to be implemented
#include "TestToolBox/SimulatedObjectBase.h"

class MySimulatedObject
    : public ISomeService
    , public TestToolBox::RegisteredSimulatedObject
{
public:
    // Constructor
    MySimulatedObject (std::string const & in_objectName);

    ...
};
Within constructor you have to pass the instance name and class type of your object to the base class. This will automatically register your object: 
MySimulatedObject::MySimulatedObject(
    std::string const & in_objectName)
    : RegisteredSimulatedObject (in_objectName, TTB_CLASS_NAME)
{
    ...
}

When within testing you need access to your simulated object, you can simply write: 

TTB::TheRegisteredSimulatedObjects()->Get("ObjectA")->DoSomethingInBase();

// DoSomethingInBase is one of the methods available within base classes
// SimulatedObjectBase or RegisteredSimulatedObject.
After you have changed your object's name, you have also to adjust the access code: 
// Access object named "A" to change its name
TTB::TheRegisteredSimulatedObjects()->Get("A")->SetObjectName("DummyB");

// From now on you have to use the new name
TTB::TheRegisteredSimulatedObjects()->Get("DummyB")->DoSomething();
Last but not least: The destructor of base class RegisteredSimulatedObject automatically unregisters the object.

Top2.8.2 Using short names

If you find it acceptable for you to get more readable code by taking the (rather low) risk of specifying a using directive, you can write "using TestToolBox::ShortNames;" at the top of your test script file and whenever you need an access to a registered object you can simply write: 
SimObject("DummyB")->DoSomethingInBase();

Top2.8.3 Access members of your derived class

Sometimes you may need to access a specific member of your derived class. Of course the testing framework only can return objects of base class type "RegisteredSimulatedObject".

But with a simple downcast function 

MySimulatedObject* FindMySimulatedObject(std::string const & in_objectName)
{
    // Downcast to your specific class
    return static_cast<MySimulatedObject*>(SimObject(in_objectName));
}
you again can simply write 
FindMySimulatedObject("DummyB")->DoSomethingInDerived();

Top2.9 Configuring output format - TheFormatter()

When a method of a simulated object gets called there is a limited number of information blocks which may be relevant for your test case. Typically this is a combination of class name, object name, method name and perhaps a user supplied string containing some information about the input and output parameters.

Depending on the specific needs for your test environment and also depending on your personal aesthetic sensibilities you may want to have influence on the basic informations written to test events.

Idea:
Use a generic mechanism to format output from method calls. User should be able to select or deselect information blocks and to rearrange their order.
Following this dea the test framework offers you the possibilty to change the default output format for method calls to your fake objects. The following information blocks are available:
  • CLASS_NAME: class name, e.g. "MySimulatedObject"
  • METHOD_NAME: method name, e.g. "CalculateTwo"
  • OBJECT_NAME: object name, e.g. "MyObjectName"
  • USER_TEXT: your specific textual information, e.g. "in_value=13 out_result=26"
An appropriate location where to configure output format could be the initialization routine where you setup your environment. 
namespace TTB = TestToolBox;
namespace FI = TestToolBox::Formating;

// Define format for output of method calls
// The sequence of calls represents the sequence within output string.
// Not added info blocks will not be used for output.
FI::FormatInstruction instruction;
instruction.AddInfoBlock(FI::InfoBlock(Formating::CLASS_NAME,"","::"));
instruction.AddInfoBlock(FI::InfoBlock(Formating::METHOD_NAME,""));
instruction.AddInfoBlock(FI::InfoBlock(Formating::OBJECT_NAME," (", ")"));
instruction.AddInfoBlock(FI::InfoBlock(Formating::USER_TEXT));

TTB::TheFormatter()->SetFormatInstructionForMethodCall(instruction);

When during testing your simulated object gets called the output will now have the following format: 

MySimulatedObject::CalculateOne (MyObjectName)
MySimulatedObject::CalculateTwo (MyObjectName) in_value=5 out_result=26
Without specific configuration the same method calls would have the following format: 
CalculateOne (MyObjectName)
CalculateTwo  in_value=5 out_result=26 (MyObjectName)

Top3 Controlling behaviour by setting options

During testing there is often the need to change the behaviour of a simulated object.

Example: Within one test case a called function of a simulated object may succeed, within the next test case the same function shall fail to test error handling of the system under test.

Top3.1 Local and global (boolean) options

Each instance of a simulated object has its own set of (local) options. Furthermore there are global options stored within singleton TheGlobalOptions():

Local and global options

By calling method _IsOptionSet() the default implementation of _MethodCall() checks a couple of predefined options. General rule: if the option is not found within the set of local options then the search continues within global set of options. If the (boolean) option cannot be found within both sets it is assumed that the option is not set, i.e. its value is "false".

Within the method implementations in your derived class you may check any option via the same mechanism.

Top3.2 (Boolean) Options and scope

Top3.2.1 Option: arbitrary boolean string expression

An option is an arbitrary text which should be associated with a boolean expression.

Example: "multiplication shall fail" or "SimulateFullContainer" are good names for options in the context of testing.

There are the following predefined options which are used within the implementation of base class SimulatedObjectBase and which you should use also within your code where appropriate:

  • Option::oSILENT (= "silent:")
    control tracking of method call,
    default: false, i.e. method call is tracked. True: method is not tracked, it will not be regarded as a relevant test event.
  • Option::oOBJECT_NAME (= "objectName:")
    controls whether a tracked method call also contains the name of the called object,
    default: false, i.e. tracked method call does not contain the object's name.
  • Option::oSYNC (= "sync:")
    controls whether the method call generates a sync event which may be used for synchronizing with the test script,
    default: false, i.e. method call does not generate a sync event.
  • Option::oBLOCKING (= "blocking:")
    controls whether the method call shall block, i.e. it will not return to the caller until some other thread calls SimulatedObjectBase::_ContinueExecution(methodName).
    default: false, i.e. method call will return immediately.
  • Option::oERROR (= "error:")
    control error behaviour,
    default: false. You can use it within your specific method implementation: 
    void MySimulatedObject::CalculateTwo (int in_val, int& out_result)
{
    out_result = in_val * 2;

    // Check whether error condition is set for the scope of this method
    if (_IsError("CalculateTwo"))
    {
        out_result = -1;
    }

    TTB_METHOD_CALL1("in_value=" << in_value << " out_result=" << out_result);

    // Remarks:
    // _IsError("CalculateTwo") is short form of
    // _IsOptionSet(Option::oERROR, "CalculateTwo")
    //
    // Alternatively check for an error flag concerning all methods:
    // _IsError()  is short form of
    // _IsOptionSet(Option::oERROR, Option::ALL)
}

Top3.2.2 Scope and default search

Each option can be attached to a specific scope where it shall be valid. The scope is an arbitrary string expression. You can choose any scope names making some sense within your test application.

When thinking about method calls to a simulated object the scope where to check for some option usually has to do with the name of the called method and the class name. Within base implementation SimulatedObjectBase::_MethodCall the options are checked by using the name of the called function as the scope.

To decide whether an option is set, SimulatedObjectBase::_IsOptionSet (someOption, "MyMethodName") searches for an option entry according to the following sequence:

  • Local options defined within the object itself
    • search scope "MyMethodName"
    • search scope "all"
  • Global options defined for the whole application
    • search scope "MyClassName::MyMethodName"
    • search scope "MyClassName::all"
    • search scope "MyClassName"
    • search scope "SimulatedObjectBase"
    • search scope "MyMethodName"
    • search scope "all"
The search stops when the first entry is found. If no entry is found the option is not set, i.e. the option value is false.

Knowing this search algorithm you can choose an appropriate scope to address the reaction you need.

Top3.2.3 Examples for using scope and local/global options

Watch only method calls from a single class:
// Switch off method tracking for all objects and methods
TTB::TheGlobalOptions()->Set(Option::oSILENT);

// Selectively switch on all methods of a single class
TTB::TheGlobalOptions()->Set(Option::oSILENT,"MyDerivedClass1",
    OptionType::NOT_ACTIVE);

// Selectively switch on a single method of a single class
TTB::TheGlobalOptions()->Set(Option::oSILENT,"MyDerivedClass2::SomeMethod",
    OptionType::NOT_ACTIVE);
Switch on sync events for some methods
// Selectively switch on two methods of a single class
TTB::TheGlobalOptions()->Set(Option::oSYNC,"MyDerivedClass::SomeMethod1");
TTB::TheGlobalOptions()->Set(Option::oSYNC,"MyDerivedClass::SomeMethod2");
Activate error reaction for a single instance
// assume you have an array of simulated objects
std::vector mySimObjects = CreateMyObjects();

// The 3rd element shall return an error when getting called
mySimObjects[2]->_SetOption(Option::oERROR,"SomeMethod");

Top3.2.4 Dynamic behaviour for options

Motivation
Basically a boolean option can have only two states: it is active or not. But when thinking of SW tests there are often cascades of repeated method calls. Within your test case you may be interested on a situation similar to one of the following:
  • How does the system behave when the first 3 method calls succeed and from the 4th method call on it constantly fails?
  • How does the sysytem behave when only the second method call fails?
  • I want to check the internal state of the system just when the 3rd call to a simulated object has happened
Using OptionType and NumRemainingCalls
When setting an option via Option::Set() you can specify besides the option text and the scope two additional parameters allowing control of the dynamic behaviour of an option: 
class Option
{
    ...

    void Set (
        std::string const & in_option,
        std::string const & in_scope      = Option::ALL,
        OptionType::Enum    in_optionType =  OptionType::PERMANENTLY_ACTIVE,
        int                 in_numRemainingCalls = 0) const;

    ...
};
The OptionType has one of the following meanings:
  • NOT_ACTIVE
    The option is switched off.
  • PERMANENTLY_ACTIVE
    The option is switched on.
  • ACTIVE_ONCE
    The option is switched on, but only for the next call of Option::IsSet(). For all further checks it will have state NOT_ACTIVE.
  • AFTER_N_CALLS_PERMANENTLY_ACTIVE
    The option is switched off, but will automatically set state PERMANENTLY_ACTIVE when Option::IsSet() is called N times
  • AFTER_N_CALLS_ACTIVE_ONCE
    The option is switched off, but will automatically return true for the Nth call of Option::IsSet(). For all further calls it will have state NOT_ACTIVE.

Top3.3 Using options of arbitrary type

The sections above treated how to change behaviour with use of a predefined set of boolean options. You could also introduce your own boolean options. But sometimes you may need a numerical type rather than a boolean type. Here is the solution:

Tip:
Class Environment of test framework supports command line options of any streamable type. You can also use this feature to introduce new options or change existing command line options at runtime.
Calling your test executable with command line params
MyTestExecutable.exe -SomeCounter 17
Using option value within test script
To access the specific type of the command line param you have to use template syntax: 
int counter = TTB::TheEnvironment()->GetCommandLineOptionVal<int>("-SomeCounter");

// change existing option value
++counter;

// store it again to make it available to all interested locations
TTB::TheEnvironment()->SetCommandLineOptionVal("-SomeCounter", counter);
Introduce new options
Also if there is no command line param given, you can introduce any param at runtime: 
// define new option of type double
TTB::TheEnvironment()->SetCommandLineOptionVal("-SomeDoubleValue", 3.14);

Top4 Multithreading advanced: blocking method calls

Motivation
Assume your system under test asynchronously generates a sequence of method calls to one or more simulated objects within your test environment. You may want to stop somewhere in the middle of this sequence for one of the following reasons:
  • check the internal or external state of your system under test while it is processing something
  • check how the system reacts on further triggers from outside while still processing (e.g. simulate a special situation from a bugreport)
  • simply have a chance to change some settings within your test environment (e.g. set/reset error behaviour of simulated objects) before continuing
Solution principle
The testing framework's Option::oBLOCKING allows to block within a specific method:

blocking call

Top4.1 Example: Relying on standard implementation (TTB_METHOD_CALL)

Assume that after some trigger the system under test generates a call sequence which can be seen in the following test script: 
TTB_INIT_SYNC(5);
SendSometriggerToTheSystemUnderTest();
TTB_WAIT_SYNC()

TTB_EXP("CalculateSomething in_val=3 outVal=6 (MySimObject1)");
TTB_EXP("CalculateSomething in_val=4 outVal=8 (MySimObject1)");
TTB_EXP("DoSomethingElse (MySimObject1)");
TTB_EXP("CalculateSomething in_val=5 outVal=10 (MySimObject1)");
TTB_EXP("CalculateSomething in_val=6 outVal=12 (MySimObject1)");
TTB_EXP("DoSomethingElse (MySimObject2)");
To be able to hold the execution on the 3rd call of "CalculateSomething" the standard implementation of the method is sufficient: 
void MySimulatedObject::CalculateSomething (int in_val, int& out_val)
{
    out_val = in_val * 2;
    TTB_METHOD_CALL1("in_value=" << in_val << " out_val=" << out_val);
}
To activate blocking you have to write within test script: 
mySimObject1->_SetOption(Option::oBLOCKING, "CalculateSomething",
    AFTER_N_CALLS_ACTIVE_ONCE, 3);

TTB_INIT_SYNC(4);
SendSometriggerToTheSystemUnderTest();
TTB_WAIT_SYNC()

TTB_EXP("CalculateSomething in_val=3 outVal=6 (MySimObject1)");
TTB_EXP("CalculateSomething in_val=4 outVal=8 (MySimObject1)");
TTB_EXP("DoSomethingElse (MySimObject1)");
TTB_EXP("CalculateSomething-Start in_val=5 outVal=10 (MySimObject1)");

// Now do something while the execution is halted
...

// Continue execution
TTB_INIT_SYNC(3);
mySimObject1->_ContinueExecution();
TTB_WAIT_SYNC()

TTB_EXP("CalculateSomething-Stop 
TTB_EXP("CalculateSomething in_val=6 outVal=12 (MySimObject1)");
TTB_EXP("DoSomethingElse (MySimObject2)");
All checking, halting and continuing of execution is done within base class implementation.

Remark: Importance of asynchronous calls
A halt of execution requires that the calls are executed asynchronously. Otherwise the test script itself would be blocked and there would be no chance to call continue() or do some other things. In the example above it was assumed that the system under test itself executes all actions asynchronously.

But when this is not the case you could delegate the trigger call within your test environment to a separate thread. Then the main thread of your test script would be free for calling continue().

Top4.2 Example: Lazy evaluation of conditions

When a blocked method call continues its execution there may regularly be the case that the returned result shall consider some conditions which may have changed while the call was halted. The base implementation supports also this case, but the client has to be prepared to set the return values in the right moment: 
void MySimulatedObject::CalculateSomething (int in_val, int& out_val)
{
    out_val = in_val * 2;

    TTB_METHOD_CALL2 (

        // write input params when function is called
        "in_value=" << in_val,

        // embedded lambda function,
        // gets called just before function returns
        std::ostringstream oss;
        if (_IsError("CalculateSomething "))
        {
            out_val = -1;
        }
        oss << "out_value=" << out_value;
        return oss.str();
    );
}
Instead of using macro TTB_METHOD_CALL2 you could also use the underlying function _MethodCall: 
void MySimulatedObject::CalculateSomething (int in_val, int& out_val)
{
    out_val = in_val * 2;

    _MethodCall(__FUNCTION__,

        // write input params when function is called
        TTB_STRING_S("in_value=" << in_val),

        // embedded lambda function,
        // gets called just before function returns
       ([&](void) -> std::string
       {
           std::ostringstream oss;
           if (_IsError("CalculateSomething "))
           {
              out_val = -1;
           }
           oss << "out_value=" << out_value;
           return oss.str();
       }));
}
The base implementation considers params 2 and 3 of function _MethodCall() for generating output. Param 2 is a simple string which is written to test events when the function is entered. Param 3 is a lambda function which also returns a string for output. When the function call is not blocked both outputs are presented in a single line. In the case of blocking the lambda function gets evaluated just before the function is left and the returned output is written to test events as a separate line.

Now you could write the following test script: 

mySimObject1->_SetOption(Option::oBLOCKING, "CalculateSomething",
    AFTER_N_CALLS_ACTIVE_ONCE, 3);

TTB_INIT_SYNC(4);
SendSometriggerToTheSystemUnderTest();
TTB_WAIT_SYNC()

TTB_EXP("CalculateSomething in_val=3 outVal=6 (MySimObject1)");
TTB_EXP("CalculateSomething in_val=4 outVal=8 (MySimObject1)");
TTB_EXP("DoSomethingElse (MySimObject1)");
TTB_EXP("CalculateSomething-Start in_val=5 (MySimObject1)");

// While the execution is halted decide that the blocked call shall return
// an error value and the next call shall automatically succeed again.
mySimObject->_SetOption(Option::oERROR, "CalculateSomething", ACTIVE_ONCE);

// Continue execution
TTB_INIT_SYNC(3);
mySimObject1->_ContinueExecution();
TTB_WAIT_SYNC()

TTB_EXP("CalculateSomething-Stop outVal=-1 
TTB_EXP("CalculateSomething in_val=6 outVal=12 (MySimObject1)");
TTB_EXP("DoSomethingElse (MySimObject2)");

Top5 Internal design

Top5.1 Design decisions

  • function names begin with underscore "_"
    SimulatedObjectBase is a base class for derived classes which are intended to realize application specific (interface) functions used by a real sw system. The underscore signals "this function is part of the test framework".
  • all base methods are const
    At least some of the application specific functions to be implemented within the derived class have to be const (because the interface function which they implement is const). To be able to use the default implementation of _MethodCall() also within such a const function without the need of adding some ugly const cast all base methods were declared as const. Nevertheless the base methods have to change class attributes. To make this possible all base attributes were declared mutable.

Top5.2 Class structure

Simulated object 2

Top5.3 Registering simulated objects

Top5.3.1 The problem of finding temporary simulated objects

In complexer sw architectures there may arouse the situation where instances of simulated classes have to be dynamically created on a request by the SUT. From the perspective of testing it often is required to trigger those objects at specific points within the test sequence (e.g. the SUT may have registered a callback at those objects and now waits to be called). The main problem in this situation is: how can the simulated object be reached by the test environment?

Registered simulated object 2

Of course there is a number of specific solutions, you can implement:

  • you can implement specific data containers to store those temporary objects. Then you have to implement proper acess functions possibly for a whole object hierarchy to give easy access for the test script.
  • for some standard situation you can configure behaviour via singleton TheGlobalOptions(). This mainly concerns configuration of sync events, silent mode and error returns. For more details see Local and global options.
The next section tries to propose a more general solution which can save you a lot of programming effort.

Top5.3.2 Automatic registration of simulated objects

Idea:
Derive your simulated object from a base class (RegisteredSimulatedObject) which automatically registers itself at a singleton instance, which is globally available.
Then you can solve the original problem from the preceeding section:

Registered simulated object 3

Choosing object names
For safely accessing registered objects you have to provide an unique object name to each instance of your class SimObjectB. A first step could be to use some hard coded name. If there is only a single instance of B within each test case then you are done.

If there are multiple instances you can try to implement a counter and add the counter value to your hard coded object name.

Furthermore you could change the name of an registered object after its creation by calling its base method SimulatedObjectBase::SetObjectName(). This will also update the object's registration. It will always be reachable under its current name.

Last modified: 07.10.2018 17:37:30, © Gerald Fahrnholz
Impressum | Home | Top