Overview:
List available test cases and select for execution
Typically you will use TestExecWindow to get an overview of available test cases contained within a test application. By selecting a single test suite or a single test case and then pushing "Run selected" TestExecWindow will automatically set appropriate command line parameters and start the corresponding test case.
It depends on the structure of your specific test application if test suites or test cases can be executed independently from each other. If this is not the case you should think about improving your test application. A good advise could be to ensure that at least each test suite could be executed alone.
TestExecWindow supports you to check which test suites or test cases can already run independently. By pushing "Run each" TestExecWindow will start your test executable separately for each item in the list of test suites or test cases. The result of each call to your test application is displayed within the log pane together with the used command line params.
Remark:
If you simply want to execute all test cases as quick as possible it is recommended to use button "Run" at the top of the window. In this case your test application will be started only once to execute all tests.
Searching for errors, extending test cases
If you want to extend your test cases or you want to search the reason for some test failure the following features may be useful:
- Protocol file
Pushing "Out" opens the protocol file of your test application within Visual Studio. Usually you could find more detailed information in this file which may allow a better understanding of the error situation. It depends on your specific test application whether there exists a protocol file. TestExecWindow assumes a protocol file with the name "YourTestApp.out" besides the test executable "YourTestApp.exe".
- Debugging
When searching for an error it may be useful to start test execution within debugger. You can simply switch debugging on and off by selecting / deselecting checkbox "Debug".
Remark: The command line params used to execute your test within debugger are stored within the project settings and will not be restored to the original values. You may have to reset them manually when running the debugger outside of TestExecWindow.
- Src - jump to source code
Pushing "Src" immediately opens the selected test function or test group within your source code. This may be useful for inspecting the test instructions when looking for a test error or for adding a new test case before or after the selected one.
- Refresh
If you have just added a new test case press "Refresh" to trigger a refresh of the list of available test cases. This may also be necessary when you have opened TestExecWindow after you have already loaded your test project. TestExecWindow currentlly listens only to notifications about a changed startup project or a change of Debug/Release. If TestExecWindow is not opened it cannot listen to such events.
Batch testing and automatic shutdown
After some time or when you are working within a team there may be a greater number of test applications and test cases. It is a good idea to execute all unit tests within the nightly or weekly build you may have introduced. You should care for a sufficient level of test output or a detailed test protocol to have enough information for later analysis of possible errors.
TestExecWin also offers the possibility to execute an arbitrary set of test applications in batch mode (for details see Batch file support). This feature may be useful in the following situations:
- execute a specific subset of tests with one button click
As a starting point for your batch you may use "Import from solution" to get a complete list of test executables from your test solution.
- verify that all your test suites can be executed alone
Use "Import test suites from startup project" to get a list of separate calls to your startup project with appropriate start args to select each test suite.
verify that all your test applications are free of memory leaks
Ensure that within detailed settings "leak check" is activated. The status of each test execution is written to log pane and has one of the following values:
- OK:
correct test result, no memory leak detected
Warning: if your test application is not configured to write memory leaks to stdout or if you have activated "no leak check" within detailed settings then you may get this status despite of having memory leaks!
- FAILED:
there are functional errors (e.g. a BOOST-Check has failed), in this case the check for memory leaks was skipped
- FAILED (MEM_LEAKS):
correct test result but memory leaks were detected
Precondition:
Your test application is configured to write memory leak messages to stdout when program terminates (C runtime supports leak check, see for keywords "_CrtSetDbgFlag", "_CRTDBG_LEAK_CHECK_DF").
- automatic shutdown and log file
Before leaving your office you can start a test sequence. If you activate checkbox "Shutdown" your computer will be switched off when all tests have terminated. The next day you can find the test results from log pane within a protocol file with the name "TestExecWin.[InfoAboutExecutedTests].[TimeStamp].Log.txt". The protocol file is located within the target directory of your startup project. If there was no startup project selected at time of execution (via batch feature) you should check your temporary directory as defined via environment variables TMP, TEMP or USERPROFILE.
Ignoring memory leaks
Sometimes you may want to extend or repair functional test cases and you don't want to solve memory leaks also occuring during testing. Possibly circumstances:
- a new component of someone else (which is part of your test environment) has leaks and he will repair them later.
- you want to focus on functionality and you currently have no time to analyze leaks
But the leaks may cause the following problems when executing test cases:
- at end of your test app a big amount of leak messages is written to std::out and you have trouble finding the test results you are interested in
- the writing of leak messages consumes a remarkable amount of time and disturbs your development of test cases
You must understand that the suppressing of memory leak detection has two steps:
- Step 1: Control your test app to avoid printing leak messages to std::out
- For BOOST test apps
set command line argument "--detect_memory_leak=0"
- For TTB test apps without BOOST
set command line argument "-checkForMemLeaks OFF"
- For all other test apps:
Memory leak messages usually are created by C runtime support for leak check. You have to change the correspondig settings within your test app (search for keywords "_CrtSetDbgFlag", "_CRTDBG_LEAK_CHECK_DF").
- Step 2: Control settings for Visual Studio extension "TestExecWin"
- Setting option "no leak check" (see More settings - defaults args and configuration of layout) will stop analyzing the leak check messages within std::out. A test app with error free test cases but generating leak messages will be regarded as successful (signalled with green color within tool window)
- The option "no leak check" has no influence on the generation of leak messages itself. This has to be done within the test app itself or within its command line params (see above)
- Remark: If you have switched off generation of leak messages for your test app then there is no need to switch off option "leak check" of extension "TestExecWin"
1.8.17