CamiTK testing framework technical overview

CamiTK provides some tools to test your code at different granularity levels. In this wiki page you will find information about our testing process, what we test and how you can use our already defined tests.

Introduction

Testing a software code is always a good practice to make sure that the code behave accordingly to the specification, to detect bugs and errors and regression. It can also be very helpful to check that code integration is operational. Several types of tests exist depending on the granularity level of the tests. There are classically three categories of tests:

• unit testing
• functional testing
• integration testing

CamiTK is mostly written in C++ and is cross-platform (Linux, Windows and MacOS). Keep this in mind when reading this document.

For each type of test, tests are built following the same principles. In the next section, this document present a (very limited and simple) explanation about cross-platform testing C++ code with CMake (please feel free to skip it). Next, it shows how testing is done in the CamiTK framework and describe the automatic test framework provided by CamiTK (and what are the plans to improve it). Finally, the last section explains the basic of adding new tests In CamiTK.

Please read to the CamiTK automatic tests framework for a practical guide on how to automatically generate tests for your extensions.

Cross-platform testing C++ code using CMake

To create any type of test (unit, functional …), three basic steps are always followed:

• Implement the C++ code that will test your C++ class method / application (you may use some dedicated tools like Qt testing framework or any other cross-platform testing framework)
• Configure your project to create some cross-platform test targets that run the tests on all the platforms.
• Check the test results.

C++ code to test your class / program

Assuming we have some C++ code to test gathered In a library (or extension), we have to basically create a C++ application (using a specific framework or a home made code) in order to:

• Load the library containing the C++ classes to tests.
• Do some tests on the class methods
• Show the test results (on standard output or in a file)

Cross-platform independent test definition and configuration

Once the test C++ application is built, it should be ran each time a modification is done on the tested code. Using more than one plateform (i.e., OS + compiler) generally makes a more stable code, therefore testing should be a “natural” (automatic) stage after configuration and build.

CMake comes with a specific module to add, configure, and run tests. Using CMake makes it possible to define and configure tests once for all platforms. To define and configure tests using CMake, two specific CMake macros should be used:

• enable_testing() to enable the testing functionality on our project
• add_test(...) to create a new test using the testing application

More specifically:

• The test executions are done using CTest, which manages which tests are run
• The test results can be sent to a dashboard using CDash

Check the test results

Once the tests are ran, the results should be sent to dashboard where the programmer can analyse the results (i.e., check which test pass, which ones fail). This is crucial to identify problems and potential bug.

Another step is to check the test coverage, i.e., how much of the actual code lines are checked during the tests. This is crucial to identify where your test code can be improved in order to check more things.

Overview of the CamiTK testing framework

CamiTK comes with a specific testing framework built on top of the CMake/CTest framework. This section describes the main level of tests that are used In the CamiTK Community Edition project.

Unit testing

Unit testing consider testing the lowest parts of the code, such as the different methods of your class.

We recommend to use QTest, a specialized unit testing framework that can handle QObject very well and take advantage of the Qt meta property system. The CamiTK team is working on providing an easy way to use QTest in CamiTK.

Functional testing

Functional testing involves testing the main functionality of a software, i.e., test the code at a higher granularity level than unit testing.

In CamiTK, we develop some automatic functional tests for component and action extensions. Those tests are automatically configured using the CamiTK CMake macros and are ran on our continuous integration system using gitlab. They are called “automatic” tests as they do not focus on a specific code, but take the advantage of the CamiTK API to automatically check the main expected behavior of component and action extensions.

Integration testing

In the CamiTK framework, we consider that integration testing consists of:

• Checking that CamiTK extensions can be loaded, can work together and can be used in a specific context. At the moment, this relies on the camitk-config application. The test uses a bash script to check the CamiTK Community Edition version number and verify that the expected number of extension for this version can be found and loaded in the application.
• Checking that the code generator can automatically creates valid C++ code, that can be configured, built and integrated in a specific context (mimicking the CEP developer context). At the moment, this relies on the camitk-cepgenerator application. The test uses a bash script and starts from document written in the cepcoreschema XML language to apply the whole build chain: generate the code using the cepgenerator library and the camitk-cepgenerator application, configure the code using CMake and the CamiTK macros, build the code using the specific compiler available on the platform, and run the camitk-config application to check that the extensions can be loaded and available in any CamiTK application
• Checking that different actions can be executed in a pipeline. At the moment, this relies on the camitk-actionstatemachine application. The developer can create action pipelines writing an action state machine XML description language. During the test phase, the pipeline is ran automatically and the output files (i.e., the component that are saved to file during the pipeline execution) can be compared to expected files in order to verify that everything went according to plan. This can also be used to profile the pipeline in terms of execution time or memory consumption.

CamiTK test results and test coverage dashboard

All the test are ran each time a modification is detected on the git source code management system (on any branch). All tests are reported on the CamiTK dashboard.

Not only the test results are visible, but CamiTK also offers a way to check the test code coverage for the Community Edition or an individual extension project

CamiTK testing framework

The CamiTK testing framework is defined as a layer on top of CMake/CTest, and provides five different type of tests:

1. Automatic test for action and component extensions
2. Automatic test using a pipeline of actions
3. A set of specific CMake macros that help defining custom tests and test coverage
4. A specific bash test to check the integration of action and component extensions on a given CamiTK Community Edition version
5. A specific bash test to check that CamiTK is properly installed and configured so everything is ready for starting development of new extension projects (including C++ code generation from the wizard, configuration, and integration)

The following sections describes these five different types of tests.

Automatic test of action and component extensions

As CamiTK defines an API for action and component extensions, this API can be used to check the default/expected basic behaviour of component and action extensions. In order to do so, the CamiTK testing framework provide two extra C++ applications: camitk-testactions and camitk-testcomponents:

• camitk-testactions provides one level of automatic test for a given action extension.
• camitk-testcomponents offers three different levels of automatic test for a given component extension.

These tests can be configured and run for any CEP thanks to the camitk_extension CMake macros (see below). They are used in the CamiTK Community Edition multi CEP and the results are visible on our dashboard, but they can also be used in any other CEP. Moreover it is possible to test other CEP component extensions with already installed action that were not aware of these component extensions when they were initially built and tested (this is called the additional component extension automatic tests).

Testing action extensions

To test each action provided in the CamiTK framework, we insure that calling it on the proper components it has been defined on, works correctly. The camitk-testactions application has five main steps:

1. Load all the available CamiTK components the action can work on
2. Load the action extension itself
3. Open a component file and set it as the action input component
4. Call the action (i.e., run the action’s apply() method)
5. Check the apply() method return status. The test passes if the apply() methods returns Action::ApplyStatus::SUCCESS or Action::ApplyStatus::ABORT (in this case it the developer responsability ot check that the ABORT status is really what is expected). The test fails otherwise.

For technical details and implementation, please check:

• CamiTKExtension.cmake (in sdk/cmake/modules/macros/camitk) the script that creates the tests thanks to the camitk_init_test and camitk_add_test macro (resp. defined in CamiTKInitTest.cmake and CamiTKAddTest.cmake in sdk/cmake/modules/macros/camitk/test, see the dedicated section below)
• the C++ source code for testactions (in sdk/applications/testactions)

For example, check any actions in the SDK.

Testing component extensions

To test each component in the CamiTK framework, we insure that opening a file associated with this component and (when possible) saving it work. To do so, the action-testcomponents application goes trough four steps for each available (or specified) files in the component’s testdata subdirectory:

1. Load the component extension
2. Open the given file (test level 1)
3. Save the file using a different name (optional test level 2)
4. Check that the saved file is identical to the original file (optional test level 3)

The level of the test depends on the AUTO_TEST_LEVEL option specified in the camitk_extension macro.

For technical details and implementation, please check:

• CamiTKExtension.cmake (in sdk/cmake/modules/macros/camitk) the script that creates the tests thanks to the camitk_init_test and camitk_add_test macro (resp. defined in CamiTKInitTest.cmake and CamiTKAddTest.cmake in sdk/cmake/modules/macros/camitk/test, see the dedicated section below)
• the C++ source code for testactions (in sdk/applications/testactions)

For example, check any components in the SDK.

Testing extensions in CamiTK Community Edition

All the extensions defined in the CamiTK Community Edition have associated tests.

Automatic test using a pipeline of actions

To apply a pipeline of actions, the CamiTK action state machine application is used to perform all the actions defined in an specific XML document based on the State Chart XML with some specific additions to describe the CamiTK actions to run. The pipeline runs the actions one after the other automatically (i.e., it simulates a user clicking on the “Next” button of the action state machine). During the XML described pipeline, input files can be opened (using the Open File action) and output files can be saved (using the Save action), to save for instance the result of an action. Any file saved during the pipeline execution is going to be compared to the expected output file. The test fails if any of the output files differ from the expected output files.

In order to automatically add a test, you need to add the following line at the end of the action or component extension CMakeLists.txt:

This process is generalized thanks to some normalization: all the required following files should be available in a integration-testdata subdirectory:

• An CamiTK SCXML document input-asm.scxml
• Any number of input-#.xxx input files
• Any number of output-#.xxx output files (this are the expected output files)

If the ENABLE_INTEGRATION_TEST option is added in camitk_extension macro and if the integration-testdata subdirectory is detected, then the test is automatically created. The camitk_extension macro uses the CamiTKAddIntegrationTest.cmake prepares the corresponding test temporary directory in the build dir by copying all the required files.

When the test is ran, it executes the CamiTKTestActionMachine.cmake script (in sdk/cmake/modules/macros/camitk/test) to run the action state machine application using the autonext mode (see below), and then compare all the files saved as output-#.xxx to the files present in the integration-testdata.

For example, check the itkfilters action extension in the imaging CEP provided in the CamiTK Community Edition, and the image/reconstruction action extension in the SDK.

Set of specific CMake macros for testing

CamiTK testing framework provides three specific CMake macros in order to more efficiently configure tests inside the CamiTK environment:

• camitk_init_test(...) helps to initialize all the common CMake configuration for all the test of the current CMake project.
• camitk_add_test(...) adds a new test for the current CamiTK project. This macro encapsulates the default add_test(...) function provided by CMake.
• setup_target_for_coverage(...), entirely based on Lars Bilke macro, simplifies the generation of test coverage report.

These macros can be called directly In the CMakeLists.txt file. This allows any developer to manually add new tests to a CamiTK extension, library or application.

In the CamiTK Community Edition project each extension are tested using camitk_init_test(...) and camitk_add_test(...) macros. These macros are used for generating automatic tests in the camitk_extension(...) macro. In the CamiTK Community Edition project, most of the applications are also tested by using camitk_init_test(...) and camitk_add_test(...) in the corresponding CMakeLists.txt. This normalize the name of all the tests as well as facilitate the declaration of test outcome expectancy (i.e., what should be expected from the test outcome).

Testing the extension integration

Once a component or action extension is built, it is important to check that it can be loaded and used In a CamiTK application. CamiTK provides the camitk-config application. For non-installed extension, it should be run from the top directory of an extension repository (e.g, the build dir of your extension). camitk-config provides all the information you need to check that an extension can be loaded (i.e., that all the shared library required by the extension can be loaded and understood by the CamiTK extension management system).

The CamiTK Community Edition project tests if the valid number of expected action and component extensions are found (this depends on the CamiTK CE version number). This test is defined in a bash script (see sdk/applications/config/testing/config-test.sh), but you can directly use camitk-config in any CEP to check that the number of extensions provided by your project is correct as well.

Testing the availability of the CamiTK development framework

Once installed, CamiTK should be ready to be used to develop a CEP.

The CamiTK Community Edition project tests that the CamiTK development framework is ready to use doing the following steps (all described in bash script, see sdk/applications/cepgenerator/testing/cepgenerator-test.sh):

1. Test that the C++ code generation from a cep core schema XML is possible: this tests the availability of the camitk-cepgenerator application.
2. Test that the generated code can be configured: this tests that the CamiTK CMake macros and environment is correctly installed and accessible
3. Test that the generated code can be compiled: this tests that the core library and other dependencies are available and linkable with all the C++ compilation tool chain
4. Test that the generated code can be loaded and integrated: this uses camitk-config to test that the newly compiled extensions are immediately available In a CamiTK application

Note that many different cep core schema XML examples are tested. All the XML examples are included inline in the bash script.

You can also create your own example from a tweaked core schema XML if your CEP provide a component or action extension than can be used itself by other developers In external CEP.

Disabling tests depending on the environment

Since CamiTK 5.0

In some specific cases, some automatic tests have to be disabled, for instance when a test depends on a specific platform or OS or version of a library and you know, by design that it can only fail if some environment requirements are not met.

There are two ways to disable some of the automatically generated tests:

• disable test unconditionally
• set specific requirements to run tests (if the requirements are not met, the tests will be disabled)

Check the automatic test documentation for more information

CamiTK Community Edition Code Coverage

You can easily generate and check a test coverage report for the CamiTK Community Edition or your own extension.

There are two requirements to generate the test coverage report:

1. you need to use the GNU cxx compiler
2. you need to have gcov and lcov installed (apt-get install is your friend!)

When configuring the CamiTK Community Edition project, you need to enable the CAMITK_TEST_COVERAGE option.

Once the CamiTK Community Edition project is built, you can run the camitk-test-coverage target:

make camitk-test-coverage

It will run all the available tests using ctest and then compute the test coverage and generated an html report. The reports will be available for visualisation in ${CMAKE_BUILD_DIR}/camitk-test-coverage/index.html

Generate test coverage for a non-CamiTK project

This section details how to add a test coverage report to any non-CamiTK based project. The same methodology is used in CamiTK to generate the test coverage report of the open-source project as well as the test coverage of any cep. If you are interested to know how it is done or to do the same for another project based on CMake but without using the CamiTK environment, this section documents the usage of macro setup_target_for_coverage.

Principles

The macro setup_target_for_coverage is defined and allow you to run the target and build the lcov report. This macro is entirely based on Lars Bilke macro, a copy of which is available in sdk/cmake/modules/macros/CodeCoverage.cmake.

Usage:

setup_target_for_coverage(NAME target-name
                          EXECUTABLE command-to-execute-with-arguments)

This will create a new target called``targetnamewhich will call command-to-execute-with-arguments, check which lines were covered by this execution and then will generate alcovhtml report in the target-name subdirectory of the build directory (as defined by${PROJECT_BINARY_DIR}`).

Example for CamiTK Community Edition test coverage

This section describes how we applied the principles described in the previous section in order to get a test coverage report in CamiTK.

If the CAMITK_COVERAGE option is enabled, the following line will be executed during configuration:

setup_target_for_coverage(NAME camitk-test-coverage
                          EXECUTABLE ctest --parallel 9)

Which will create a specific camitk-test-coverage target, that when run will launch all the test known by ctest in parallel and generate a report in ${PROJECT_BINARY_DIR}/camitk-test-coverage