The CamiTK Automatic Test framework

(Since: CamiTK 4.1)

Introduction

Testing can be a time consuming task, this is why CamiTK comes with automatic tests tools directly implemented in the CamiTK framework. Every action and component extensions of the CamiTK opensource framework and of any other CEP can be automatically tested thanks to the CamiTK testing framework.

It is very simple to set up a testing environment for the development of your extension, this page details the different steps and possibilities.

The CamiTK automatic testing module is built with three Kitware tools:

• CMake
• CTest
• CDash

CamiTK automatic test framework overview

Based on the common API for Component and Action extension, the CamiTK testing framework can automatically test some aspects of any extension. There is also specific way to disable some of the automatically generated tests for special cases (e.g., when a test depends on a specific platform or OS or version of a library).

You can refer to the technical overview whitepaper about testing in CamiTK to learn how testing is done in the CamiTK framework and in CamiTK Community Edition.

Components tests

For each component extension, the test procedure has three possible level from level 1 (basic testing) to level 3 (most advanced testing so far). By default the component extension is tested at level 3. If for some (good) reason you don’t want to run the test at level 3, you can downgrade the level to level 2 or level 1.

Testing a component extension requires some data. The files to test are expected to be in the testdata subdirectory of a component extension.

Level 1

This level performs two steps:

• Load the component extension
• Open a file

It checks that the component extension can be loaded by a CamiTK application and that a test file can be opened.

Level 2

Level 2 is the same as level 1, but it adds an extra step:

• Save the loaded component to another file

It checks that the component can be saved. This level can only be run if the component extension implements the save(..) method.

Level 3

Level 3 is the same as level 2, but it adds one other extra step:

• Make a diff between the input file and the output file.

It checks that the input and output data are exactly the same.

Actions tests

For each action in the action extension, the action is tested using a test file. Basically the test consists in calling the apply() method of the action on the loaded component. The test pass if the apply() method returns SUCCESS or ABORTED and fails otherwise.

Testing an action extension requires some data. The files to test are expected to be in the testdata build directory (i.e., the files used to tests action should be present in one of the testdata directory of one of the component extensions).

At the time of writing, there is only one level of test for actions, it consists in the following steps:

• Load all the component extensions
• Open a test file (should be found in one of the component extensions testdata directory)
• Run the action on the instantiated component
• Check the action status

Pipelined integration tests

Since CamiTK 4.1

Since CamiTK 4.1, you can also define a pipeline of actions to perform using the automatic integration test framework.

In this case, the CamiTK action state machine application is used to perform the actions defined in your pipeline one after the other automatically. Any file saved during the pipeline execution is going to be compared to the expected file and the test fails if any of the output files differ from the expected output files.

For instance, you can define a pipeline to:

• open an medical image from input-1.mha
• perform one filter
• save the result to output-1.mha
• perform a mesh 3D reconstruction
• save the result to output-2.vtk

When the test is executed all these actions are performed automatically one after the other and the resulting output-1.mha and output-2.vtk are compared to expected files. The test succeed if the output files are equals to the expected output files.

You can record a first version of your pipeline using the “Save History” feature of camitk-imp. You will have to edit the resulting SCXML file in order to format the names of the input and output files. The SCXML, input and output files should be placed in a integration-test subfolder of either your component or action extension.

Step-by-step: How to automatically test your extension

You can easily add one or all of the following tests to your extension:

• automatic test of your extension (as described at step #2)
• additional test for existing actions that are already installed (as described at step #3)
• pipelined integration tests (as described at step #4)

Step 1: Enable the CamiTK testing framework

The first thing to do is to tell CMake that for this project you want to enable your test. In the top level CMakeLists.txt of your project (if you’re working on a CamiTK Extenstion Project, open the top level CMakeLists.txt file and add the option ENABLE_TESTING to the camitk_extension_project(...) parameters:

camitk_extension_project(...
                         ENABLE_TESTING
)

Step 2: Enable automatic test of your extension

In the CMakeLists.txt of your component or action extension, add the ENABLE_AUTO_TEST option to automatically let the CamiTK test framework create some test. Tests will be automatically generated for your extension.

camitk_extension(COMPONENT_EXTENSION
                 CEP_NAME MYCEP
                 DESCRIPTION "my extension description"
                 ENABLE_AUTO_TEST
                 [TEST_FILES file1 file2...]
                 [AUTO_TEST_LEVEL level] # only for component extension
)

As you can see above, there are two other option you can set to match specific requirements for your test:

• the TEST_FILES option
• the AUTO_TEST_LEVEL option (for component extension)

The option TEST_FILES let you specify the list of files to use for testing. If provided, only the filenames are required (not the absolute paths), each of them should be found in the testdata build dir. If no test files are provided then:

• For component extension: all the files in the component testdata subdirectory are used by default
• For action extension: all the files available in the testdata build dir (i.e., all the test files found in all the component testdata subdirectories) are used by default (note that it might be a lot of files!).

For component extension, the parameter AUTO_TEST_LEVEL let you specify the level of automatic test to perform (if not specified, the default is 3, see also above):

• Level 1: load the component extension, open and close a test component file
• Level 2: same as level 1, but save the component before closing it.
• Level 3 (default): same as level 2, but also compare the saved component to the input.

Step 3: Define additional test for existing actions

In some cases, it can be useful to test your component with actions that are already installed (e.g., with an action provided by the CamiTK Community Edition).

For instance: if you develop a new component extension that derived from ImageComponent, an additional test can be automatically generated by the CamiTK test framework in order to check that the existing ITK Otsu filter action (provided by the CamiTK Community Edition) works on your new component.

The macro camitk_additional_action_test is here to do just that: it creates an additional level 1 test for an existing (installed) CamiTK action using test data from the current component extension.

This macro can be used to check the quality of the integration level of a new component extension defined within a CEP introduces. It will take new test files and run the defined action. If the new component class derives from an existing (installed) component class for which there are existing (installed) actions, these actions should indeed works using the new derived component class.

For example: if a CEP introduces a new MeshComponent derived class, then all the actions that works on MeshComponent should also work on the new component.

To define an additional test for an existing action, you only need to modify the component extension CMakeLists.txt, and add one line just after the camitk_extension(COMPONENT_EXTENSION ...) macro, at the end of the CMakeLists.txt:

camitk_additional_action_test(ACTION_EXTENSIONS action1 action2 ...
                              [TEST_FILES  file1 file2 ...]
                              [TEST_SUFFIX name]
)

Where:

• ACTION_EXTENSIONS is the list of the action extension that will also be tested with the given test files. Only the action extension name (not the full library/dll name nor the path) should be provided. This macro checks the existence of the extension library/dll in the following directories (in this order): build dir; user install dir and global install dir. The first action that is found is used for the tests.
• TEST_FILES is an optional argument. It can specify the list of files to use for testing. If not set, then all the files in the testdata subdirectories are used. If provided, only the filenames are required (not the absolute paths) and all the given files should be in the testdata subdirectory of the current component.
• TEST_SUFFIX is an optional argument. It provides the suffix name for the additional test. Default value is the name of the current component extension. The resulting test will be called action-A-additional-C-level1-i (where A is the action name, C the component name and i the test order number depending of the filename order).

Step 4: Add pipelined integration tests

To add a test that will run a pipeline of actions, apply them and check the result against resulting output, you just need to:

• create a integration-testdata directory in your component or action extension source directory
• add the ENABLE_INTEGRATION_TEST option to the camitk_extension macro:

camitk_extension(...
                 ...
                 ENABLE_INTEGRATION_TEST
)

• create a asm-input.scxml SCXML document that describes the action pipeline. To create a SCXML file, the best way to start is to run camitk-imp, select, set the parameters and apply the actions you want to perform in your test (do not forget to use the Save action to output some or all of the resulting component). Then go to the File menu and choose Save History. You will then have to manually edit the resulting XML file in order to fullfil the requirements (see below)
• rename all the input files needed to run your pipeline to the normalized form input-#.xxx and copy them to the integration-testdata subdirectory
• rename all the expected output files to compare with actual output to the normalized form output-#.xxx and copy them to the integration-testdata subdirectory
• run the CMake configuration again, et voilà, a new test called extensiontype-integration-test should be available

You can find an example in the itkfilters action extension of the imaging CEP (provided in the CamiTK Community Edition), and in the image/reconstruction action extension of the SDK.

Here are the steps required to transform a saved history SCXML to an functional asm-input.scxml:

1. Move all your input files to the integration-testdata subdirectory and rename them using the normalized form input-#.xxx where # is the unique index
2. Move the asm-input.scxml in the integration-testdata subdirectory
3. Change (if any) Open action into Open File instead (Open File does not require user interaction or the use of an open dialog):

    ...
    <camitk:action>
        <camitk:name>Open</camitk:name>
        <camitk:parameters/>
        ...

Becomes:

    <camitk:action>
        <camitk:name>Open File</camitk:name>
            <camitk:parameters>
                  <camitk:parameter name="File Name" value="input-#.mha" type="QString"/>
             </camitk:parameters>
            ...

4. For each action that produce a new component that will be later saved and compared to the expected result, you need to rename the component output name throughout the SCXML document to the normalized form output-#.xxx, where # is a unique index and xxx is the proper extension to save. For instance:

<camitk:action>
    <camitk:name>An Action That Creates a New Component</camitk:name>
    <camitk:parameters>
             ...
    </camitk:parameters>
    <camitk:inputs>
              ...
    </camitk:inputs>
    <camitk:outputs>
        <camitk:component name="named created automatically by the action (sometimes without extension)" type="ComponentClass"/>
    </camitk:outputs>
    ...

Becomes:

<camitk:action>
    <camitk:name>An Action That Creates a New Component</camitk:name>
    <camitk:parameters>
        ...
    </camitk:parameters>
    <camitk:inputs>
        ...
    </camitk:inputs>
    <camitk:outputs>
        <camitk:component name="output-#.xxx" type="ComponentClass"/>
    </camitk:outputs>
    ...

5. Change (if any) Save As action into Save instead. Also make sure that the input parameter of the Save action is using a component name corresponding to one of the output-#.xxx that you renamed in the previous step.
6. You may also rename the state’s id attributes in order to use more explicit names (do not forget to modify the corresponding transition’s target attributes)
7. test the asm-input.scxml using the action state machine:

# On the command line, go to the directory where you stored all input files and asm-input.xml documents
camitk-actionstatemachine --file asm-input.xml --output-dir .
# you should only have to press "Next"
# The execution ends up with all the expected output in a new YYYY-MM-DDTHH-MM-SS directory
# There should be no diff between the expected output and the actual output (sometimes you need to run it once to produce the proper expected output)
# if that works well try the autonext option (it should not just run without any user interaction)
camitk-actionstatemachine --file asm-input.xml --output-dir . --autonext

Step 5: Run your tests

Run all the tests

CMake automatically creates a target to run all the tests you have configured, that include the automatic test generated by the CamiTK testing framework.

• On the command line the target’s name is test so simply run:

cmake --build . --target test

• In a IDE (e.g., in Visual Studio), simply build the project RUN_TESTS

Run specific tests

It is possible to have a finer control on running the tests.

Your build directory must contain a file named CTestTestfile.cmake (generated during the configuration stage). This will allow you to type the following command, to execute some of the tests defined:

ctest -C <configuration> -V # note that "-C <configuration>" is optional on Linux and MacOS

For example:

ctest -C Debug -V   # note that "-C Debug" is optional on Linux and MacOS

-V option is for verbose, -VV is for extra verbose

For a specific test identified by its index use the -I option:

ctest -C debug -I <start index>,<end index>

For example:

ctest -C debug -I 5,8 # this will execute test #5, #6, #7 and #8

For a specific test identified by its name, use the -R option:

ctest -VV -R <test-name-regex>

For example:

ctest -VV -R action-myactionname # this will execute all the tests automatically created for myactionname

Step 6: Check test coverage of your CEP

A report on test coverage can also be generated automatically for any CamiTK Extension Project. The requirements are that:

1. you need to use the GNU cxx compiler
2. you need to have gcov and lcov installed (apt-get install is your friend !)

Then, it is very easy: open the top level CMakeLists.txt file of your CamiTK Extenstion Project and add the option ENABLE_TEST_COVERAGE to the camitk_extension_project(...) parameters:

camitk_extension_project(...
                         ENABLE_TEST_COVERAGE
)

Once your project is built, you can then run the camitk-test-coverage target:

make camitk-test-coverage

It will run all the available tests of your CEP using ctest and then compute the test coverage and generat an html report. The reports will be available for visualisation in ${CMAKE_BUILD_DIR}/camitk-test-coverage/index.html (the camitk-test-coverage subdirectory of the build dir).

So for instance, just type:

firefox camitk-test-coverage/index.html

Step 7: Add continuous integration

If your project is hosted on a gitlab server, you can quickly add continuous integration (CI) so that each modification triggers a configure/build/test pipeline. CI is very useful for mature code to set high level of code quality in order to protect your code from any regression.

In order to do that you need:

• to install gitlab runners (they are in charge of executing the pipeline on the VM/OS of your choice)
• to add a specific .gitlab-ci.yml configuration file in the root directory of your CEP.

CamiTK Community Edition has continuous integration in place, so that:

• each push to the gitlab hosted code (in any branch) triggers the CI pipeline
• each day a nightly pipeline is executed on the develop branch.

CamiTK Community Edition uses 3 different runners: three on Linux (debian stable and Ubuntu LTS, all using a docker executor) and one on Windows (Win10 also using docker).

In the end we get a summary of all the results (as well as report artefacts and emails to the developer in case of an error) send to the CamiTK Community Edition dashboard. For instance, this shows where a the nightly build:

Contact us if you would like more information about this and some support and help on how to set CI in place powerful for your project.

Disabling tests

Since CamiTK 5.0

In some specific cases, some automatic tests have to be disabled, for instance when a test depends on a specific platform or OS or version of a library and you know, by design that it can only fail if some environment requirements are not met.

There are two ways to disable some of the automatically generated tests:

• disable test unconditionally
• set specific requirements to run tests (if the requirements are not met, the tests will be disabled)

Unconditionally disable some tests

In order to unconditionally disable some tests, use the camitk_disable_tests CamiTK CMake macro. For instance, in an action CMakeLists.txt, you can do:

camitk_extension(ACTION_EXTENSION
                 ...
                 ENABLE_AUTO_TEST
                 ...
)

...

# Specific test management
camitk_disable_tests(TESTS action-myaction-level1-10
                     REASON "This test is expected to fail until such and such is implemented. Disable for now"
)

This will disable the test named action-myaction-level1-10 (which suppose that the test named action-myaction-level1-10 was automatically generated by the CamiTK test framework when the developer added the ENABLE_AUTO_TEST flag to the camitk_extension macro).

Basically, camitk_disable_tests macro encapsulates CMake set_tests_properties in order to manage version older that CMake 3.9 and to force you to give a (good) reason for bypassing the corresponding tests. If CMake version is lower than 3.9, the listed tests are still executed but with the WILL_FAIL property (i.e., the tests will failed, but will be considered as passed). There are no other way to disable test in CMake prior to version 3.9.

You can list as many tests as required in the TESTS argument.

You can of course use this macro for any existing test, even for tests that you added manually without the macro provided by the CamiTK test framework.

Define specific requirement for some tests

Sometimes some tests (manually generated or automatically generated by the CamiTK test framework) need to be disabled because they can only run properly in a given environment/testbed. If the specific environment does not match with the actual environment, theses tests are known to fail, and that is expected. For instance, some test can only pass if compiled and linked with a specific version of the VTK library, or if it is run on a specific OS.

Use the camitk_tests_requirement macro to specify the requirement of specific tests. For instance:

camitk_extension(ACTION_EXTENSION
                 ...
                 ENABLE_AUTO_TEST
                 ...
)

 ...

# Specific test management

# disable one test on Win32 platform (this is a two-parts requirement statement)
camitk_tests_requirement(TESTS action-itkfilters-integration-test
                         REQUIRES "NOT WIN32"
                         REASON "WIN32 OpenGL failure on a VM
 This test will always fail when run inside a VM due to OpenGL crash.
 This test will pass when run directly from a physical windows machine."
)

# disable three tests if the action is not compiled using a specific VTK version (this is a three-parts requirement statement)
camitk_tests_requirement(TESTS action-myaction-level1-3 action-myaction-level1-6 action-myaction-level1-11
                         REQUIRES "${VTK_VERSION} VERSION_EQUAL 6.3"
                         REASON "VTK version is not equals than 6.3
 The default test files are written using the currently supported version of VTK (VTK 6.3).
 This test will therefore fail when comparing the input to the output if another version of VTK is used."
)

camitk_tests_requirement will automatically disable one or more tests if a specific requirement is not met. This macro allows for specific test management in case the developper knows that a given test requires a specific given environment or can not be run (or will fail) in a given environment. It encapsulates CMake set_tests_properties, needs a specific requirement statement (using the REQUIRES parameter) and force you to give a (good) reason for bypassing the tests.

The requirement statement is a string that looks like a CMake test statement. It defines a subset of possible tests that should be used to check the current environment (OS, library/dependencies version…etc…). If the requirements checked by the REQUIRES parameter are not met, the listed tests are disabled.

A REQUIRES statement looks like a CMake test statement but is restricted to a small subset of possible CMake if statement. A REQUIRES statement can have one, two or three parts (in CamiTK 5.0):

• use a one-part statement to check the OS. Possible values are "WIN32", "WIN64", "MSVC", "APPLE" or "UNIX". This allows you that you want a test to be ran only on a specific OS.
• use a two-parts statement to check the opposite of a one-part statement using the NOT prefix. In the first example given above, the test action-itkfilters-integration-test is ran only if the requirement "NOT WIN32" is met (e.g., if the platform is not WIN32). Possible values are therefore "NOT WIN32", "NOT WIN64", "NOT MSVC", "NOT APPLE" or "NOT UNIX"
• use a three-parts statement to check the version of a library/dependency. A three-parts statement is composed of a left value (that is evaluated by the caller CMakeList.txt, i.e., evaluated in the context of the CMakeList.txt that uses the camitk_tests_requirement macro) followed by a comparator and a right value (also evaluated by the caller CMakeList.txt). Possible comparator are VERSION_LESS, VERSION_EQUAL or VERSION_GREATER. In the second example above, tests action-myaction-level1-3 action-myaction-level1-6 action-myaction-level1-11 are ran only if the variable VTK_VERSION (as set at configuration time depending of what version of VTK is detected) is equals to 6.3 (VERSION_EQUAL is a CMake test that is aware of major, minor and patch version, in this case the requirement is met when VTK version 6.3.x is used, independently of the value of x).