test

CamiTK CMake macros used for testing. More...

Collaboration diagram for test:

Modules
	level
	CamiTK CMake macros used generating the mediawiki tab containing the list of all functionnal tests, classified by CamiTK extensions with their given LEVEL.

Functions
	camitk_add_integration_test ()
	camitk_add_integration_test is a macro to add a new test to the CTest infrastructure It encapsulates CMake add_test and use the action state machine to run a set of actions. More...
	camitk_add_test ()
	camitk_add_test is a macro to add a new test to the CTest infrastructure It encapsulates CMake add_test and adds useful way of testing programs. More...
	camitk_additional_action_test ()
	camitk_additional_action_test is a macro to create additional level1 test of CamiTK action to the CTest infrastructure. More...
	camitk_disable_tests ()
	camitk_disable_tests is a macro to unconditionally disable one or more tests. More...
	camitk_init_test ()
	camitk_init_test is a macro to initialize a group of test (for the same command) It is used to initialize a series of tests for a given target. More...
	camitk_tests_requirement ()
	camitk_tests_requirement is a macro to disable one or more tests. More...
	set (CAMITK_TEST_COMMAND_FILE ${CAMITK_TEST_OUTPUT_DIR}/command) set(CAMITK_TEST_COMMAND_RESULT_FILE $
	This CMake script run the action state machine and compare the output files to the expected files The intended use is to run this script from the camitk_add_test_actionstatemachine. More...

Detailed Description

CamiTK CMake macros used for testing.

Function Documentation

camitk_add_integration_test()

camitk_add_integration_test

(

)

camitk_add_integration_test is a macro to add a new test to the CTest infrastructure It encapsulates CMake add_test and use the action state machine to run a set of actions.

The set of actions are described in a CamiTK SCXML document. This macro is used to add integration tests.

Details on the runned test can be found in directory ${CMAKE_BINARY_DIR}/Testing/Temporary/${TYPE_EXTENSION}-${EXTENSION_NAME}-integration-test

It is typically usued inside the camitk_extension(..) macro and expects the following variable to be set beforehand:

• TYPE_EXTENSION action or component)
• EXTENSION_NAME name of the current action or component

It does add a test if and only if:

• ${CMAKE_CURRENT_SOURCE_DIR}/integration-testdata exists
• ${CMAKE_CURRENT_SOURCE_DIR}/integration-testdata contains a file called "asm-input.scxml"
• ${CMAKE_CURRENT_SOURCE_DIR}/integration-testdata contains at least one file name "output-*.*"

Usage:

camitk_add_integration_test()

camitk_add_test()

camitk_add_test

(

)

camitk_add_test is a macro to add a new test to the CTest infrastructure It encapsulates CMake add_test and adds useful way of testing programs.

It cannot operate on its own, you need to call camitk_init_test first (and only once) before calling camitk_add_test

Details on the runned test can be found in ${CMAKE_BINARY_DIR}/Testing/Temporary/target# where target is the executable name (see camitk_init_test() macro), and # is the test order number.

Usage:

camitk_add_test(EXECUTABLE_ARGS "arg1 arg2 arg3"
                PASS_FILE_OUTPUT pristineOutput
                PASS_REGULAR_EXPRESSION regexp
                FAIL_REGULAR_EXPRESSION regexp
                PROJECT_NAME name
                TEST_SUFFIX name
)

Parameters

EXECUTABLE_ARGS	(optional) The executable arguments (all in double quotes), typically each test will have different arguments. Can be empty
PASS_FILE_OUTPUT	(optional) If specified the test to perform is to compare the output of the command to this file. It the output is the same, then the test is passed, otherwise it is failed.
PASS_REGULAR_EXPRESSION	(optional) This is equivalent to "PASS_REGULAR_EXPRESSION regexp" property for the test, see http://www.cmake.org/Wiki/CTest:FAQ#My_test_does_not_return_non-zero_on_failure._What_can_I_do.3F
FAIL_REGULAR_EXPRESSION	This is equivalent to "FAIL_REGULAR_EXPRESSION regexp" property for the test, see http://www.cmake.org/Wiki/CTest:FAQ#My_test_does_not_return_non-zero_on_failure._What_can_I_do.3F
PROJECT_NAME	Base name for the test, can be used for ctest -S
TEST_SUFFIX	Suffix added to the test name, just after PROJECT_NAME and before the test index

If no argument are given, it does the equivalent as add_test(name target)

Note

You can only choose one between nothing, PASS_FILE, PASS_REGULAR_EXPRESSION and FAIL_REGULAR_EXPRESSION

Example invocation:

add_executable(myprogram)
 
...
# Start the test series for myprogram
camitk_init_test(myprogram)
camitk_add_test(EXECUTABLE_ARGS "-a inputfile.xml -c" PASS_FILE pristineOuputToCompareTo.xml) # will be called myprogram1
...
camitk_add_test(...) # myprogram2

See also

camitk_init_test

References a, endif(), execute_process(), and set().

Here is the call graph for this function:

camitk_additional_action_test()

camitk_additional_action_test

(

)

camitk_additional_action_test is a macro to create additional level1 test of CamiTK action to the CTest infrastructure.

It should be used when a CEP introduces a new component (with new test files) and an existing action should be run using this new component and test files.

Usage:

camitk_additional_action_test(ACTION_EXTENSIONS action1 action2 ...
                              [TEST_FILES  file1 file2 ...]
                              [TEST_SUFFIX name]
)

Parameters

ACTION_EXTENSIONS	List of the action extension that will be additionnaly tested with the given test files. Only the action extension name (not the full library/dll name nor the path) should be given. This macro checks the existence of the extension library/dll in the following directories (in this order) : build dir user install dir and global install dir. The first one that is found is used for the tests.
TEST_FILES	Optional. List of files to use for testing. If not set, then all the files in the testdata dir are used. If provided, only the filenames are required (not the absolute paths) and all the given files should be in the testdata subdir.

This will add tests named as follow: "component-[name of the current component extension]-additional-[name of the action extension]-level1-[index]"

See also

camitk_extension

camitk_disable_tests()

camitk_disable_tests

(

)

camitk_disable_tests is a macro to unconditionally disable one or more tests.

It encapsulates CMake set_tests_properties in order to manage version older that CMake 3.9 and to force the developper to give a (good) reason for bypassing tests. If CMake version is lower than 3.9, the test(s) is/are executed but with the WILL_FAIL (i.e., a failed test is considered as passed).

Use in conjunction with the camitk_extension macro. In camitk_extension using ENABLE_AUTO_TEST automatically generate a number of tests. Some of these tests might need to be unconditionally disabled during the development process, in this case use this macro.

Usage:

camitk_disable_tests(TESTS test1 [test2...]
                     REASON "reason"
)

Parameters

TESTS	Names of the test(s) to disable (names separated by space)
REASON	A text to explain why the tests is/are unconditionally disabled.

Example invocation:

camitk_extension(ACTION_EXTENSION
                ...
                ENABLE_AUTO_TEST
                ...
)
 
...
# Specific test management
camitk_disable_tests(TESTS action-myaction-level1-10
                     REASON "This test is expected to fail until such and such is implemented. Disable for now"
)

See also

camitk_tests_requirement

References endif(), and set().

Here is the call graph for this function:

camitk_init_test()

camitk_init_test

(

)

camitk_init_test is a macro to initialize a group of test (for the same command) It is used to initialize a series of tests for a given target.

Usually this is placed in the same CMakeLists.txt as the add_executable() cmake instruction or camitk_application() macro

It does few useful things:

• check if the application is properly defined
• initialize test id (then automatically incremented in camitk_test_declare
• initialize test output directory

Usage:

camitk_init_test(applicationname)

Parameters

applicationname

(required) The name of the test application (in camitk context, name of the project

Example invocation:

camitk_application()
 
...
# Start the test series for myprogram
camitk_init_test(myprogram)
camitk_add_test(...) # will be called myprogram1
...
camitk_add_test(...) # myprogram2

See also

camitk_add_test

References endif(), set(), and unset().

Here is the call graph for this function:

camitk_tests_requirement()

camitk_tests_requirement

(

)

camitk_tests_requirement is a macro to disable one or more tests.

It allows for specific test management in case the developper knows that a given test requires a given environment or can not be run in a given environment.

It encapsulates CMake set_tests_properties and adds CamiTK specificities and force the developper to give a (good) reason for bypassing tests.

Use this macro in conjunction with the camitk_extension macro. In camitk_extension using ENABLE_AUTO_TEST automatically generate a number of tests. Some of these tests might need to be disabled due to the environment (e.g. a different VTK version, an OpenGL problem on windows VM...). Use camitk_tests_requirement below the camitk_extension (i.e. once the tests are created).

Usage:

camitk_tests_requirement(TESTS test1 [test2...]
                         REASON "reason"
                         REQUIRES environment_test_statement
)

Parameters

TESTS	Names of the test(s) to disable separated by space
REASON	A text to explain why the test is disabled. This will appear during configuration if the test(s) is/are disabled due to unmet requirement.
REQUIRES	A CMake test statement that can check the current environment (OS, library/dependencies version...etc... If the requirements are not met, the listed test(s) is/are disabled REQUIRES statement looks like a CMake test statement but is restricted to a subset of CMake if statement. There is three types of REQUIRES statement: one-part statement are used to test the OS. Possible value are "WIN32", "WIN64", "MSVC", "APPLE" or "UNIX" two-parts statement are used to test the opposite of the one-part statement using the "NOT" prefix. Possible values are "NOT WIN32", "NOT WIN64", "NOT MSVC", "NOT APPLE" or "NOT UNIX" three-parts statement are used to check the version of library/dependencies. It is composed of a left value (that is evaluated by the caller CMakeList, not here), followed by a test statement and a right value (also evaluated by the caller CMakeList). Possible test statements are VERSION_LESS, VERSION_EQUAL or VERSION_GREATER

Example invocation:

 camitk_extension(ACTION_EXTENSION
                 ...
                 ENABLE_AUTO_TEST
                 ...
 )
 
 ...
 # Specific test management
 
 # a three-parts statement to disable test if VTK version is not a specific version
 camitk_tests_requirement(TESTS action-myaction-level1-3 action-myaction-level1-6 action-myaction-level1-11
                          REQUIRES "${VTK_VERSION} VERSION_EQUAL 6.3"
                          REASON "VTK version is not equals than 6.3
  The default test files are written using the currently supported version of VTK (VTK 6.3).
  This test will therefore fail when comparing the input to the output if another version of VTK is used."
)
 
 # a two-parts statement to disable test on WIN32
 camitk_tests_requirement(TESTS action-itkfilters-integration-test
                          REQUIRES "NOT WIN32"
                          REASON "WIN32 OpenGL failure on a VM
  This test will always fail when run inside a VM due to OpenGL crash.
  This test will pass when run directly from a physical windows machine."
 )

See also

camitk_disable_tests

References a, endif(), set(), and camitk::transform.

Here is the call graph for this function:

set()

set

(

CAMITK_TEST_COMMAND_FILE ${CAMITK_TEST_OUTPUT_DIR}/

command

)

This CMake script run the action state machine and compare the output files to the expected files The intended use is to run this script from the camitk_add_test_actionstatemachine.