This directory contains tests that run CMake and/or other tools while precisely checking their return code and stdout/stderr content. The RunCMake infrastructure is useful for testing error cases and diagnostic output.
See also `../README.rst`_ and the `CMake Source Code Guide`_.
where ``Sample1`` through ``SampleN`` are sample project directories in the ``RunCPack/`` directory adjacent to this file.
4. Create file ``/CMakeLists.txt`` in the directory containing::
cmake_minimum_required(...) project(${RunCMake_TEST} NONE) # or languages needed include(${RunCMake_TEST}.cmake)
where ``${RunCMake_TEST}`` is literal. A value for ``RunCMake_TEST`` will be passed to CMake by the ``run_cmake`` macro when running each case.
5. Create a ``/.cmake`` file for each case named above containing the actual test code. Optionally create files containing expected test results:
``-result.txt`` Regex matching expected process result, if not ``0`` ``-stdout.txt`` Regex matching expected stdout content ``-stderr.txt`` Regex matching expected stderr content, if not ``^$`` ``-check.cmake`` Custom result check.
To specify platform-specific matches, create files of the form ``-{stdout,stderr}-.txt``.
Note that trailing newlines will be stripped from actual and expected test output before matching against the stdout and stderr expressions. The code in ``-check.cmake`` may use the `RunCMake Variables`_. On failure the script must store a message in ``RunCMake_TEST_FAILED``. The check script may optionally set ``RunCMake_TEST_FAILURE_MESSAGE`` with additional text to be included in the message if the test fails.
RunCMake Commands =================
A ``RunCMakeTest.cmake`` script, after ``include(RunCMake)``, may use the following commands.
``run_cmake()`` Run CMake or another command and check expected results described by ``-{result,stdout,stderr}.txt`` and ``-check.cmake``. The command is executed by a call of the form::
Behavior may be customized by setting `RunCMake Variables`_ before the call.
``run_cmake_command( ...)`` Sets ``RunCMake_TEST_COMMAND`` to ``;...`` and calls ``run_cmake()``.
This is useful to run an arbitrary command.
``run_cmake_script( ...)`` Sets ``RunCMake_TEST_COMMAND`` to ``${CMAKE_COMMAND};...;-P;${RunCMake_SOURCE_DIR}/.cmake`` and calls ``run_cmake()``.
This is useful to run CMake in script mode without configuring a project.
``run_cmake_with_options( ...)`` Sets ``RunCMake_TEST_OPTIONS`` to ``...`` and calls ``run_cmake()``.
``run_cmake_with_raw_args( "")`` Calls ``run_cmake()`` with the underlying ``execute_process()`` call extended with the content of ```` treated as literal source code of CMake language command arguments::
This is useful to pass arguments to the test command that cannot be encoded in CMake language ``;``-separated lists.
RunCMake Variables ==================
The behavior of `RunCMake Commands`_ such as ``run_cmake()`` may be customized by setting the following variables before a call.
``RunCMake_GENERATOR`` CMake generator to use when configuring projects. This provided to ``RunCMakeTest.cmake`` scripts automatically when they are executed, based on the CMake generator used to configure the test suite.
For some generators, additional variables are also provided:
``RunCMake_GENERATOR_PLATFORM`` Specifies the ``CMAKE_GENERATOR_PLATFORM``.
``RunCMake_GENERATOR_TOOLSET`` Specifies the ``CMAKE_GENERATOR_TOOLSET``.
``RunCMake_GENERATOR_INSTANCE`` Specifies the ``CMAKE_GENERATOR_INSTANCE``.
``RunCMake_GENERATOR_IS_MULTI_CONFIG`` Boolean value indicating whether ``${RunCMake_GENERATOR}`` is a multi-config generator. This provided to ``RunCMakeTest.cmake`` scripts automatically when they are executed, based on the CMake generator used to configure the test suite.
``RunCMake_SOURCE_DIR`` Absolute path to the ``Tests/RunCMake/`` directory in the CMake source tree. This provided to ``RunCMakeTest.cmake`` scripts automatically when they are executed.
``RunCMake_BINARY_DIR`` Absolute path to the ``Tests/RunCMake/`` directory in the CMake binary tree. This provided to ``RunCMakeTest.cmake`` scripts automatically when they are executed.
``RunCMake_TEST_SOURCE_DIR`` Absolute path to the individual test case's source tree. If not set, defaults to ``${RunCMake_SOURCE_DIR}``.
``RunCMake_TEST_BINARY_DIR`` Absolute path to the individual test case's binary tree. If not set, defaults to ``${RunCMake_BINARY_DIR}/-build``.
``RunCMake_TEST_NO_CLEAN`` Boolean value indicating whether ``run_cmake()`` should remove the ``${RunCMake_TEST_BINARY_DIR}`` directory before running the test case. If not set, or if set to a false value, the directory is removed.
This is useful to run `Multi-Step Test Cases`_.
``RunCMake_TEST_COMMAND`` The command for ``run_cmake()`` to execute. If not set, defaults to running CMake to generate a project::
``RunCMake_TEST_COMMAND_WORKING_DIRECTORY`` The working directory in which ``run_cmake()`` to execute its command. If not set, defaults to ``${RunCMake_TEST_BINARY_DIR}``.
``RunCMake_TEST_OPTIONS`` Additional command-line options for ``run_cmake()`` to pass to CMake when configuring a project with a default ``RunCMake_TEST_COMMAND``. If not set, defaults to empty. If ``RunCMake_TEST_COMMAND`` is set, ``RunCMake_TEST_OPTIONS`` is forced to empty.
``RunCMake_TEST_OUTPUT_MERGE`` Boolean value indicating whether ``run_cmake()`` should redirect the test process's ``stderr`` into its ``stdout``.
``RunCMake_TEST_TIMEOUT`` Specify a timeout, in seconds, for ``run_cmake()`` to pass to its underlying ``execute_process()`` call using the ``TIMEOUT`` option.
Multi-Step Test Cases =====================
Normally each ``run_cmake()`` call corresponds to one standalone test case with its own build tree. However, some test cases may require multiple steps to be performed in a single build tree. This can be achieved as follows::
In this example, ``block() ... endblock()`` is used to isolate the variable settings from later cases. A single build tree is used for all cases inside the block. The first step cleans the build tree and runs CMake to configure the case's project. The second step runs ``cmake --build`` to drive the generated build system and merges the build tool's ``stderr`` into its ``stdout``. Note that each call uses a unique case name so that expected results can be expressed individually.
Running a Test ==============
Each call to ``add_RunCMake_test(Example)`` in `CMakeLists.txt`_ creates a test named ``RunCMake.Example`` that may be run with ``ctest``::
$ ctest -R "^RunCMake\.Example$"
To speed up local testing, you can choose to run only a subset of ``run_cmake()`` tests in a ``RunCMakeTest.cmake`` script by using the ``RunCMake_TEST_FILTER`` environment variable. If this variable is set, it is treated as a regular expression, and any tests whose names don't match the regular expression are not run. For example::
Timo Röhling