CMake Scripting Of CTest: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
(→‎CTest Scripting: Problem with copying from e-mail)
(Replace content with link to new CMake community wiki)
 
(39 intermediate revisions by 14 users not shown)
Line 1: Line 1:
==Testing With CTest==
{{CMake/Template/Moved}}


In [[CMake Testing With CTest|Testing With CTest]], you learned how to
This page has moved [https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Scripting-Of-CTest here].
use CTest to perfom simple testing. That page also describes how to use
CTest to submit dashboards. The typical sequence of commands that
dashboard contributor will do involves:
 
cd /location/of/binary/Directory
rm -rf BinaryDirectory
mkdir BinaryDirectory
cd BinaryDirectory
ccmake /location/of/source/Directory
ctest -D Nightly
 
This is done on a typical UNIX system. On Windows system, the
contributor needs to do something similar using Windows command
interpreter, using something like Cygwin, or using a scripting language
such as Perl, Python, or Tcl. Unfortunately none of these solutions is
portable.
 
==CTest Scripting==
 
CTest has a built-in scripting mode, which significantly simplifies
generating dashboards. Here is example of this script:
 
<pre>
SET (CTEST_SOURCE_DIRECTORY "C:/Hoffman/My Builds/CMake")
SET (CTEST_BINARY_DIRECTORY "C:/Hoffman/My Builds/CMakeVSNMake71")
 
SET (CTEST_CVS_COMMAND "C:/cygwin/bin/cvs.exe")
SET (CTEST_CVS_CHECKOUT  "${CTEST_CVS_COMMAND} -d:pserver:hoffman@www.cmake.org:/cvsroot/CMake co -d\"${CTEST_SOURCE_NAME}\" CMake")
 
# which ctest command to use for running the dashboard
SET (CTEST_COMMAND
  "C:/Program Files/CMake/bin/ctest.exe -D Nightly"
  )
 
# what cmake command to use for configuring this dashboard
SET (CTEST_CMAKE_COMMAND
  "C:/Program Files/CMake/bin/cmake.exe"
  )
 
 
####################################################################
# The values in this section are optional you can either
# have them or leave them commented out
####################################################################
 
# should ctest wipe the binary tree before running
SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
 
# this is the initial cache to use for the binary tree, be careful to escape
# any quotes inside of this string if you use it
SET (CTEST_INITIAL_CACHE "
MAKECOMMAND:STRING=nmake -i
CMAKE_MAKE_PROGRAM:FILEPATH=nmake
CMAKE_GENERATOR:INTERNAL=NMake Makefiles
BUILDNAME:STRING=Win32-nmake71
SITE:STRING=VOGON.kitware
CVSCOMMAND:FILEPATH=C:/cygwin/bin/cvs.exe
")
 
# set any extra environment variables here
SET (CTEST_ENVIRONMENT
)
</pre>
 
This script defined all the information CTest needs to submit CMake
dashboard on Windows XP computer using NMake build system. The syntax
used is the same as in CMake. Also, most commands available in CMake are
available here, except things that require generator, such as
TRY_COMPILE, ADD_LIBRARY, etc.
 
The script can be run by executing:
 
  ctest -S /path/to/script.cmake
 
or in this case:
 
  c:/Program Files/CMake/bin/ctest -S c:/Hoffman/DashboardScripts/vogon_cmake_nmake.cmake
 
===Main Settings===
 
Every CTest script has to contain the source and binary directory:
 
  SET (CTEST_SOURCE_DIRECTORY "$ENV{HOME}/Dashboards/My Testing/Insight")
  SET (CTEST_BINARY_DIRECTORY "$ENV{HOME}/Dashboards/My Testing/Insight-bin")
 
The "$ENV{HOME}" gets replaced by the environment variable "HOME", which
on most systems points to user's home directory.
 
Second thing we need is the command that perform initial configuration
of the project. For example, if the project uses CMake, then the command
would be something like:
 
  SET (CTEST_CMAKE_COMMAND "/usr/local/bin/cmake")
 
That assumes that CMake is installed in /usr/local. Similarly to setting
up CMake command, we need to setup the way CTest will be called. Let say
in your case you run CTest using:
 
  ctest -D Nightly
 
Then the command in the CTest script should be:
 
  SET (CTEST_COMMAND "/usr/local/bin/ctest -D Nightly")
 
The final important piece we need is the initial cache that CMake uses
to configure the project. This should be minimal set of settings that
will allow the system to test certain aspect of the project. Example
initial cache would be:
 
<pre>
SET (CTEST_INITIAL_CACHE "
//Name of generator.
CMAKE_GENERATOR:INTERNAL=Visual Studio 7
 
//Name of the build
BUILDNAME:STRING=Win32-vs70
 
//Name of the computer/site where compile is being run
SITE:STRING=DASH2.kitware
 
//Build VTK with shared libraries.
BUILD_SHARED_LIBS:BOOL=ON
 
//Path to the CVS
CVSCOMMAND:FILEPATH=C:/cygwin/bin/cvs.exe
 
//ITK TCL and Python wrapping
ITK_CSWIG_TCL:BOOL=ON
ITK_CSWIG_PYTHON:BOOL=ON
")
</pre>
 
This will set the generator to be "Visual Studio 7", it will use shared
libraries and turn on TCL and Python wrapping. Also, in the cache we can
set the SITE and BUILDNAME variables, which will be used to display the
testing results on the dashboard.
 
This is it. The CTest script can now be tested using:
 
  ctest -S /path/to/script/script.cmake -V
 
The -V argument tells CTest to be extra verbose when doing dashboard.
 
===More Settings===
 
The main settings will generate dashboard for most systems. There are
however issues in certain setups.
 
Most dashboard contributors want to start dashboard and not touch those
systems for months. The problem is that the binary directory of the
project will grow larger and larger. Common practice therefore is to
wipe out binary directory every time dashboard is run. We can achieve
this in CTest script by setting:
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY:
 
  SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
 
Now CTest will wipe out the binary tree every time. Make sure that the
source tree is not equal to binary tree.
 
Second useful option is setting of environment in which CTest runs. Let
say you have Cygwin, Borland, Visual Studio, and Intel compiler on your
Windows system and you do not want to have system's global PATH pointing
to various directories (Borland and Cygwin use "make", which one should
be used). Now you want to do Borland dashboard. You can set environment
variables using CTEST_ENVIRONMENT:
 
  SET (CTEST_ENVIRONMENT
    "PATH=c:/WINDOWS/system32\;c:/WINDOWS\;c:/Programs/Borland/Bcc55/Bin")
 
Note that CTEST_ENVIRONMENT can accept multiple environment variables.
For example:
 
  SET (CTEST_ENVIRONMENT
    "DISPLAY=:0"
    "GLIBCPP_FORCE_NEW=1"
    "CC=gcc-3.4"
    "CXX=g++-3.4"
  )
 
Another useful thing to do is to perform extra update on external
repository. For example, VTK dashboard need to update VTK data. This can
be achieved using CTEST_EXTRA_UPDATES_*. This also requires the
CTEST_CVS_COMMAND to be set. An example of extra update is:
 
  SET (CTEST_CVS_COMMAND "/usr/bin/cvs")
  SET (CTEST_EXTRA_UPDATES_1 "$ENV{HOME}/Dashboards/MyTests/VTKData" "-dAP")
 
There can be up to ten CTEST_EXTRA_UPDATES_* settings:
CTEST_EXTRA_UPDATES_1, CTEST_EXTRA_UPDATES_2, ...,
CTEST_EXTRA_UPDATES_10.
 
===Advanced Settings===
 
When performing complex dashboards setups, some more CMake/CTest
knowledge is needed. This section will describe solutions to some common
CTest problems.
 
Doing memory checking using Valgrind or similar product can take
significant amount of time. There are two improvements the dashboard
contributor can do to make dashboard more useful. First one is to submit
partial results. This can be achieved by breaking down the CTest run
into individual runs:
 
  SET (CTEST_COMMAND
    "/usr/local/bin/ctest -D NightlyStart"
    "/usr/local/bin/ctest -D NightlyUpdate"
    "/usr/local/bin/ctest -D NightlyConfigure"
    "/usr/local/bin/ctest -D NightlyBuild"
    "/usr/local/bin/ctest -D NightlyTest"
    "/usr/local/bin/ctest -D NightlySubmit"
    "/usr/local/bin/ctest -D NightlyMemCheck"
    "/usr/local/bin/ctest -D NightlySubmit"
    )
 
The first six CTest commands will perform standard Nightly dashboard.
The last two will perform only Memory checking and submission. This the
whole dashboard submission will be there except the memory checking.
Once the memory checking is completed, it will be submitted.
 
The second thing is to add special flags to make the CTest only test
subset of tests:
 
  SET (CTEST_COMMAND
    ...
    "/usr/local/bin/ctest -D NightlyMemCheck -I 0,,17"
    "/usr/local/bin/ctest -D NightlySubmit"
    )
 
The "-I 0,,17" will tell CTest to only run memory checking of every 17th
test.
 
Let say you do not want to require the source of the project to be
available. This is useful if the dashboards are performed on the
temporary drive that can get erased. The following setting will checkout
the source tree of CMake out of CVS repository before the testing is
performed:
 
  SET (CTEST_CVS_CHECKOUT
    "${CTEST_CVS_COMMAND} -q -z3
    -d :pserver:anoncvs@www.cmake.org:/cvsroot/CMake
    co -d \"CMake\" -D yesterday CMake")
 
Another useful feature is to backup source tree before performing
dashboard. This way if the build fails, you can restore the old tree.
This is useful when you require the source tree to be in good condition.
This option has to be used in combination with CTEST_CVS_CHECKOUT. To
turn this feature on, add this to CTest script:
 
  SET (CTEST_BACKUP_AND_RESTORE TRUE)
 
When setting up Continuous dashboard, the CTest has to be run
periodically to check if there were any changes in the source. When the
change appears, the full dashboard is tested. This can be achieved by
running CTest from system scheduler or some script, or by using
Continuous feature of CTest script. First of all, make sure that
CTEST_COMMAND is using Continuous model:
 
  SET (CTEST_COMMAND
    "C:/Program Files/CMake20/bin/ctest.exe -D Continuous"
    )
 
Also, if the number of tests is high, you may want to only run subset of
tests:
 
  SET (CTEST_COMMAND
    "C:/Program Files/CMake20/bin/ctest.exe -D Continuous -I ,,3"
    )
 
Now set the continuous parameters:
 
  SET (CTEST_CONTINUOUS_DURATION 600)
  SET (CTEST_CONTINUOUS_MINIMUM_INTERVAL 10)
  SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY_ONCE 1)
 
The CTEST_CONTINUOUS_DURATION is the duration of Continuous testing in
minutes. In this case it will run Continuous dashboards for 10 hours. If
it starts at 8 AM, the last continuous will be done around 6 PM. The
CTEST_CONTINUOUS_MINIMUM_INTERVAL is the minimum interval between
Continuous checks. In this case it will perform check every 10 minutes.
CTEST_START_WITH_EMPTY_BINARY_DIRECTORY_ONCE will remove binary tree
once the first Continuous is run. This way Continuous tests will take
less time, since it will only rebuild modified files.
 
===Available Variables===
 
When running CTest scripts, there are several variables available that
can simplify writing CTest scripts. These variable include:
 
{| border="1" cellpadding="2" cellspacing="0"
|- bgcolor="#abcdef"
! Variable !! Description
|-
| CTEST_SCRIPT_DIRECTORY
| Location of directory in which the CTest script is
|-
| CTEST_SCRIPT_NAME
| Name of the CTest script
|- valign="top"
| CTEST_SCRIPT_ARG
| The extra arguments passed to CTest -S. For example, when running:
 
<pre>
ctest -S /home/scripts/mysystem.cmake,Experimental
</pre>
 
then the CTEST_SCRIPT_ARG will be set to Experimental. Then this can be
used inside the script:
 
<pre>
SET(MODEL Nightly)
IF(${CTEST_SCRIPT_ARGS} MATCHES Experimental)
  SET(MODEL Experimental)
ENDIF(${CTEST_SCRIPT_ARGS} MATCHES Experimental)
SET (CTEST_COMMAND
    "/usr/local/bin/ctest -D ${MODEL}")
</pre>
 
Now we can specify Nighlty or Experimental without editing CTest script.
|-
|}
 
==Conclusion==
 
CTest scripts can significantly simplify dashboard contribution. Using
simple template script with slight modification, the whole cluster of
various systems can submit dashboards.

Latest revision as of 15:41, 30 April 2018


The CMake community Wiki has moved to the Kitware GitLab Instance.

This page has moved here.