CMake Generating Testing Files: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
(→‎Description Of The System: Add public dashboard information)
(Replace content with link to new CMake community wiki)
 
(20 intermediate revisions by 7 users not shown)
Line 1: Line 1:
==Introduction==
{{CMake/Template/Moved}}


CTest is a powerful testing client compatible with
This page has moved [https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Generating-Testing-Files here].
[http://public.kitware.com/Dart Dart] Dashboard system. It is distributed as a
part of [http://www.cmake.org CMake] but can be used with or without CMake.
This tutorial explains some background behind the testing with CTest and the
process of generating CTest testing files.
 
==CTest Overview==
 
CTest currently uses the old Dart input configuration file. New input format is
being developed, but the old format will be valid in the future. For legacy
purposes, the input files are called ''DartConfiguration.tcl'', and
''DartTestfile.txt''. ''DartConfiguration.tcl'' describes the system on which
tests are performed, while ''DartTestfile.txt'' provides the list of tests.
 
CTest reads ''DartConfiguration.tcl'' and depending on the options provided it
performs certain stage of testing.
 
===Description Of The System===
 
Even though the extension of ''DartConfiguration.tcl''
file is ''.tcl'', it is not TCL file. It is simply list of key/value pairs.
Line starting with ''#'' are considered comments and are ignored. This file has to appear at the toplevel directory of the project's build directory.
 
Example ''DartConfiguration.tcl'' file is:
 
<pre>
SourceDirectory: /path/to/project/source/directory
BuildDirectory: /path/to/project/build/directory
CVSUpdateOptions: -d -A -P
Site: my-system-name
BuildName: build-name
DropSite: public.kitware.com
DropLocation: /cgi-bin/HTTPUploadDartFile.cgi
DropMethod: http
TriggerSite: http://public.kitware.com/cgi-bin/Submit-CMake-TestingResults.pl
NightlyStartTime: 21:00:00 EDT
ConfigureCommand: "/path/to/source/directory/Project/configure"
MakeCommand: /usr/bin/make -i
CVSCommand: /usr/bin/cvs
CoverageCommand: /usr/bin/gcov
TimeOut: 1500
</pre>
 
Most entries are pretty self explanatory. ''Site'' refers to the system name.
Example ''Site'' would be ''andoria.kitwarein.com''. ''BuildName'' is a breif
description of the system and the project. Example would be
''FLTK-Linux-g++334'' which would be for project ''FLTK'' build on ''Linux''
system running ''g++'' version 3.3.4.
 
''NightlyStartTime'' is the time when Nightly dashboard starts. This is useful
when doing Nightly dashboard. The Nightly dashboard will update the source
directory to the state at given time.
 
''TimeOut'' is a maximum time the test will be allowed to run. Once test time
expires, the test will be interrupted and the status will be ''Timeout''.
 
''DropSite'', ''DropLocation'', ''DropMethod'', and ''TriggerSite'' describe the
location where test results are submitted. The location specified in the example
is the [http://public.kitware.com/dashboard.php?name=public Public Dashboard]
provided as a service by [http://www.kitware.com Kitware].
 
===Description of Tests===
 
For legacy purposes and for compatibility with ''Dart'', the test description
file is called ''DartTestfile.txt''. It is parsed by full CMake parser, but the
CMake commands that are available are ''SUBDIRS'' and ''ADD_TEST''. Example
''DartTestfile.txt'' is:
 
<pre>
ADD_TEST(testProcess-1 "/my/project/testProcess" "1")
ADD_TEST(testProcess-2 "/my/project/testProcess" "2")
ADD_TEST(testProcess-3 "/my/project/testProcess" "3")
ADD_TEST(testProcess-4 "/my/project/testProcess" "4")
ADD_TEST(testProcess-5 "/my/project/testProcess" "5")
ADD_TEST(testProcess-6 "/my/project/testProcess" "6")
</pre>
 
Each of the ''ADD_TEST'' adds the test for CTest. The first argument is the
unique name of test. It should be unique, so that it can be differentiated
among all results. The second argument is the executable run as a part of the
test. This is followed by arbitrary number of arguments for the executable.
Executable can be build as a part of the project, or system executable can be
used. For example, to test a ''Python'' feature:
 
<pre>
ADD_TEST(testTkInter "/usr/bin/python" "-c" "import Tkinter")
</pre>
 
It is a good practice to specify full path to the executable name, since that
way CTest will always pick the desired executable. For example, let say the
test executable name is ''python'' but there is a python installed on the
system, the CTest may pick either the ''python'' executable built by our build
process or use the installed one.
 
===Group Tests in Subdirectories===
 
Project tests are usually organized into some structure based on the part of the project they test. For example, if the project contains some common code, IO code, processing code, and GUI code, there may be group of tests for each subsection. Common way of organizing this would be in subdirectories. For example, let say the project has the following directory structure:
 
<pre>
/Project
        /Common
        /IO
        /Processing
        /GUI
</pre>
 
It is fairly easy to reproduce this structure for testing:
 
<pre>
/Project
        /Testing
                /Common
                /IO
                /Processing
                /GUI
</pre>
 
CTest will not automatically search all subdirectories and find all
''DartTestfile.txt''. The project needs to tell it where the tests are. This
can be achieved using ''SUBDIRS'' command in ''DartTestfile.txt''. An example use of ''SUBDIRS'' command is:
 
<pre>
SUBDIRS(Testing/Common)
SUBDIRS(Testing/IO)
SUBDIRS(Testing/Processing Testing/GUI)
</pre>
 
The arguments to the ''SUBDIRS'' command are subdirectories containing the
tests. If this file is placed in the top of our project, CTest will search for
files ''DartTestfile.txt'' inside directories /.../Project/Testing/Common,
/.../Project/Testing/IO, and so on to /.../Project/Testing/GUI.
 
==Generating Testing Files==
 
Testing files ''DartConfiguration.tcl'' and all ''DartTestfile.txt' must be
generated in the same tree as the CTest is run. In most cases this is the build
tree of the project. Since the testing files contain build and system  specific
information, they should be generated as a part of testing process and not
permanently part of source code.
 
===Using CMake===
 
CMake has a automated way of generating CTest input files. There are three pieces to performing testing:
 
  ENABLE_TESTING()
 
Enables testing for the project. This command will gather all tests and produce
''DartTestfile.txt'' files with apropriate ''ADD_TEST'' and ''SUBDIRS''
commands.
 
  INCLUDE(Dart)
 
Including ''Dart'' CMake module will generate default ''DartConfiguration.tcl''
file, which will contain information about updating, configuring, and building
the project.
 
  ADD_TEST(test1 ${EXECUTABLE_OUTPUT_PATH}/test1)
 
Describes test that will be put in ''DartTestfile.txt''. The syntax of ADD_TEST on CMake level is the same as once it is written in ''DartTestfile.txt''.
 
More details about how to generate test files with CMake is described in
[[CMake Testing With CTest|Testing With CTest]] tutorial.
 
===Without Using CMake===
 
CTest is distributed together with CMake, but it can be used independently. The
''DartConfiguration.tcl'' file can be easily generated using ''autoconf'' or
any other tool, or even written by hand. Whatever the method of generating that
file, the author has to make sure all the necessary information is written in
it. Same goes for ''DartTestfile.txt'' files.
 
==Conclusion==
 
Though we consider CMake to be a great build configuration tool, CTest can be
used without CMake. Because of its simplicity, it can be added to any project
without changing the rest of the project development. Using a small number of
configuration files, CTest can easily automate most if not all of the commonly
used testing procedures.

Latest revision as of 15:41, 30 April 2018


The CMake community Wiki has moved to the Kitware GitLab Instance.

This page has moved here.