TWiki
>
Gaudi Web
>
GaudiTestingInfrastructure
(2021-06-28,
PatrickSKoppenburg
)
(raw view)
E
dit
A
ttach
P
DF
---+!! Gaudi Testing Infrastructure %TOC% ---++ Introduction The importance of tests in software development is known to everybody... or it should, so I'll not talk about it. To effectively run tests for a piece of software, it is important to have a testing infrastructure. Such an infrastructure should allow the developer to run the tests and check the results in the simplest possible way (if not by clicking on a button, it should be by executing a simple command). Adding new tests should be easy as well, so the introduction of tests for newly discovered bugs will not require much time, that can be devoted, instead, to fix them. LCG provides tools for testing in its collection of libraries and tools. One of these tools is [[http://www.codesourcery.com/qmtest/][QMTest]], which is easily extensible to cover the special needs that may come along in big projects like Gaudi. This is the tool I used to prepare a testing infrastructure for Gaudi. <noautolink> ---++ GaudiTest.GaudiExeTest: the Gaudi extension to QMTest </noautolink> Although QMTest already comes with a good set of classes to run the tests, they are not flexible enough for the special cases that may be encountered in the tests for Gaudi, e.g. the need to compare the standard output with a reference file that my be different for different platforms, or to check for the presence (or absence) of a specific message in the output. To overcome the limitations of QMTest standard tests, I wrote a Gaudi specific test class that inherits from the standard !ExecTestBase class (provided by QMTest). The new test class, as it is for its base class, is meant to run an executable and to check its return code and standard output and error streams. The special features added are: * keep the standard output and standard error if the program is terminated by a signal (QMTest usually discards them, but we can profit from the automatic stack trace that is printed in case of a SEGFAULT) * allow a comparison of stdout and stderr with reference files, which may be overridden on specific platforms (by a convention) * if the command is a python script, it is executed through the python interpreter (for portability) * allow to specify in the test description a piece of code to validate the output of a test * define a working directory different from the one from which qmtest is run * allow to specify options directly in the test description The parameters that can be defined for the test are: $ Program ( =program=): This field indicates the path to the program. If it is not an absolute path, the value of the 'PATH' environment variable will be used to search for the program. If not specified, $GAUDIEXE or Gaudi.exe are used. $ Argument List ( =args=): If this field is left blank, the program is run without any arguments.%BR% Use this field to specify the option files.%BR% An implicit 0th argument (the path to the program) is added automatically. $ Options ( =options=): This field allows to pass a list of options to the main program without the need of a separate option file.%BR% The content of the field is written to a temporary file which name is passed the the application as last argument (appended to the field "Argument List"). $ Working Directory ( =workdir=): If this field is left blank, the program will be run from the qmtest directory, otherwise from the directory specified. $ Reference Output ( =reference=): Path to the reference file for standard output.%BR% If this field is left blank, any standard output will be considered valid.%BR% If the reference file is specified, any output on standard error is ignored. $ Reference for standard error ( =error_reference=): Path to the reference file for the standard error.%BR% If left blank, any output on the standard error will be considered a failure. $ Unsupported Platforms ( =unsupported_platforms=): Platform on which the test must not be run.%BR% List of regular expressions identifying the platforms on which the test is not run and the result is set to UNTESTED. $ Validator ( =validator=): Code to validate the output of the test.%BR% If defined, the function is used to validate the products of the test. If specified, overrides standard output, standard error and reference files. The code will have access to the following variables * self: the test class instance * stdout: the standard output of the executed test * stderr: the standard error of the executed test * result: the Result objects to fill with messages * causes: a list of causes of failure $ Standard Error ( =stderr=): Text to which the standard error has to be compared (overridden by =error_reference=) $ Prerequisite Tests: List of tests that should be run before the current test and should produce the specified outcome in order to allow the execution $ Standard Input: Optional text to be passed to the executable as standard input $ Time Out ( =timeout=): Maximum time allowed to the process to complete. The default is at least 600s, but you can specify a bigger integer number, or set the environment variable =QMTEST_IGNORE_TIMEOUT= to ignore the time-out (Gaudi specific extension). $ Temporary directory ( =use_temp_dir=): Boolean value to specify if the test has to be executed in a temporary directory, false by default. The temporary directory is shared among the tests executed but a single instance of QMTest (useful in case of dependencies between output of a test and input of anohter). $ Expected Exit Code (=exit_code=): Integer value representing exit code value expected from the program. If the program produces an exit code value different from this one, the test fails. (Note: few parameters are missing, but they are not very relevant) ---++ How to add tests to a package Including a package in the testing procedure is simple. The first thing is to add the line <verbatim> apply_pattern QMTest </verbatim> at the end of the =requirements= file, then the directory =tests/qmtest= has to be created in the package (e.g. !GaudiExamples/v19r5/tests/qmtest). With these two simple steps, the package will be included in the run of the tests described below. Of course this is not enough to actually run any test: we need to explain to QMTest which tests to run and how. This is simple if you know how to write QMTest test description files: it is enough to add them into the directory =tests/qmtest=. One needs to run <verbatim>make <project>/configure</verbatim> to make the infrastructure aware of the new test. ---+++ Best Practices When adding tests to a package is a good practice to group them in a directory with the same name of the package, all lower case, with extension =.qms= (see the sections about [[#How_to_define_which_tests_to_run][grouping tests]]). For example, for the package !GaudiKernel the tests description files should go in =tests/qmtest/gaudikernel.qms=. This will make it easier to find where a test is coming from when the results of a whole project are merged. Moreover, if such a directory exists, by default only the tests there specified will be run, giving the possibility of using another =.qms= directory for tests that should be run by hand. When writing a custom validator, it is important to take into account that the values added to the list =causes= must be a few short hints on the type of failure, while the =results= must contain enough details to (hopefully) understand what was the actual problem. For example, in a test that checks if some Python modules can be imported, for a failure we should add something like "import error" to =causes= and the full list of modules that could not be imported to an entry in the =results=, possibly with the Python stack trace. ---+++ Test description for QMTest In QMTest, each test is described with an XML file ending with the extension =.qmt= and with *only lowercase letters in the name*. I'm not going to explain all the possible variants of this file, I simply provide few examples to start with for Gaudi. ---++++ Run Gaudi with an option file The test is successful if nothing is printed on the standard error and if the return code is 0. <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="args"><set><text>path/to/options/myjob.opts</text></set></argument> </extension> </verbatim> To use more then one option file: <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="args"><set> <text>path/to/options/myjob1.opts</text> <text>path/to/options/myjob2.opts</text> </set></argument> </extension> </verbatim> ---++++ Run Gaudi with options (without an option file) The test is successful if nothing is printed on the standard error and if the return code is 0. <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="options"><text> from Gaudi.Configuration import * from Configurables import MyAlg alg = MyAlg(Cut = 10) ApplicationMgr().TopAlg.append(alg) </text></argument> </extension> </verbatim> ---++++ Run Gaudi with an option file and a reference output The test is successful if the standard output is equivalent to the reference file and if the return code is 0. <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="args"><set><text>path/to/options/myjob.opts</text></set></argument> <argument name="reference"><text>path/to/reference/file.ref</text></argument> </extension> </verbatim> If a platform specific reference file is needed it must have the name =path/to/reference/file.ref.XXX= where =XXX= has to be replaced by the first 3 letters of the value of CMTCONFIG. (This will probably change in the future) <noautolink> ---++++ Run a python script (GaudiPython) </noautolink> Almost identical to the two previous example with only one extra line: <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>path/to/myscript.py</text></argument> <argument name="args"><set><text>path/to/options/myjob.opts</text></set></argument> <argument name="reference"><text>path/to/reference/file.ref</text></argument> </extension> </verbatim> ---++++ Exclude a platform from a test Starting from the basic example, let's skip the execution of the test on Windows (which is usually the one creating problems) <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="args"><set><text>path/to/options/myjob.opts</text></set></argument> <argument name="unsupported_platforms"><set><text>win.*</text></set></argument> </extension> </verbatim> Note: the regular expression =win.*= means "every string that contains 'win' followed by any character (see the [[http://www.python.org/doc/current/lib/re-syntax.html][Regular Expression Syntax]] page in the Python documentation). ---++++ Run a test and check if a string is present in the standard output This is a bit complicated because it requires a custom validator for the test. Starting from the simple Gaudi test with option file: <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="args"><set><text>path/to/options/myjob.opts</text></set></argument> <argument name="validator"><text> expected_string = "The message I expect" if stdout.find(expected_string) == -1: causes.append('missing string') result['GaudiTest.expected_string'] = result.Quote(expected_string) </text></argument> </extension> </verbatim> Notes: * since the file is XML, the special characters have to be escaped a-la XML (e.g. ="= for ="=) * the messages added to =causes= will be printed in case of failure as "Unexpected =message=.", so, in this example, "Unexpected missing string." * the object =result= is used to pass/record useful information, in this example, if the test fails, we add the string we were expecting (the standard output is already added by default) ---++++ Run a test and check if chunk of lines is present in the standard output A function to find and compare a block in the standard output is available to the user-defined validator. Starting from the simple Gaudi test with option file: <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="args"><set><text>path/to/options/myjob.opts</text></set></argument> <argument name="validator"><text> block = """ MyAlg INFO Event 1 MyAlg INFO Event 2 MyAlg INFO Event 3 MyAlg INFO ==> Good Event MyAlg INFO Event 4 """ findReferenceBlock(block, signature_offset = 3) </text></argument> </extension> </verbatim> Notes: * =signature_offset= tells the function which line to use to recognize the presence of the block in the standard output (in the example it is the 4th non-empty line of the block: the one saying "Good Event"). ---++++ Use a different time-out for the test A test should not take a long time to be able to run of all the tests in a reasonable time. Unfortunately, sometime it is not possible, so, when the 10 min. default time-out is not enough, you can change that value in the .qmt file adding a line like: <verbatim> <argument name="timeout"><integer>1200</integer></argument> </verbatim> specifying the desired time-out period in seconds. The .qmt file could look like this: <verbatim> <?xml version="1.0" ?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="GaudiTest.GaudiExeTest" kind="test"> <argument name="program"><text>gaudirun.py</text></argument> <argument name="args"><set><text>path/to/options/myjob.opts</text></set></argument> <argument name="timeout"><integer>1200</integer></argument> </extension> </verbatim> ---++++ Test that have to be executed in a temporary directory When a test has to be executed in a temporary directory, it is enough to add the following line to the .qmt file.<br /><verbatim> <argument name="use_temp_dir"><enumeral>true</enumeral></argument></verbatim>Note that all the tests executed by an instance of QMTest are sharing the same temporary directory. It is possible to force the temporary directory to be a specific directory (for debugging) setting the environment variable =QMTEST_TMPDIR=. ---++++ Check that a non-zero exit code is produced by the test Add a line like <verbatim> <argument name="exit_code"><integer>1</integer></argument> </verbatim> to the =.qmt= file. The number is the expected return code, of course. You can find a list of valid exit codes in [[https://svnweb.cern.ch/trac/gaudi/browser/Gaudi/trunk/GaudiKernel/GaudiKernel/AppReturnCode.h][GaudiKernel/AppReturnCode.h]]. ---+++ Common examples of validators All the following examples must be surrounded by the lines <verbatim> <argument name="validator"><text> </verbatim> and <verbatim> </text></argument> </verbatim> ---++++ Check for presence of a reference block %SYNTAX{ syntax="python" }% block = """ MyAlg INFO Event 1 MyAlg INFO Event 2 MyAlg INFO Event 3 MyAlg INFO ==> Good Event MyAlg INFO Event 4 """ findReferenceBlock(block, signature_offset = 3) %ENDSYNTAX% In case of more than one reference blocks, one call to findReferenceBlock is needed for each of them, passing always a different =id=) ---++++ Check for presence of WARNING, ERROR and FATAL messages %SYNTAX{ syntax="python" }% countErrorLines() %ENDSYNTAX% The function =countErrorLines= accepts a dictionary as argument, to declare how many error messages to expect. E.g. %SYNTAX{ syntax="python" }% countErrorLines({"FATAL":2, "ERROR":7, }) %ENDSYNTAX% ---++ How to define which tests to run automatically and/or group of tests In QMTest, it is possible to define what is called a "suite" of tests. Of course it requires an XML file with an all lower-case name, but this time with the extension =.qms=. XML file for a suite of tests: <verbatim> <?xml version="1.0" ?> <!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="explicit_suite.ExplicitSuite" kind="suite"> <argument name="test_ids"><set> <text>mytest1</text> <text>mytest2</text> <text>another_test</text> </set></argument> </extension> </verbatim> XML file for a suite of suites: <verbatim> <?xml version="1.0" ?> <!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="explicit_suite.ExplicitSuite" kind="suite"> <argument name="suite_ids"><set> <text>mysuite1</text> <text>mysuite2</text> <text>another_suite</text> </set></argument> </extension> </verbatim> XML file for a mixed suite: <verbatim> <?xml version="1.0" ?> <!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'> <extension class="explicit_suite.ExplicitSuite" kind="suite"> <argument name="suite_ids"><set> <text>mysuite1</text> </set></argument> <argument name="test_ids"><set> <text>mytest1</text> <text>mytest2</text> </set></argument> </extension> </verbatim> An alternative way of defining a suite of tests is to create a directory with a lower-case name ending with =.qms= and put the =.qmt= (or other =.qms= files or directories) into it. In this case, the tests will be identified with =suitename.testsname= (without extensions). If a suite with the same name of the package (all lower-case) is found in the =tests/qmtest= directory, only the tests in it will be executed automatically. ---++ How to run the tests To run all tests: <verbatim>make test</verbatim> to run a single test: <verbatim>make test ARGS="-R test_davinci_tupling_advanced"</verbatim> (where test_davinci_tupling_advanced is an example test name). you may need =make configure= or =make purge && make= to get them to re-run. ---++ How to run the tests (obsolete) *WARNING: This section is outdated as CMT is not in use anymore.* To execute the tests of a package, you have to go into the =cmt= directory of that package and execute the following command: <verbatim> cmt TestPackage </verbatim> This will give an output like: <verbatim> #-------------------------------------------------------------- # Now trying [cmt qmtest_run] in .../Gaudi/vXrY/cmt (18/18) #-------------------------------------------------------------- -tag_add=QMTest Execute action qmtest_run => python .../GaudiPolicy/vXrY/cmt/fragments/run_qmtest.py Gaudi ==========> Running tests for package Gaudi ==========> Entering '../tests/qmtest' ==========> Running 'qmtest run -o ../../slc4_ia32_gcc34_dbg/results.qmr' --- TEST RESULTS ------------------------------------------------------------- import_opts : PASS import_py : PASS import_py_err : PASS --- TESTS THAT DID NOT PASS -------------------------------------------------- None. --- STATISTICS --------------------------------------------------------------- 3 tests total 3 (100%) tests PASS </verbatim> To run a specific test (or a set of them) it is enough to pass them as arguments to the action: <verbatim> cmt TestPackage mytest1 mytest2 </verbatim> Note that if the tests are inside a directory test suite, you should specify the name of the test as it is printed by QMTest, i.e. with all the directory names preceding the test name, without extensions and replacing =/= (slash) with =.= (dot). For example, the test =mypackage.qms/test_group1.qms/a_test.qmt= can be run with: <verbatim> cmt TestPackage mypackage.test_group1.a_test </verbatim> If you have the pattern =QMTestSummarize= in your requirements, you can execute all the test of the project and collect the results in one go with <verbatim> cmt TestProject </verbatim> ---+++ Running the tests from a read/only location It is possible to run the tests using a read/only (shared) installation of Gaudi. Just set the environment variable =QMTESTRESULTSDIR= to point to a writable directory, before going to a package cmt directory (e.g. =GaudiRelease/cmt=) and run <verbatim> cmt TestPackage </verbatim> or <verbatim> cmt TestProject </verbatim> ---++ Conclusion I didn't cover all possible aspects of the testing infrastructure for Gaudi, but what is here should be enough to start and go a rather long way before the need of more details. If you reach that point, drop me and e-mail ;-) Note: Updated for Gaudi v21r11 -- Main.MarcoClemencic - 2009-10-12
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r20
<
r19
<
r18
<
r17
<
r16
|
B
acklinks
|
V
iew topic
|
WYSIWYG
|
M
ore topic actions
Topic revision: r20 - 2021-06-28
-
PatrickSKoppenburg
Log In
Gaudi
Gaudi Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Public webs
Public webs
ABATBEA
ACPP
ADCgroup
AEGIS
AfricaMap
AgileInfrastructure
ALICE
AliceEbyE
AliceSPD
AliceSSD
AliceTOF
AliFemto
ALPHA
Altair
ArdaGrid
ASACUSA
AthenaFCalTBAna
Atlas
AtlasLBNL
AXIALPET
CAE
CALICE
CDS
CENF
CERNSearch
CLIC
Cloud
CloudServices
CMS
Controls
CTA
CvmFS
DB
DefaultWeb
DESgroup
DPHEP
DM-LHC
DSSGroup
EGEE
EgeePtf
ELFms
EMI
ETICS
FIOgroup
FlukaTeam
Frontier
Gaudi
GeneratorServices
GuidesInfo
HardwareLabs
HCC
HEPIX
ILCBDSColl
ILCTPC
IMWG
Inspire
IPv6
IT
ItCommTeam
ITCoord
ITdeptTechForum
ITDRP
ITGT
ITSDC
LAr
LCG
LCGAAWorkbook
Leade
LHCAccess
LHCAtHome
LHCb
LHCgas
LHCONE
LHCOPN
LinuxSupport
Main
Medipix
Messaging
MPGD
NA49
NA61
NA62
NTOF
Openlab
PDBService
Persistency
PESgroup
Plugins
PSAccess
PSBUpgrade
R2Eproject
RCTF
RD42
RFCond12
RFLowLevel
ROXIE
Sandbox
SocialActivities
SPI
SRMDev
SSM
Student
SuperComputing
Support
SwfCatalogue
TMVA
TOTEM
TWiki
UNOSAT
Virtualization
VOBox
WITCH
XTCA
Cern Search
TWiki Search
Google Search
Gaudi
All webs
Copyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use
Discourse
or
Send feedback