Developing Moore
Whether you want to edit the C++ algorithms used in Moore or Moore’s Python
configuration, it’s generally simplest to use the lb-stack-setup utility.
The README provided on the project homepage will guide you
through steps that will give you full check-outs of Moore and all of it’s LHCb
and Gaudi dependencies (such as the Phys and Rec projects). You are then free
to edit any file in any project, re-build, and re-run Moore to see what
changes!
When editing files that live in Moore, there are a few conventions
to follow which are listed on this page, but before then it’s important to
check the edits don’t break the tests!
Running tests
A good way to start developing is to run the tests, so-called ‘QM tests’, which
are run automatically in the nightly build system. This has the big advantage
that you start from an options file which is guaranteed to work and you can
test yourself that you are not introducing any unwanted changes.
You can find out which tests are available by running:
make Moore/test ARGS='-N'
The specification of most tests is done in files with a .qmt extension,
which are located in directories like Hlt/Hlt2Conf/tests/qmtest. The name
of a test is given by the directory it is in and the filename. Inside the
.qmt file you will find the options which are executed by the test and what
the requirements for the test are to pass.
To run a test, the following options are available:
# Show all available tests.
make Moore/test ARGS='-N'
# Run all tests
make Moore/test
# Run all tests with 4 parallel jobs
make Moore/test ARGS='-j 4'
# Run test(s) matching a regular expression
make Moore/test ARGS='-R hlt1_example$'
# Verbose output showing test (failure) details
make Moore/test ARGS='-R hlt1_example -V'
# Run test(s) N to M
make Moore/test ARGS='-I N,M'
# Run failed tests again
make fast/Moore/test ARGS='--rerun-failed'
# Reconfigure Moore after adding additional tests
make Moore/configure
You can also use make fast/Moore/test … if you have only changed code in the
Moore project. This skips checking upstream projects for changes.
The log files of a test can be found in the directory Moore/build.<platform>/html/<name of test>.
The options of a test, without executing the validators, can also be run by hand with gaudirun.py:
./Moore/run gaudirun.py Moore/<Hat>/<Package>/tests/qmtest/<test name>.qmt
With the introduction of lbexec and qmtexec commands any test can be run using the qmtexec command:
./Moore/run qmtexec Moore/<Hat>/<Package>/tests/qmtest/<test name>.qmt
A more detailed description can be found on the Gaudi testing infrastructure
TWiki page (which is outdated in some places).
Authoring tests
Often you will need to modify an existing QM test or add a new one.
To add a new test, just create a new file in the tests/qmtest directory of
the package that will host it.
Make it look like the example shown below, re-configure with make Moore/configure
to make your new test known to the build system, and run using the instructions above.
You can find a detailed specification of the available features on the
Gaudi testing infrastructure TWiki page (which is outdated in some places),
but you can also look at the Gaudi or LHCb testing
infrastructure code.
Let’s dissect the test definition of the hlt1_reco_baseline.qmt test in
Hlt/RecoConf as a typical example:
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE extension PUBLIC '-//QM/2.3/Extension//EN' 'http://www.codesourcery.com/qm/dtds/2.3/-//qm/2.3/extension//en.dtd'>
<!-- (c) Copyright ... -->
<!--
Make sure HLT1 baseline reconstruction does not change.
-->
<extension class="GaudiTest.GaudiExeTest" kind="test">
<argument name="program"><text>gaudirun.py</text></argument>
<argument name="args"><set>
<text>$MOOREROOT/tests/options/mdf_input_and_conds.py</text>
<text>$MOOREROOT/tests/options/download_mdf_input.py</text>
<text>$RECOCONFROOT/options/hlt1_reco_baseline.py</text>
</set></argument>
<argument name="use_temp_dir"><enumeral>true</enumeral></argument>
<argument name="reference"><text>../refs/hlt1_reco_baseline.ref</text></argument>
<argument name="validator"><text>
from Moore.qmtest.exclusions import ref_preprocessor
validateWithReference(preproc=ref_preprocessor)
from Moore.qmtest.exclusions import remove_known_warnings
countErrorLines({"FATAL": 0, "ERROR": 0, "WARNING": 0},
stdout=remove_known_warnings(stdout))
</text></argument>
</extension>
You can see some XML boilerplate at the top, the copyright and a useful
description of the intent of the test.
The program that is run is most often gaudirun.py or sometimes python
(e.g. for GaudiPython scripts).
The arguments follow after that, where each of them must be put in a separate
<text> tag (spaces would be interpreted as part of the argument, the same
as if you quote them in the shell).
We set the use_temp_dir flag so that tests output in a temporary directory
in the build directory (e.g. build.x86_64-centos7-gcc9-opt/Hlt/RecoConf/tests_tmp
for this test).
The most important part of the test definition is the validator, which is
the logic that defines the condition for success (and failure) of the test.
In most tests you will write a validator explicitly, and for that you use Python
code embedded in the XML as shown above.
There are a couple of frequently used validation patterns.
First, every test should check that no warning, error or fatal messages appear
in the standard output. This is done with the code:
from Moore.qmtest.exclusions import remove_known_warnings
countErrorLines({"FATAL": 0, "ERROR": 0, "WARNING": 0},
stdout=remove_known_warnings(stdout))
You can notice that the standard output is preprocessed with the callable
remove_known_warnings, defined in exclusions.py.
The purpose of this central list of exclusions is to temporarily mask warnings
that do not affect the correctness of running the application.
It is important that a reasonable attempt is made to fix the source of the
warning before extending the central list, and if the list is extended, it
should point to an issue where the resolution of the underlying problem is
tracked.
Warnings that are expected only in a particular test, but would be problematic
if any other test produces them, should not be added to exclusions.py.
Error and fatal messages should never be ignored but they should be addressed
at their source.
Second, many tests will validate correctness by comparing to a reference.
This is done with:
from Moore.qmtest.exclusions import ref_preprocessor
validateWithReference(preproc=ref_preprocessor)
Here ref_preprocessor (from exclusions.py) is used to filter the “noise”
from the standard output, which consists of parts that will vary between runs,
such as the timing table, the host on which the test runs, etc.
In Moore ref_preprocessor also removes everything before the finalization
of the job, which is where the most interesting parts reside (counters, PrChecker
tables, etc.)
From what is left, counters, histogram and TTree summaries are extracted and
compared to those in the reference (../refs/hlt1_reco_baseline.ref),
and everything else is compared line by line.
You might need to debug the test interactively, especially if you define a more
complex validator.
Since the test runner (CTest) captures the standard output, if you try to use
pdb or similar tools, the test will seemingly hang when you run it with
make Moore/test ... waiting for input.
The way to overcome this problem is to run the test manually, by extracting
the relevant command using make Moore/test ARGS='-R ... -V -N':
make fast/Moore/test ARGS='-R hlt1_reco_baseline$ -V -N'
# ...
# 20: Test command: /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/529/stable/x86_64-centos7/bin/xenv "--xml" "/home/rmatev/stack/Moore/build.x86_64-centos7-gcc9-opt/config/Moore-build.xenv" "python" "-m" "GaudiTesting.Run" "--skip-return-code" "77" "--report" "ctest" "--common-tmpdir" "/home/rmatev/stack/Moore/build.x86_64-centos7-gcc9-opt/Hlt/RecoConf/tests_tmp" "--workdir" "/home/rmatev/stack/Moore/Hlt/RecoConf/tests/qmtest" "/home/rmatev/stack/Moore/Hlt/RecoConf/tests/qmtest/hlt1_reco_baseline.qmt"
# ...
Moore/run "python" "-m" "GaudiTesting.Run" "--skip-return-code" "77" "--report" "ctest" "--common-tmpdir" "/home/rmatev/stack/Moore/build.x86_64-centos7-gcc9-opt/Hlt/RecoConf/tests_tmp" "--workdir" "/home/rmatev/stack/Moore/Hlt/RecoConf/tests/qmtest" "/home/rmatev/stack/Moore/Hlt/RecoConf/tests/qmtest/hlt1_reco_baseline.qmt"
Coding conventions
When writing Python in Moore, you should try to follow the PEP8 guidelines and other general good practices:
Include comments and documentation when the intent of a line, function,
class, or module is not obvious.
Factor code into functions when there is repetition and/or for clarity.
When modifying an existing file, following the conventions of the surrounding
code.
For examples of how to do things, just have a look around what already exists
in Moore.
Continuous integration
Whenever a new commit is made to the Moore repository, a CI pipeline runs
that performs some style and syntax checks:
The LHCb copyright statement should be present at the top of every source file.
The LHCb Python formatting rules must be adhered to.
The Python code must have valid syntax and not raise any flake8 error codes.
Timing, throughput and profiling
An important question comes up sooner or later when developing an algorithm or making changes to an existing one: “How fast is this thing?”.
In many scenarios, the timing table that is printed at the end of a job can give a sufficient estimate.
The timing table prints the total and average time per event for every algorithm,
and it is advisable to compare the timing changes of your algorithm relative to some algorithm earlier in
the reconstruction chain, like the Velo tracking.
If you need more details and for example want to create a flamegraph, we recommend following the instructions in the PRConfig repository.
If you want to use valgrind tools (e.g. callgrind) to profile your code, see the docstring of the PyConf.application.make_callgrind_profile
function and follow the instructions in the “Job Options Driven Profiling” section of the CodeAnalysisTools Twiki page.
Footnotes