Running Moore

Broadly speaking, Moore is a repository of Python files which can be used to configure a Gaudi application. In this way, “running Moore” is the same as running any other LHCb application (such as Brunel or DaVinci):

lb-run Moore/latest gaudirun.py some_options_file.py

The interesting part is the contents of the options files passed to gaudirun.py, which can define application configuration such as for HLT1 or HLT2.

This page details some of the specifics of running these applications, which can be useful if you’re developing Moore itself or authoring a trigger selection.

Choosing a version

Versioned releases of the Moore project are made periodically. Releases are required to run Moore in the Online environment and in productions on the Grid.

Releases can be useful for individuals to quickly run options files. The version of Moore you choose depends on the task: if you want to reproduce a particular result then choose the same version, but if you just want to run something then choosing the latest version is usually sufficient.

Note

When changing between versions, it’s useful to check the release notes. These list the changes made since the previous release.

Released versions of Moore can be run using lb-run with the version number you’re interested in:

lb-run Moore/v50r0 gaudirun.py some_options_file.py

If you want to be able to make changes to Moore, you should consult the Developing Moore page for instructions on how to get started. Once that’s done, you’ll be able to run Moore like this:

./Moore/run gaudirun.py some_options_file.py

From now on, this page will assume you’re running your own development version of Moore. If that’s not the case, you can still run the examples beginning with ./Moore/run by replacing this part with lb-run Moore/<version>.

The first run

We’re going to end up with an options file that runs HLT1 (yes, really!). Let’s start by placing these lines into a file called hlt1_example.py:

from Moore import options, run_moore


def make_lines():
    return []


run_moore(options, make_lines)

What are all these things we’re importing, and why are we defining this almost-empty function? It’s always worth remembering that this is just Python, so we can inspect objects and poke around as usual. Just use python -i to run the code (rather than gaudirun.py to run the application) and get an interactive prompt:

$ ./Moore/run python -i hlt1_example.py
<...some logs...>
>>> help(options)
Help on ApplicationOptions in module PyConf.application object:

class ApplicationOptions(GaudiKernel.Configurable.ConfigurableUser)
 |  Holder for application configuration.
 |
 |  Configuration can be mutated until `.finalize()` is called. At that
 |  point any dynamic defaults based on other properties are resolved.
 |

We’ve used the help() builtin to see that the options object is a “Holder for application configuration” (an instance of ApplicationOptions). What parameters it holds can be seen by printing it:

>>> print(options)
/***** User ApplicationOptions/ApplicationOptions **************************************************
|-use_iosvc         = False  (default: False)
|-histo_file        = ''  (default: '')
|-data_type         = 'Upgrade'  (default: 'Upgrade')
|-input_type        = ''  (default: '')
|-output_file       = ''  (default: '')
|-n_event_slots     = 1  (default: -1)
|-simulation        = True  (default: True)
|-output_level      = 3  (default: 3)
|-event_store       = 'HiveWhiteBoard'  (default: 'HiveWhiteBoard')
|-data_flow_file    = 'data_flow.gv'  (default: '')
|-input_raw_format  = 0.5  (default: 0.5)
|-dddb_tag          = ''  (default: '')
|-control_flow_file = 'control_flow.gv'  (default: '')
|-output_type       = ''  (default: '')
|-n_threads         = 1  (default: 1)
|-conddb_tag        = ''  (default: '')
|-evt_max           = -1  (default: -1)
|-input_files       = []  (default: [])
\----- (End of User ApplicationOptions/ApplicationOptions) -----------------------------------------

These are fairly self explanatory if you’ve run a LHCb applications before. Most of the default values are sufficient for running Moore.

Next, run help(run_moore) and read the documentation for Moore.run_moore. You’ll see that this is an important helper function; it will do some work behind the scenes to set up the full application configuration, given the options object we’ve just been looking at and a make_lines function.

The line-maker function we’ve written in our example returns an empty list, so we’re not running any specific algorithms. Let’s exit the Python shell and try running the application:

./Moore/run gaudirun.py hlt1_example.py

In the log that follows, you’ll see an error message!:1

PyConf.utilities.ConfigurationError: Required options not set: ['input_type', 'dddb_tag', ‘conddb_tag']

Moore is telling us, rather cryptically, that it doesn’t have all the information it needs to run our options. In this case, it’s because it doesn’t know what type of input data we have. Without knowing about the input data, which defines, for example, how the raw event is laid out, it cannot unambiguously configure the application context.

Note

Even though we haven’t told Moore to run anything, it will run some default algorithms such as DecReport and SelReport creators, and an output writer. These algorithms need to know what the input data looks like, so cannot be configured without us giving Moore some input data.

To remedy this, we can use some input data from the test file database. If you look at the output of help(options), you’ll see the options.set_input_and_conds_from_testfiledb method can help us.

Create a new options file called input_data.py with the following contents:

from Moore import options

options.evt_max = 100
options.set_input_and_conds_from_testfiledb("MiniBrunel_2018_MinBias_FTv4_MDF")

Aside from the input, this also sets the conditions to run under. The conditions define how to correctly interpret the input data (e.g. how the IDs in the raw data translate to detector geometry) but also how to process them (e.g. which VELO alignment constants to use, which affects the VELO tracking). Most often the conditions that you’ll use are those used during data taking or for the MC production, however they can be different in specific cases.

Re-run the application, now including the additional options file:

./Moore/run gaudirun.py input_data.py hlt1_example.py

Note

The order of options files is important. Because the run_moore function is called in hlt1_example.py, we must do all configuration before that point. The gaudirun.py script will execute each options file one-by-one, so we put other files before the ‘main’ options file.

The application will run, but comes to an abrupt stop with an error message:

assert hlt1 ^ hlt2 ^ spruce ^ passthrough, 'Expected exclusively all Hlt1, all Hlt2, all Spruce or all Pass lines' \
AssertionError: Expected exclusively all Hlt1, all Hlt2, all Spruce or all Pass lines

This is actually good news! The Python configuration completed successfully and the application began running. A C++ algorithm then stopped the execution because, quite rightly, we didn’t define any lines to run. So let’s do that!

Note

Understand the you are now running Moore. The general pattern of the two options files you’ve created is important to grasp:

1. Application-wide parameters are defined on the options object. This is accessible across each options file you pass to gaudirun.py (it is global).

2. You explictly configure a Moore application using the run_moore function. This takes the application-wide options object and a function that creates the Control flow and (by extension) the data flow, and configures required services.

Running all HLT1 lines

The way a line is defined isn’t particularly important here, so we’ll just use a helper that gives us all default HLT1 lines. More information on writing lines can be found on the Writing an HLT2 line page.

In our hlt1_example.py file, we’ll using a function already in Moore called all_lines:

from Moore import options, run_moore
from Hlt1Conf.settings import all_lines

run_moore(options, all_lines)

Run the application as before, and you’ll see it completes successfully. 🎉

This covers the important aspects of of getting up and running with Moore. If you’re used to configuring applications like DaVinci, things are much more explicit, so although it may feel very different the idea is that the configuration as a whole is easier to reason about.

Analysing the output

Moore comes with several tools for debugging applications, but the most important is always the application log, that is the print-out you see in your terminal. There is a wealth of information in the log, so it’s worth getting familiar with it.

Control flow table

Near the bottom of the log is a representation of the control flow tree. The top level node, moore, contains the hlt_decision node and the report_writers node. The former contains one node per line as defined by the all_lines function we passed to run_moore, whilst the latter contains algorithms for writing output files. Child nodes can be identified by their indentation.

LAZY_AND: moore                                                                   #=100     Sum=5           Eff=|( 5.000000 +- 2.17945 )%|
 NONLAZY_OR: hlt_decision                                                         #=100     Sum=5           Eff=|( 5.000000 +- 2.17945 )%|
  LAZY_AND: Hlt1TrackMVA                                                          #=100     Sum=3           Eff=|( 3.000000 +- 1.70587 )%|
   DeterministicPrescaler/DeterministicPrescaler                                  #=100     Sum=100         Eff=|( 100.0000 +- 0.00000 )%|
   PrGECFilter/PrGECFilter                                                        #=100     Sum=96          Eff=|( 96.00000 +- 1.95959 )%|
   VoidFilter/require_pvs                                                         #=96      Sum=95          Eff=|( 98.95833 +- 1.03623 )%|
   PrFilter__PrFittedForwardTracksWithPVs/PrFilter__PrFittedForwardTracksWithPVs  #=95      Sum=3           Eff=|( 3.157895 +- 1.79419 )%|
  LAZY_AND: Hlt1TwoTrackMVA                                                       #=100     Sum=1           Eff=|( 1.000000 +- 0.994987)%|
   <...>
  <...>
 NONLAZY_OR: report_writers                                                       #=5       Sum=5           Eff=|( 100.0000 +- 0.00000 )%|
  bankKiller/bankKiller                                                           #=5       Sum=5           Eff=|( 100.0000 +- 0.00000 )%|
  ExecutionReportsWriter/ExecutionReportsWriter                                   #=5       Sum=5           Eff=|( 100.0000 +- 0.00000 )%|
  HltDecReportsWriter/HltDecReportsWriter                                         #=5       Sum=5           Eff=|( 100.0000 +- 0.00000 )%|

(You’ll need to scroll horizontally to see the full table.)

Control flow nodes can, unsurprisingly, control the execution flow of the application by returning an ‘accept’ or ‘reject’ status. The fraction of ‘accept’ statuses from each node is given in the right-most column, and is relative to the number of times the node was executed.

Each control flow node can have child algorithms and child control flow nodes. How a parent decides which of its child to execute, given their return statuses, is determined by its node logic, as enumerated in PyConf.control_flow.NodeLogic.

The control flow table is extremely useful in understanding how the application is filtering events. We can see that HLT1 accepted 5 events in total in our example, with 3 of them being accepted by at least the Hlt1TrackMVA (again, the numbers may be slightly different depending on the version of Moore).

Data and control flow graphs

You can get Moore.run_moore to generate data and control flow graphs if you manually configure it with:

options.control_flow_file = 'control_flow.gv'
options.data_flow_file = 'data_flow.gv'

The outputs are Graphviz source files, which can be converted into images.

To convert the source files, the dot program must be available on your system. It is installed on lxplus, so you can transfer the source files there if you don’t have it on your development machine. An example command to generate a PDF of the control flow graph is:

dot -Tpdf control_flow.gv > control_flow.pdf

You can similarly generate PNG, SVG, and other file types.

The data flow graph for our example looks like this:2

The control flow graph looks like this:

Looking at these graphs can help you to visualise how your selections are connected to the reconstruction, and how the control flow is defined.

Footnotes

1

The printout of the errors encountered throughout this tutorial may depend on the version of Moore. The error listed here corresponds to (among others) v53r4.

2

These graphs are generated automatically each time Moore’s master branch changes, so what you see here is the most up-to-date reconstruction flow we have.