Running HLT1 and HLT2 over simulation
This tutorial assumes that you are working within a local copy of the LHCb software stack, as introduced in the Developing Moore tutorial.
With a compiled stack, the process of running HLT2 over simulation breaks down into two steps:
Pick an existing/write a new options file defining the run
Run!
Once you have an options file (e.g. at Moore/my_dir/my_options.py), step 2 is as simple as:
Moore/run gaudirun.py Moore/my_dir/my_options.py
Which will produce a very informative log detailing what happened - hopefully ending with:
ApplicationMgr INFO Application Manager Finalized successfully
ApplicationMgr INFO Application Manager Terminated successfully
and any output files that you requested (more on that below). There are tips on analysing the log output in the Analysing HLT2 output section of the docs.
The following sections below give some instructions on step 1. There are different sections depending on whether you are running over “old” (produced before around mid-2022) simulation, or newer simulation samples. The latter are compatible with the DD4HEP conditions database, while the former are not. In both cases though, running Moore is done in the same way as shown above.
With the help of these examples, hopefully you will be able to run Moore over any LHCb simulation sample you can find. However, different samples will generally require configuration to be passed. The Extra configuration parameters you may need section goes over a few common cases of different configuration that you may need. See also the Input samples with different attributes section of the documentation. If things still aren’t working, the Upgrade HLT2 Mattermost channel is a good place to ask for help.
Finally, the last section of this page points to some modern simulation samples that you can use.
Tip
gaudirun.py can accept as many options files as you like e.g.
Moore/run gaudirun.py my_options1.py my_options2.py and so on. This is
useful if you wish to factorise the options: one use case might be that you
wish to run the same set of HLT2 trigger lines over multiple samples. You
can have one options file defining only the trigger lines, and then a few
different options files defining the sample. Running over different samples
then means changing only the latter options - no code need be duplicated.
Instructions for the stack with “old” simulation
If the simulation sample you wish to run over was produced before mid-2022
(roughly), then the default platform of the LHCb stack will not be compatible.
This is because the stack now uses the DD4HEP condition database, as opposed to
the detdesc conditions database used before. Therefore, to run HLT2 over
this “old” simulation, we must compile the stack with a (non-default)
detdesc-compatible platform.
Switching to a detdesc-compatible platform
Using either an existing stack, or one you have freshly downloaded, you can find
instructions to switch platforms in the “Change the platform” section of the
lb-stack-setup README. A wise choice of platform would be one that is
compiling and testing successfully in the nightly builds - just look for one
that has the +detdesc piece to right of the compiler e.g.
x86_64_v3-centos7-gcc12+detdesc-opt+g.
With the steps in the README completed, it should be sufficient to make
Moore again, however if you already have a stack fully compiled, you may have
to do a make purge first.
With the stack (re-)installed, you just need to write an options file that sets the configuration of the job you want to run. Writing options files for HLT1 and HLT2 are the subjects of the next sections.
Options files for running HLT1 over simulation
See also the HLT1->HLT2 section of the documentation.
A basic, tested options file for running Allen (as compiled for CPU and
steered as Gaudi algorithm through Moore) with the LHCb stack is
hlt1_allen_via_moore_example.py:
###############################################################################
# (c) Copyright 2024 CERN for the benefit of the LHCb Collaboration #
# #
# This software is distributed under the terms of the GNU General Public #
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". #
# #
# In applying this licence, CERN does not waive the privileges and immunities #
# granted to it by virtue of its status as an Intergovernmental Organization #
# or submit itself to any jurisdiction. #
###############################################################################
from Moore.config import run_allen
from Moore.options import options
from RecoConf.hlt1_allen import allen_gaudi_config as allen_sequence
from RecoConf.decoders import default_ft_decoding_version
default_ft_decoding_version.global_bind(value=4)
options.set_input_and_conds_from_testfiledb(
"MiniBrunel_2018_MinBias_FTv4_DIGI")
options.evt_max = 100
options.output_file = 'HLT1.dst'
options.output_type = 'ROOT'
with allen_sequence.bind(sequence="hlt1_pp_forward_then_matching_no_ut"):
run_allen(options)
After the copyright block and the block of imports, the options file then sets configuration parameters relating to the sample we want to run over (here using a sample stored in the TestFileDB), how many events to run over, and some details about the output you wish to receive.
In the case of HLT1, the Allen “sequence” defines the reconstruction
configuration and the trigger lines that will run. The retentions of each line
can be seen in the list of counters for GaudiAllenCountAndDumpLineDecisions,
looking something like:
GaudiAllenCountAndDumpLineDecisi... INFO Number of counters : 67
| Counter | # | sum | mean/eff^* | rms/err^* | min | max |
|*"Selected by Hlt1BeamGasDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1D2KKDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1D2KPiAlignmentDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1D2KPiDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1D2PiPiDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonDrellYanDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonDrellYan_SSDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonDrellYan_VLowMassDecision"| 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonDrellYan_VLowMass_SSDecision"| 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonHighMassAlignmentDecision"| 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonHighMassDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonJpsiMassAlignmentDecision"| 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonLowMassDecision" | 100 | 1 |( 1.000000 +- 0.9949874)% |
|*"Selected by Hlt1DiMuonNoIPDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonNoIP_ssDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiMuonSoftDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DiPhotonHighMassDecision" | 100 | 2 |( 2.000000 +- 1.400000)% |
|*"Selected by Hlt1DisplacedDiMuonAlignmentDecision"| 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DisplacedDiMuonDecision" | 100 | 1 |( 1.000000 +- 0.9949874)% |
|*"Selected by Hlt1DisplacedDielectronDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1DisplacedLeptonsDecision" | 100 | 1 |( 1.000000 +- 0.9949874)% |
|*"Selected by Hlt1ErrorBankDecision" | 100 | 0 |( 0.000000 +- 0.000000)% |
|*"Selected by Hlt1GlobalDecision" | 100 | 17 |( 17.00000 +- 3.756328)% |
...
Options files for running HLT2 over simulation
A very simple options file that is tested in the nightlies is hlt2_all_lines_with_reco.py, which looks like
###############################################################################
# (c) Copyright 2019 CERN for the benefit of the LHCb Collaboration #
# #
# This software is distributed under the terms of the GNU General Public #
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". #
# #
# In applying this licence, CERN does not waive the privileges and immunities #
# granted to it by virtue of its status as an Intergovernmental Organization #
# or submit itself to any jurisdiction. #
###############################################################################
"""Options for running HLT2 lines with on-the-fly reconstruction.
Run like any other options file:
./Moore/run gaudirun.py hlt2_all_lines.py
"""
from Moore import options, run_moore
from RecoConf.global_tools import stateProvider_with_simplified_geom
from RecoConf.reconstruction_objects import reconstruction
from Hlt2Conf.lines import all_lines
import re
options.evt_max = 200
options.output_file = 'hlt2_all_lines_with_reco.dst'
options.output_type = 'ROOT'
options.output_manifest_file = "hlt2_all_lines_with_reco.tck.json"
# temporarily use HiveWhiteBoard, see lhcb/LHCb!2878
options.use_iosvc = False
options.event_store = 'HiveWhiteBoard'
def remove_lines(lines_dict, pattern_to_remove):
filtered = {
name: line
for name, line in lines_dict.items()
if re.match(pattern_to_remove, name) is None
}
print("Removed lines: ", set(lines_dict) - set(filtered))
return filtered
# Remove lines which either use gamma or pi0 or jets.
# gamma and pi0 can be added back when Moore!593 is merged.
pattern_to_remove = "(?i)(hlt2jets)"
hlt2_lines = remove_lines(all_lines, pattern_to_remove)
# Remove lines which require hlt1_filter_code
to_remove = [
'Hlt2IFT_SMOG2GECPassthrough', 'Hlt2IFT_SMOG2LumiPassthrough',
'Hlt2IFT_SMOG2MBPassthrough', 'Hlt2IFT_Femtoscopy_InclLambdaLL',
'Hlt2IFT_Femtoscopy_InclXiLLL', 'Hlt2IFT_Femtoscopy_InclOmegaLLL',
'Hlt2IFT_Femtoscopy_LambdaP', 'Hlt2IFT_Femtoscopy_LambdaP_lowK',
'Hlt2IFT_Femtoscopy_LambdaLambda', 'Hlt2IFT_Femtoscopy_LambdaLambda_lowK',
'Hlt2IFT_Femtoscopy_XiP', 'Hlt2IFT_Femtoscopy_XiP_lowK',
'Hlt2IFT_Femtoscopy_XiLambda', 'Hlt2IFT_Femtoscopy_XiLambda_lowK',
'Hlt2IFT_Femtoscopy_XiXi', 'Hlt2IFT_Femtoscopy_OmegaP',
'Hlt2IFT_Femtoscopy_OmegaP', 'Hlt2IFT_Femtoscopy_OmegaP_lowK',
'Hlt2IFT_Femtoscopy_OmegaLambda', 'Hlt2IFT_Femtoscopy_OmegaXi',
'Hlt2IFT_Femtoscopy_OmegaOmega', 'Hlt2QEE_DiElectronPrompt_PersistPhotons',
'Hlt2QEE_DiElectronPrompt_PersistPhotons_FULL',
'Hlt2QEE_DiElectronDisplaced_PersistPhotons',
'Hlt2QEE_DiElectronDisplaced_PersistPhotons_FULL',
'Hlt2QEE_DiElectronPrompt_PersistPhotonsSS',
'Hlt2QEE_DiElectronDisplaced_PersistPhotonsSS', 'Hlt2QEE_IncJet10GeV',
'Hlt2QEE_IncDiJet10GeV', 'Hlt2QEE_IncDiJet15GeV', 'Hlt2QEE_IncDiJet20GeV',
'Hlt2QEE_IncDiJet25GeV', 'Hlt2QEE_IncDiJet30GeV', 'Hlt2QEE_IncDiJet35GeV'
]
for remove in to_remove:
hlt2_lines = remove_lines(hlt2_lines, remove)
print("Number of HLT2 lines {}".format(len(hlt2_lines)))
def make_lines():
return [builder() for builder in hlt2_lines.values()]
public_tools = [stateProvider_with_simplified_geom()]
with reconstruction.bind(from_file=False):
config = run_moore(options, make_lines, public_tools)
Similarly to above, there are blocks of code for copyright, imports, and
configuration for the sample, number of events to run over and details of the
output sample. Unlike HLT1, where the sequence defines the lines that run, a
make_lines function is commonly used in options files to define what trigger
lines you wish to run. Here, the easy choice is taken to run all HLT2 lines
registered in Moore at this time. The final block
public_tools = [stateProvider_with_simplified_geom()]
with reconstruction.bind(from_file=False):
config = run_moore(options, make_lines, public_tools)
finalises the configuration, ending in a call to run_moore. Note here the line
which tells us that the reconstructed information stored in the file is used. In
general, you’ll want to set this to False and run the current HLT2
reconstruction in your job.
A slightly more complicated and more typical options file (that is also tested in the nightlies, and therefore should work out-of-the-box) is hlt2_example.py:
###############################################################################
# (c) Copyright 2019 CERN for the benefit of the LHCb Collaboration #
# #
# This software is distributed under the terms of the GNU General Public #
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". #
# #
# In applying this licence, CERN does not waive the privileges and immunities #
# granted to it by virtue of its status as an Intergovernmental Organization #
# or submit itself to any jurisdiction. #
###############################################################################
"""Options for running HLT2 lines.
Run like any other options file:
./Moore/run gaudirun.py hlt2_example.py
"""
from Moore import options, run_moore
from Hlt2Conf.lines.test.hlt2_test import dzero2kpkm_line, dstarp2dzeropip_dzero2kpkm_line, b2d0mu_dzero2kpkm_line
from RecoConf.global_tools import stateProvider_with_simplified_geom
# TODO stateProvider_with_simplified_geom must go away from option files
input_files = [
# MinBias 30000000
# sim+std://MC/Upgrade/Beam7000GeV-Upgrade-MagDown-Nu7.6-25ns-Pythia8/Sim09c-Up02/Reco-Up01/30000000/LDST
# 'root://xrootd.echo.stfc.ac.uk/lhcb:prod/lhcb/MC/Upgrade/LDST/00069155/0000/00069155_00000878_2.ldst'
# D*-tagged D0 to KK, 27163002
# sim+std://MC/Upgrade/Beam7000GeV-Upgrade-MagDown-Nu7.6-25ns-Pythia8/Sim09c-Up02/Reco-Up01/27163002/LDST
'root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/MC/Upgrade/LDST/00070317/0000/00070317_00000033_2.ldst'
]
options.input_files = input_files
options.input_type = 'ROOT'
options.input_raw_format = 4.3
# When running from Upgrade MC, must use the post-juggling locations of the raw
# event
options.evt_max = 100
options.simulation = True
options.data_type = 'Upgrade'
options.dddb_tag = 'dddb-20171126'
options.conddb_tag = 'sim-20171127-vc-md100'
options.geometry_version = 'run3/trunk'
options.conditions_version = 'master'
options.persistreco_version = 0.0
from RecoConf.muonid import make_muon_hits
make_muon_hits.global_bind(geometry_version=2)
options.output_file = 'hlt2_example.dst'
options.output_type = 'ROOT'
options.output_manifest_file = "hlt2_example.tck.json"
def make_lines():
return [
dzero2kpkm_line(),
dstarp2dzeropip_dzero2kpkm_line(),
b2d0mu_dzero2kpkm_line()
]
public_tools = [stateProvider_with_simplified_geom()]
config = run_moore(options, make_lines, public_tools)
The first key difference here is that the PFNs (this time for a signal simulation rather than minimum bias) are specified directly rather than using the TestFileDB:
input_files = [
# MinBias 30000000
# sim+std://MC/Upgrade/Beam7000GeV-Upgrade-MagDown-Nu7.6-25ns-Pythia8/Sim09c-Up02/Reco-Up01/30000000/LDST
# 'root://xrootd.echo.stfc.ac.uk/lhcb:prod/lhcb/MC/Upgrade/LDST/00069155/0000/00069155_00000878_2.ldst'
# D*-tagged D0 to KK, 27163002
# sim+std://MC/Upgrade/Beam7000GeV-Upgrade-MagDown-Nu7.6-25ns-Pythia8/Sim09c-Up02/Reco-Up01/27163002/LDST
'root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/MC/Upgrade/LDST/00070317/0000/00070317_00000033_2.ldst'
]
options.input_files = input_files
This requires that the user set also options related to the conditions used in
the simulation - in the previous example these were implicitly accessed from the
TestFileDB. Secondly, only a small selection of HLT2 lines are imported and
used. Finally, note that there is no reconstruction.bind call around the
run_moore call. At the time of writing, the default reconstruction option
was from_file=True, meaning here the reconstructed information stored in the
file is used to make candidates and selections. See also the
Enable real-time reconstruction page.
Tip
There are many tests of HLT2 running on simulation that run in the nightlies
every night, and each has a corresponding set of options files. Peruse them
under Moore/Hlt/Hlt2Conf/tests/ to see a wide range of different
configurations you can use.
Instructions for the stack with DD4HEP-compatible simulation
The above two options files ask to run over simulation that is “old”, and
therefore running them with a stack using the default platform (or any platform
that doesn’t have detdesc in the name) will fail with something like this:
ApplicationMgr INFO Application Manager Finalized successfully
ApplicationMgr ERROR Application Manager Terminated with error code 3
and if you scroll up through the log, you should see other ERROR or
FATAL messages that caused this error code to be given.
As stated above, more modern simulation should work fine, and the tests in Moore
are gradually migrating to use newer samples. An example of the usage of a
newer sample is the hlt2_pp_thor_without_UT.py test (code here),
which uses the options file mdf_input_and_conds_hlt2.py:
###############################################################################
# (c) Copyright 2019 CERN for the benefit of the LHCb Collaboration #
# #
# This software is distributed under the terms of the GNU General Public #
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". #
# #
# In applying this licence, CERN does not waive the privileges and immunities #
# granted to it by virtue of its status as an Intergovernmental Organization #
# or submit itself to any jurisdiction. #
###############################################################################
from Moore import options
from RecoConf.decoders import default_VeloCluster_source
# Use velo retina decoding:
default_VeloCluster_source.global_bind(bank_type="VPRetinaCluster")
# To be removed once data is updated to use a tag including
# https://gitlab.cern.ch/lhcb-conddb/DDDB/-/merge_requests/107
#from DDDB.CheckDD4Hep import UseDD4Hep
#if UseDD4Hep:
# from Configurables import DDDBConf
# DDDBConf().GeometryVersion = 'run3/before-rich1-geom-update-26052022'
options.set_input_and_conds_from_testfiledb(
'upgrade_Sept2022_minbias_0fb_md_mdf')
# Override settings used by TestFileDB to set geometry version correctly
# To be removed once
# https://gitlab.cern.ch/lhcb-datapkg/PRConfig/-/merge_requests/334
# is merged and in a released tag.
from DDDB.CheckDD4Hep import UseDD4Hep
if UseDD4Hep:
from Configurables import DDDBConf
DDDBConf().GeometryVersion = 'run3/before-rich1-geom-update-26052022'
options.input_files = list(set(options.input_files)) # remove dups
options.persistreco_version = 0.0
options.evt_max = 300
options.use_iosvc = True
options.event_store = 'EvtStoreSvc'
Running with this options file should work fine for DD4HEP and detdesc platforms.
If you are not using the TestFileDB, then to run with DD4HEP,
options.geometry_version and options.conditions_version must be
provided. The geometry version refers to a directory in the Detector project,
for example run3/trunk refers to the trunk (main) description of the Run
3 detector description, which lives at Detector/compact/run3/trunk. The
conditions_version corresponds to a Git branch in lhcb-conddb, and the
“conditions” more broadly tell us the state of the detector at a given time,
e.g. alignment of the VELO or some set of calibration parameters 1.
Extra configuration parameters you may need
Below are a few common extra bits of configuration you may need to get your Moore job to work. You will have seen some of them in use above. See also Input samples with different attributes.
input_raw_format
Calorimeter raw banks
MC samples produced with the CondDB tag sim-20220614-vc-mu100 or later (see
date in the tag) were produced with the new CALO raw banks. Moore expects these
raw banks by default, and so no configuration would be needed for them to be
used in the HLT2 calorimeter reconstruction. If your sample is older, then you
should request for the old calorimeter raw banks to be used. This is done by
adding the lines:
from RecoConf.calorimeter_reconstruction import make_digits
make_digits.global_bind(calo_raw_bank=False)
Note
If your file has old CALO raw banks and you do not provide the above lines, the calorimeter reconstruction will not find the banks - therefore do no CALO reco - but will not fail. It’s therefore important to get this right if you are reconstructing signals reliant on the calorimeters, for example electrons.
Muon geometry decoding version
Similarly, the muon_decoding_version has been updated over the years, and is
(as of January 2024) at version 3 by default. Most older samples require
version 2. If you provide the wrong version, the muon decoding will crash at
runtime, so you will know to add this option. To change the version, add the lines:
from RecoConf.muonid import make_muon_hits
make_muon_hits.global_bind(geometry_version=2)
persistreco_version
For technical reasons, in July 2023 the structure of Moore output files was
adjusted in this
merge request. This becomes relevant when we are running Moore using the
reconstructed information in the file, rather than running the reconstruction
on-the-fly. To perform selections using the reconstructed information stored
in the file, Moore will need to know how the file structure is organised.
The structure is encoded in the persistreco_version parameter. Altogether,
if you are running with reconstruction from file and the file was produced with
by Moore/Brunel job ran before approximately July 2023, you will need to add the
line
options.persistreco_version = 0.0
If the reconstructed information in the file is of the latest version, then no line is needed. Similarly, if you are running the reconstruction on-the-fly, no line is needed.
Modern simulation samples generated for the HLT1 bandwidth division
Information in this section is more likely to become outdated than those above.
A good source of modern simulation samples for both minimum bias and signal MC
are the samples produced to carry out the HLT1 bandwidth division. At the
time of writing (January 2024), the HLT1 bandwidth division for 2024 is underway
in earnest, and lots of simulated samples can be found under
/eos/lhcb/wg/rta/WP3/bandwidth_division/. To run over these files, the
following configuration will be needed:
options.dddb_tag = "dddb-20231017"
options.conddb_tag = "sim-20231017-vc-md100" # Or mu for "Mag Up"
options.simulation = True
options.input_type = 'ROOT'
options.data_type = 'Upgrade'
options.input_raw_format = 0.5
The availability and location of these files is not guaranteed to stay the same, but you may find them a useful resource. If the files have moved/disappeared, please edit this documentation to reflect this!
Footnotes