Running LHCb Software

Introduction

This page is a description of the tools used to prepare the local development environment and to run LHCb applications.

Note: Because of the migration of the LHCb build system from CMT to CMake, we had to update the tools (so far represented by SetupProject) and we decided to try to improve the developer/user experience.

Disclaimer: all the versions of the projects presented in this page are those available at the moment of writing and there is no implicit or explicit suggestion about which version to use or not to use. They are just examples.

Runtime

The way we build and distribute our software require that we set several environment variables to run an application. This is taken care of by the script lb-run.

lb-run behavior is modeled after the standard Unix command env (see man 1 env) with the addition of concepts like prepend and append, and the knowledge of LHCb software projects.

The simplest way to use it is

lb-run <Project>/<version> <command> <command args...>

for example:

lb-run Brunel/prod gaudirun.py MyBrunelOptions.py

It possible to tune in several ways the behaviour of lb-run, through the various command line options (see lb-run --help), for example to use a project from the nightly builds:

lb-run --nightly lhcb-head DaVinci/HEAD gaudirun.py MyOptions.py

or to fine tune the environment used to run the command:

lb-run -p PYTHONPATH=$HOME/python Bender python

(in this case we prepend $HOME/python to the variable PYTHONPATH defined by the Bender runtime environment and then call the command python).

More exotic project versions can be accessed with the Project/version syntax, e.g.

lb-run --nightly-cvmfs --nightly lhcb-2016-patches Brunel/2016-patches gaudirun.py MyBrunelOptions.py

It’s also possible to change the platform to use on the fly without messing with the current environment:

lb-run -c x86_65-slc5-gcc49-dbg Gaudi/latest python

As it happens with the Unix command env, if no command is passed to lb-run, it prints the modified environment. It’s useful, though, to print only one variable, for example with something like

lb-run Hlt/prod printenv LD_LIBRARY_PATH

For shell skripting, you may want to use an environment variable. For this it is important that the variable gets resolved in the lb-run environment and not in your user shell before lb-run gets called.

# do this:
lb-run Brunel/v54r0 echo '$BRUNELROOT' >> ${HOME}/foo.txt    # works with single quotes
lb-run Brunel/v54r0 echo \$BRUNELROOT >> ${HOME}/foo.txt    # works with escaped dollar
# don't do that
lb-run Brunel/v54r0 echo $BRUNELROOT >> ${HOME}/foo.txt    # doesn't work w/o quotes
lb-run Brunel/v54r0 echo "$BRUNELROOT" >> ${HOME}/foo.txt    # doesn't work w/ double quotes

Note that to distinguish from the options for lb-run and the options to the command that lb-run will execute, it is mandatory to specify lb-run options before the name of the project. So in

lb-run Gaudi/prod myScript --nightly

the option --nightly will be passed to the command myScript and not to lb-run.

Sometimes (e.g. for debugging purposes) it is useful to execute several commands in the modified environment, and wrapping all the calls with lb-run can be tedious and error prone. In that case it’s easy to start a subshell with the modified environment:

lb-run LHCb/prod bash --norc

or, for the tcsh lovers,

lb-run LHCb/prod tcsh -f

(The -f option avoids the invocation of /etc/bashrc or other /etc files, which would invoke the LbLogin scripts and manipulate the environment variables on top of lb-run) Once done, you can exit from the subshell and get back to the initial environment. This is very handy to have nested environments that do not conflict with each other, for example to run in sequence Gauss, Bool and Brunel, each of them in the same LHCbDirac environment.

#SetupProjectFallback lb-run leverages on the metadata produced when building the projects with CMake, but it falls back on SetupProject if it cannot find them. In this case the command line options are passed t SetupProject, but some options are only available in lb-run (for example the prepend option -p), so the command line might need some tuning before it succeeds.

Comparison with SetupProject

Development

The main differences between SetupProject --build-env and lb-dev are:

  • lb-dev does not modify the runtime environment and does not call cd to switch to the created directory

  • the default name of the local project created by lb-dev is different from that used by SetupProject; this is because the names used with SetupProject will confuse the CMake build system, and with the new names it will be more clear which is a released project and which a local one

  • the default version of a project used by lb-dev is prod, instead of picking up the latest one

  • by default SetupProject creates the local project in $User_release_area, while lb-dev creates it in the current (or specified) directory

  • local projects created with lb-dev contain details about the environment they require, so it is enough to enter the directory and build, while with SetupProject --build-env one has to call every time

  • lb-dev supports CMake-based LHCb software projects while SetupProject does not.

  • if the destination directory already exist SetupProject will silently do nothing (or prepare a possibly wrong build time environment), while lb-dev will print an error message (once created the directory, there is no need to call lb-dev again)

Runtime

The biggest difference between SetupProject and lb-run is that the first modifies the environment of the shell, while the second prepare the environment to run a command. The sequence of commands

SetupProject LHCb
CondDBBrowser

is equivalent to

lb-run LHCb/latest CondDBBrowser

but in the first case you have to exit from the shell to get back to a clean environment.

If you prefer the SetupProject behaviour, use:

lb-run LHCb/latest tcsh -f

(or bash --norc or zsh -f depending on your favourite shell)

Other important differences are

  • SetupProject does not understand the metadata produced with CMake builds and lb-run does not understand project built with CMT, but it falls back on SetupProject if needed

  • lb-run does not support (yet) some of the less used features of SetupProject, mainly adding external libraries

Development

Software development in LHCb is based on the development of subsets of the software on top of an existing release of a software project (see LHCbSoftwareTutorials).

The script lb-dev helps to prepare the local working directories (“local projects” or “satellite projects”) for the developers.

The simplest way to call lb-dev is

lb-dev <Project>/<version>

This will create a new subdirectory called <Project>Dev_<version> in the local directory and will create all the support files to be able to develop against <Project>.

Another very common use case is to build against the version of a project build in the LHCb Nightly Build System, for example:

lb-dev --nightly lhcb-gaudi-head Gaudi/HEAD

or

lb-dev --nightly lhcb-gaudi-head Thu Gaudi/HEAD

They will both create GaudiDev_HEAD bound to the nightly slot lhcb-gaudi-head, but the first one to the Today build and the second on the build of Thursday (this connection can be changes editing the file build.conf in GaudiDev_HEAD).

Very useful options of lb-dev are

  • --name to specify the name of the local project, like in lb-dev --name MyAnalysis DaVinci/prod

  • --dest-dir to specify the parent directory of the created local project (instead of ‘.’)

  • -c or --platform PLATFORM used to select a specific platform (a.k.a CMTCONFIG or BINARY_TAG). By default lb-dev tries to find the most appropriate PLATFORM for the current environment, but currently it fails when used in coordination with the --nightly option. In this case, the workaround is to specify the platform with the -c option.

More options can be explored with lb-dev --help.

In the local project directory you will also find a script called run. It is a shortcut to run a command in the environment of your local project, for example you can do something like:

cd MyAnalysis
./run gaudirun.py MyOptions.py

Similarly, if you want to do more than just run gaudirun.py once, you can start a shell with the runtime environment:

cd MyAnalysis
./run tcsh
echo $TRACKKERNELROOT

If you want to use lb-run to run your modified projects created with lb-dev you need:

  1. Run make install in the project folder. 2. Make sure the project folder is in the search path. If not, you can use –user-area <parent_folder_of_the_project_folder> 3. Make sure you specify the version number. 4. The name of the project will be different. For example Moore -> MooreDev

Example:

lb-run --user-area $HOME MooreDev/v23r5p2 gaudirun.py

You can use the –runtime option to add projects, for example:

lb-run --runtime DaVinci v36r5 Moore/v23r5p2 gaudirun.py

Tunings

Using ccache

If you’re going to compile identical files over and over again (when building full projects, but only manipulating a few files; whilst calling make clean in between, or changing git branches), it may pay off to enable ccache. As also described on the git4lhcb twiki page (https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb) this can be done with

# Use ccache
export CCACHE_DIR=/<somewhere>/.ccache   # local directory where the cache will be kept
ccache --max-size=10G                    # choose wisely based on the available space and the size of the projects

# tell CMake to use ccache
export CMAKEFLAGS="-DCMAKE_USE_CCACHE=ON"  # option 1: enable as environment variable
# or
cd Rec # project you want to build
echo 'set(CMAKE_USE_CCACHE ON CACHE BOOL "")' >> cache_preload.cmake   # option 2: use a cache_preload file
# cache_preload.cmake can hold some predefined variables, so you can skip 'CMAKEFLAGS'

colorized compiler warnings/errors

Modern compilers colorize their printouts (errors red, notes blue) to make them easier to read, especially when it’s several pages of them. With ninja however compilers cannot determine if they print to a log file or to a terminal. Modern ninja versions take over that task, but require that compilers will print everything in color w/o auto detection on the compiler side. The setting to enable colors with ninja is (depending on the Gaudi version)

cd Rec # project to build
echo 'set(GAUDI_DIAGNOSTICS_COLOR ON CACHE BOOL "")' >> cache_preload.cmake       # the latest Gaudi versions, as cache_preload file
# or
echo 'set(GAUDI_DIAGNOTICS_COLOR ON CACHE BOOL "")' >> cache_preload.cmake        # some intermediate Gaudi versions, as cache_preload file
# or
export CMAKEFLAGS="-DGAUDI_DIAGNOSTICS_COLOR=ON" #  latest Gaudi, through environment variables
# or
export CMAKEFLAGS="-DGAUDI_DIAGNOTICS_COLOR=ON"  #  some intermediate Gaudi versions, through environment variables

(There are a few things to iron out at the ninja side, but it should work okay’ish in many cases https://github.com/ninja-build/ninja/issues/1214 https://github.com/ninja-build/ninja/pull/1382 https://github.com/ninja-build/ninja/pull/1268)