Checkreq.sh - checking package dependencies
Table of contents
1. How it works
The checkreq tool performs several consistency checks within each software package. It also does a few checks across the set of packages of a release in order to detect bad inter-project dependencies, and to detect use of bad package versions. The main purpose is detecting any dependencies between packages which are either unnecessary or have been overlooked in defining the CMT requirements file for the package. Checkreq is implemented as a shell script (checkreq.sh) and is maintained within the SIT. The tool is routinely run to check each referenced package during the nightly build procedures.
The following text and the figure further below sketch how checkreq works.
Checkreq does a limited analysis of the requirements file of the package to collect the names of other packages referenced. For this purpose, it parses the use statements, linkopts statements, some apply_pattern statements, and constituent (library/application) definitions.
It then does a limited analysis of the source files and header files (file extensions .cpp, .cxx, .c, .cc, .F, .h, .hh, .hpp, .age, .icc) contained in the package to assemble a list of packages referenced there via include statements. #include statements are analysed. Checkreq restricts the analysis to those files actually used in constituents definitions, i.e. to those which appear in the compilation. In addition, any header (.h, .icc) files visible to outside packages - i.e., header files in the subdirectory named like the package - are checked, unless option -x is chosen.
#include statements usually take the form #include [...]package/file, i.e. the name of the directory containing the included file is the name of the package containing that file, i.e. the directory can be used as the package reference. However there are exceptions to this simple structure. Checkreq deals with these exceptions by using a translation table and some additional case decisions.
A comparison is performed whether the included files, translated to package names, match the packages listed in the requirements file. Appropriate warnings are issued if there is no match.
Additional checks are performed on the full set of packages within a release, namely on the consistency of package versions and inter-project dependencies. This is done by means of a cache file which holds, for each package of a given release, project name and package name with version (see options -v, -c, -u below).
Checkreq does a limited analysis of the Python files (.py) contained in the package. The main purpose here is to detect wrong run-time dependencies between projects, such as from project AtlasReconstruction to AtlasSimulation. Python include and import statements are analysed. They take the form include ("package/xxx"...), from package.xxx ..., or import package.xxx ... .
For the inter-project dependencies, runtime dependencies are checked as just described (message "Project runtime dependency" in case of bad dependency), but also wrong dependencies coming from #includes ("Static project dependency") and wrong dependencies coming from use statements ("Project dependency") are flagged.
In addition to this, checkreq does some miscellaneous semanic checks in the requirements file.
Checkreq works with the directory structure used in project builds and with the one used in monolithic builds. For monolithic builds the option -m must be used (see below).
Depending on the outcome of the checks, the script exits with a code corresponding to the severity of the problem encountered. For exit code 3 (or above), the NICOS system generates emails to the person in charge of the package.
| exit 0 | no problems found |
| exit 1 | use statement for AtlasPolicy is missing, or
use statement for COOL instead AtlasCOOL is present, or likewise for POOL, SEAL package statement has wrong package name or is missing |
| exit 2 | one or more unnecessary use statements are present, or
bad inter-project dependency at runtime |
| exit 3 | use statements are missing for one or more required packages,
debugging options are present, bad inter-project dependency at build-time (via #includes or use statements) library as well as dual_use_library statements are present, or mismatch between a version stated in use statement and version present in release. |
Note that containers, external, run-time, and policy packages are always skipped by checkreq.
For the most recent source of checkreq, see http://atlas-sw.cern.ch/cgi-bin/viewcvs-atlas.cgi/offline/Tools/Scripts/share/checkreq.sh .
2. Usage
|
checkreq.sh [-h] [-d] [-e] [-l] [-w] [-v] [-p] [-m] [-y] [-x] [-i <package>] [-f <file>] [-c <file>] [-r <rel> <packages>] |
|
| -h(elp) | write this information. |
| -d(etail) | write detail information to stdout. |
| -e(xit) | do not exit with exitcode after each package. |
| -l(ocal) | use requirements file from working directory. needs specification of a release with -r. copies requirements over from there if a local requirements file does not exist. filename is requirements_package_release. |
| -w(riteuse) | write out use-statements generated from required packages. For option -local, merges extra use statements into requirements, and comments out unnecessary use statements. |
| -v(ersions) | suppress versions checking. |
| -p(rojects) | interprete directory structure according to project based builds (default) |
| -m(onolithic) | interprete directory structure according to monolithic builds. |
| (-y|-python) | (this option disabled at present - check include statements in the *.py jobOption against use statements, like #includes in the sources and headers.) |
| -x(tended) | do not extend analysis to all header files visible from other packages. |
| -i(gnore) package | ignore this package in the checks. |
| -f(ile) file | ignore the packages listed in file in the checks. |
| -c(ache) file | when generating the cache file, also copy it to <file>. |
| -u(secache) file | use <file> as filename for cache file, to allow use of a pre-existing file, or to avoid making a file for each project. If <file> does not exist, it is generated. |
| -r(elease) rel package,... | run on release rel with the given package-names, instead of on the current directory. |
2.1 Usage examples
cd /afs/cern.ch/atlas/software/builds/AtlasAnalysis/13.0.10/ or for a dev or bugfix nightly: checkreq.sh - check the package pointed to by the current directory. Do checkreq -v to omit version checking. |
| checkreq.sh -r AtlasCore/13.0.10 . -e > logfile (note the dot)
- check all leaf packages for release project AtlasCore, release 13.0.10, and save output. Containers, external, runtime, and policy packges are skipped. |
| checkreq.sh -e -r nightlies/dev/AtlasCore/rel_0 . (note the dot)
- check all leaf packages of development nightly rel_0 of project AtlasCore. | checkreq.sh -r nightlies/bugfix/AtlasProduction/rel_0 Generators/EvgenJobTransforms
- check package Generators/EvgenJobTransforms of bugfix nightly rel_0 of project AtlasProduction. |
| checkreq.sh -r AtlasSimulation/13.0.10 InnerDetector -e
- check all leaf packages including and below InnerDetector for project/release AtlasSimulation/13.0.10 and return highest exit code found in all packages. |
| checkreq.sh -r AtlasSimulation/13.0.10 InnerDetector -e -v
- similar but omit version checking. |
| checkreq.sh -r AtlasSimulation/13.0.10 InnerDetector -e -u /tmp/mycache
- similar but use /tmp/mycache as the version cache file. If the file does not exist, it is generated, otherwise this file is used which makes operation faster. |
| checkreq.sh -r AtlasEvent/13.0.10 InnerDetector/InDetRawUtils
- check this specific package for release AtlasEvent/13.0.10 and return exit code. |
| checkreq.sh -r AtlasEvent/13.0.10 InnerDetector/InDetRawUtils -d | more
- similar but provide detailed information on this package, including the requirements file, a map of which files of this package are #including which packages, and a list of #include directories which this package defines. - pipe to more since the output will not fit on one screen. |
| checkreq.sh -r AtlasEvent/13.0.10 InnerDetector/InDetRawUtils -l
- work on this package, but use a copy of the package's requirements in the current working dir, named requirements_InDetRawUtils_13.0.10. Create this file if it does not exist yet. This is meant for fast turnaround editing/testing of the requirements file. |
| checkreq.sh -r AtlasEvent/13.0.10 InnerDetector/InDetRawUtils -l -w
- similar, but automatically edit the copy of the requirements file if needed. That is, unnecessary use statements are commented out, and missing ones are appended. Thus another execution of the same command should not report any unnecessary or missing use statements. |
| checkreq.sh -m -r 11.0.2 InnerDetector/InDetRawUtils
- for a monolithic build, check this specific package for release 11.0.2 and return exit code. - ...and equivalently all of the examples above also hold for monolithic builds. |
2.2 Sample warning output
There are many examples of checkreq warnings in the NICOS results page of every nightly build. The following table gives an example of a checkreq warning output in which both unnecessary and missing use statements appear, together with some explanation.| message line: | explanation: |
| checkreq-v070719 warning for LArCondCnv (LArCondCnv-00-20-29): | checkreq version, package, and package version are given |
| Project runtime dependency: LArCondCnv(AtlasEvent) -> RecExTB(AtlasAnalysis) in files: | project Analysis is incorrectly referenced by this package which is in the "higher-up" project AtlasEvent |
| ../share/LArIOV_testFill_jobOptions.py ../share/LArIOV_testH6_jobOptions.py ../share/LArIOV_testH8_jobOptions.py |
reference to RecExTB(AtlasAnalysis) appears in these files |
| The following packages are apparently not needed. | |
| -- AtlasCOOL -- CaloIdentifier -- CondDBMySQL |
use AtlasCOOL etc. apparently are not needed in requirements file |
| The following packages (referenced as indicated) have no use statements. | |
| ++ LArCondData (from linkopts LArCondDataDict) | use LArCondData is apparently missing in requirements file.
In a linkopts statement, LArCondDataDict is referenced - this is translated to LArCondData
In case the reference happens in a #include, the files where the references appear are listed |
See also
- checkreq results in NICOS nightly build. Select a release, then click on an entry in the column Checkreq Test to see the checkreq result for a given package.