3.13 Testing and Validation

3.13.2 Types of Testing

ATLAS offline testing activities to date include unit testing, small-scale integration testing, large-scale integration testing, and physics validation.

Both the level and type of testing has been left up to the developers. The testing group has concentrated on building test frameworks to ease the burden of testing, thereby encouraging developers to provide tests, and allowing management to monitor the level of testing and the test results themselves. The various types of testing are:

Unit tests, or simple and fine-grained tests that typically deal with individual classes or functions. The CppUnit tool is provided for automation of C++ code unit tests. Providing complete unit tests for a package involves a large amount of work, but provides high confidence in code quality.
Small-scale integration tests do not strive for complete code coverage. Rather, they test some small amount of functionality. The implicit assumption of such tests is that success infers that an unspecified amount of code needed to generate the result also ran successfully.
Large-scale integration test jobs, which run on computing farms, usually perform regression tests on the output of a physics job such as text files, histograms and n-tuples.

The test frameworks have also been a useful way of regularly running standard jobs where the testing step is replaced by a human examination of job output, usually log files. These frameworks are also used to run quality-assurance jobs which run source-code metrics and check adherence to coding standards.

3.13.3 Testing Frameworks

As a result of the need for offline testing, work started independently on three separate frameworks. These are AtNight (ATN), Kit Validation (KV) and the Run Time Tester (RTT).

While each framework runs tests and displays their results, there are differences in the intention as to how these frameworks are to be used. Often the tests use the results of a job - such as an Athena job - as input. This requires the job to be run before the tests can be carried out.

All frameworks can be run locally, where the developer has complete control of the framework. Local running allows the developer to run tests on code that has been checked out of the repository and modified.

ATN and the RTT are also run regularly on the nightly and numbered releases. Both run KV as part of this procedure.

In practice, the test frameworks are run in different environments. This is not a fundamental necessity, but is a result of the differing emphases of the framework developers. ATN is run on the machines used to run the nightly builds. This limits the resources available for testing, and tests run under ATN are usually short. KV testing is usually run "by hand" as part of kit building, and once again the tests are usually short. The RTT is currently run daily on a farm at UCL. With these resources, more extensive testing is possible. The full set of RTT results currently take a number of hours to generate.

3.13.3.1 ATN

The ATN framework contains little that is specific to ATLAS. Knowledge of such details resides in user-provided scripts which are run by the ATN framework. Further, ATN was designed to be able to run other test frameworks such as QMtest, thereby leveraging the work invested in such products.

ATN is closely coupled to the nightly build system (NICOS). The results of the ATN framework are integrated into the NICOS status pages.

Domains currently being tested using ATN include Athena, database, detector systems (muon spectrometer, liquid argon calorimeter), simulation, reconstruction, trigger and kit validation.

3.13.3.2 KV

KV is a set of scripts used to test the ATLAS offline distribution kits and releases on AFS. It is easy to run, with quick turn-round which makes it a convenient tool for kit builders.

KV is currently used for kit validation and releases on AFS, and runs nightly under ATN and RTT.

3.13.3.3 RTT

The RTT adopts the point of view that ATLAS-specific information is to be understood by the framework, with the intention that the RTT should reduce the load on the ATLAS user to the minimum. To this end, the knowledge on how to run Athena, and other jobs, which provide the material which serves as input to the tests, is incorporated in the RTT. Further, common testing tasks are identified and placed in RTT libraries. As a result, the user does not need to write scripts to carry out the tasks of running standard jobs and standard tests, but interacts with the framework through configuration files. For those cases where the RTT-provided machinery is inadequate to fulfil users' needs, the user can provide a specialized test presented as a Python class. Under the RTT, information flows both from the user to the framework - the user specifies the test class and the arguments via the configuration file - and from the RTT towards the test instance: the RTT passes an information object when invoking the run method of the test instance.

Domains currently being tested using RTT include fast simulation, reconstruction, e-gamma, trigger, graphics, muon tracking, and QA.

3.13.4 Testing Future

The testing group is moving towards providing users with a coherent testing environment. The three independently developed test frameworks are run in different manners. Work is currently under way to harmonize this situation. Care will be taken to ensure that the frameworks continue to function during this process.

Testing will be organized by package. Each package will provide a single configuration file containing a section for each framework. Each framework will specify the possible content of the the corresponding configuration file section.

Tests which are common to the frameworks will be moved into libraries. To this end, it has been agreed to use the same scripting language. A common interface to the tests will make it possible to carry them out from any of the frameworks.

More work is needed to define the following:

Description and storage of results;
User interface to mine results;
Grid integration.

3.13.5 Physics Validation

Physics validation is an investigation of the results produced by the offline software. The closest testing activity to physics validation is that of the large-scale integration tests. The activities differ in that validation is focused on what has gone wrong, whereas testing monitors deviations of software that has been established to be working correctly.

The final users of the Software and Computing infrastructure are ATLAS physicists who interact with this infrastructure every day to achieve their scientific goals. The quality of the software, both in terms of functionality and performance, needs to be monitored and feedback from the users brought to the attention of the computing management.

To facilitate the interaction between the software and physics communities, a Physics Validation forum was set up. The two main roles of this forum are:

Inform the physics community of the latest news, e.g. software releases to use, problems that they may find and possible solutions. In this respect the Physics Validation forum provides a service to the physics working groups so that they can collect this information from a single source;
Collect feedback from the users, identify possible problems and missing functionality. This aspect is complementary to the activity of the testing group, as users identify possible new problems that appear while trying to pursue a particular task. The participation of physicists from different working groups ensures that all areas of the ATLAS computing and software are stressed and used.

Physics validation meetings are held every two weeks and last for 1 to 2 hours. The meetings are informal and include presentations from people from different physics groups and provide a frequent information exchange to find common problems and solutions. A Physics validation coordinator organizes these meetings, and reports directly to the Physics coordinator. The same person also acts as a liaison for the physics requirement and feedback within the software management board.