Data storage accounting and verification in LHC experiments

All major experiments at Large Hadron Collider (LHC) need to measure real storage usage at the Grid sites. This information is equally important for the resource management, planning, and operations.

To verify consistency of the central catalogs, experiments are asking sites to provide full list of files they have on storage, including size, checksum, and other file attributes. Such storage dumps provided at regular intervals give a realistic view of the storage resource usage by the experiments. Regular monitoring of the space usage and data verification serve as additional internal checks of the system integrity and performance. Both the importance and the complexity of these tasks increase with the constant growth of the total data volumes during the active data taking period at the LHC.

Developed common solutions help to reduce the maintenance costs both at the large Tier-1 facilities supporting multiple virtual organizations, and at the small sites that often lack manpower.

We discuss requirements and solutions to the common tasks of data storage accounting and verification, and present experiment-specific strategies and implementations used within the LHC experiments according to their computing models.

• Summary: Comparative analysis of the CMS, ATLAS, and LHCb solutions to the storage accounting and verification

• Primary authors : RATNIKOVA, Natalia, SERFON, Cedric, LANCIOTTI, Elisa (IT-ES)
• Co-Authors : MAGINI, Nicolo (IT-ES), WILDISH, Tony, SANCHEZ HERNANDEZ, Alberto, ZHANG, xiaomei
• Track classification : Distributed Processing and Analysis on Grids and Clouds
• https://indico.cern.ch/abstractDisplay.py?abstractId=224&confId=149557

-- NicoloMagini - 27-Oct-2011