Monitoring of large-scale federated data storage: XRootD and beyond.
The computing models of the LHC experiments are gradually moving from hierarchical data models with centrally managed data pre-placement
towards federated storage which provides seamless access to data files independently of their location and dramatically improved recovery due
to fail-over mechanisms. Enabling loosely coupled data clusters to act as a single storage resource should increase opportunities for data
analysis and should enable more effective use of computational resources at sites with limited storage capacities. Construction of the data
federations and understanding the impact of the new approach to data management on user analysis requires complete and detailed monitoring.
Monitoring functionality should cover the status of all components of the federated storage, measuring data traffic and data access
performance, as well as being able to detect any kind of inefficiencies and to provide hints for resource optimization and effective data
distribution policy. Data mining of the collected monitoring data provides a deep insight into new patterns of usage of the storage resources,
beyond that provided by other monitoring strategies.
In the WLCG context, there are several federations currently based on the
XRootD technology. The talk will focus on monitoring for the ATLAS
and CMS
XRootD federations (Federated Atlas
XRootD (FAX) and Any Data, Any Time, Anywhere (AAA)) implemented in the Experiment Dashboard
monitoring framework. Both federations consist of many dozens of sites accessed by many hundreds of clients and they continue to grow in
size. Handling of the monitoring flow generated by these systems has to be well optimized in order to achieve the required performance.
The talk will demonstrate that though FAX and AAA Dashboards are being developed for
XRootD federations, the implementation is generic and can
be easily adapted for other technologies, such as HTTP/WebDAV federations.
- Track: Distributed Processing and Data Handling
- Co-Authors from IT-ES: A. Beche, J. Andreeva, D. Giordano, P. Saiz, D. Tuckett
- Presentation Type: Parallel
- Author list:: A. Beche, J. Andreeva, D. Giordano, P. Saiz, D. Tuckett , D. Diéguez Arias, S. Belov, M. Tadel, I. Vukotic, D. Oleynik, A. Petrosyan