WLCG Containers Working Group

Mandate

The mandate discussed at the first meeting and agreed at the 11 July 2017 GDB is:

To understand and track the use of containers within WLCG for compute deployment, and to make recommendations to sites on potential deployment patterns that they may wish to adopt. To ensure that Singularity and its support services are deployable for the LHC experiments.

We agreed two main goals:

1. Singularity is sensibly deployable in WLCG and used by all experiments, and we’ve documented and understand the operational aspects.
2. We should have surveyed and made recommendations on potential deployment patterns (and technologies) for container-based compute deployment and provisioning, with an aim to reduce operational effort on the sites, and to meet the needs of the ongoing analysis reproducibility work going on in the experiments. At the end, we should have clearly documented a few reasonable paths for a site (should it wish to) to migrate their compute to these technologies.

Work areas

Singularity deployment

The group's current focus is coming up with a baseline recommendation for the deployment of Singularity for sites, while trying to ensure as much commonality as possible between the experiments in the way they use it.

Container-based compute deployment recommendations

The final baseline document for Singularity is attached below.

Actions

DONE	Who	What	Status
WC1	Maarten, Alessandra	Contact David Love in Manchester (the current Singularity EPEL maintainer) to discuss our community's needs	Done. Brian now maintains EPEL.
WC2	Andrew	Describe more fully the potential workaround for ATLAS' PWD bind-mounting	Done / Superseded with WC6.
WC3	Andrej	Understand whether WC6 solution would work better for ATLAS	Yes, thought that underlay should help a lot.
WC4	Dave, Jakob	Understand the issue with bind-mount overlay on CVMFS unpacked images, and whether a CVMFS workaround is appropriate	Understood: CVM-1434. Workaround is to use underlay.
WC5	Gavin	Start google doc to track emerging baseline recommendation and issues	Done, see http://cern.ch/go/Q6rP
WC6	Vincent, Dave	Track feature request with upstream Singularity for unprivileged overlay trick	Now supported by EPEL 2.6 & upstream 3.0,
WC7	Alessandra / Andrej	Test the new underlay feature for ATLAS and comment on pull-request	Reported https://docs.google.com/spreadsheets/d/1SGKyja47Veu_8IUXlXWOOEferuFoD62O4m64pgTNgSk/edit
WC8	Olga / Gavin	Test the new underlay feature for CERN and comment on pull-request	Running in production now
WC9	Olga / Gavin	CERN to setup HammerCloud based CI test for epel-testing including Singularity based test	Not HC, but our batchtest resources run this in production now
WC11	Gavin, Maarten	Write down initial baseline recommendation based on 3 July meeting and iterate on list	FINAL Done

Meetings

The Indico event area is https://indico.cern.ch/category/9769/.

• 3 July 2019
• 30 January 2019
• 17 October 2018
• 2nd July 2018
• 4th April 2018
• 28th February 2018
• 17th January 2018
• 6th December 2017
• 22nd November 2017
• 11 July 2017 (pre-GDB)

Reports

• WLCG Operations Coordination meeting, 7 December 2018, Gavin
• GDB, 10 January 2018, Gavin

Mailing lists

The mailing list is wlcg-containers@cernNOSPAMPLEASE.ch - you can subscribe from egroups.cern.ch.