Week of 210531

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
  • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Kate (chair, DB), Julia (WLCG), Maarten (ALICE, WLCG), Borja (monitoring), Michal (ATLAS), Xavier (KIT), Darren (NDGF), Christoph (CMS), Andrew (TRIUMF), Vincenzo (CNAF), Nils (computing), David B (IN2P3), Dave M (FNAL), Pepe (PIC), Pinja (security)

Experiments round table:

• ATLAS reports (raw view) -
  • Issues:
    • Stage-in failures of reprocessing jobs at NDGF-T1 (GGUS:152126)
    • lost files at pic (1646 lost files that were not saved to tape). Declared lost by the DDM ops.
    • "The available CRL has expired" transfer errors to IN2P3-CC (GGUS:152231)

• CMS reports (raw view) -
  • Authentication issues at T1_FR_CCIN2P3 GGUS:152234
    • Possibly affects also other French sites
  • Migration of production transfers to WebDAV is continuing
    • Usually involves a hand shake with sites via GGUS ticket
  • Overall good CPU resource usage and no major issues

• ALICE
  • French sites were affected by the expiration of the AC-GRID-FR CRL on Sunday
    • Looks fixed now

• LHCb reports (raw view) -

Sites / Services round table:

• ASGC: nc
• BNL: US National Holiday Today - NTR
• CNAF: NTR
• EGI: nc
• FNAL: NTR
• IN2P3: AC GRID-FR CRL expired on Sunday morning (10:00 AM CEST) All French sites are affected. CRL has been updated at 12:00 CEST, but the updated CRL file has to be fetched by all WLCG services and sites. About GGUS tickets, French site admins wait GGUS to get the CRL to acces and answer the tickets. I have sent a mail to support@ggusNOSPAMPLEASE.org to ask the CRL to be fetched.
• JINR: nc
• KISTI: nc
• KIT: Storage issues on Monday, 24th of May, caused access timeouts for all VOs but was quickly fixed.
• NDGF: NTR
• NL-T1:
  • SARA-MATRIX: LHCb reported failures because some internal IPv6 addresses were handed out in TURLs. These addresses were Dell iDrac9 addresses. This morning we changed dCache settings and restarted our GridFTP doors to use only external addresses but the problem is not over yet. GGUS:152211
  • Unable to dial in, apologies
• NRC-KI: nc
• OSG: nc
• PIC: NTR
• RAL: NTR
• TRIUMF: Updating to frontier-squid 4.15 on squid2.lcg.triumf.ca, scheduled outage for Tuesday @ 17:00hrs UTC. GOCDB:https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=30707

• CERN computing services: Update of local Condor version tomorrow 1'st of June should be fully transparent to Grid users. ( OTG:0064093 ) CE511 was degraded Fri/Sat ( OTG:0064194 )
• CERN storage services:
• CERN databases: NTR
• GGUS: NTR
• Monitoring: NTR
• Middleware: NTR
• Networks: NTR
• Security: NTR

AOB: