Week of 200914

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: Borja (Monitoring, Chair), Miro (DB), Concezio (LHCb), Dave (FNAL), Darren (RAL), Luis (Compute), Andrew (TRIUMF), Onno (NL-T1), Maarten (ALICE), Xin (BNL), Xavier (KIT), Jens (NDGF), Eva (DB), David (IN2P3)

Experiments round table:

• ATLAS reports ( raw view) -
  • storage issues at UKI-SCOTGRID-GLASGOW

• CMS reports ( raw view) -
  • (Virtual) CMS Week is running in parallel - limited attendance from CMS
  • Business as usual

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • generally smooth operations
  • some issues with jobs at KIT when using ARC-CEs (see below). Under investigation.

Sites / Services round table:

• ASGC: NC
• BNL: NTR
• CNAF: NC
• EGI: NC
• FNAL: NTR
• IN2P3: Site will be in scheduled maintenance on September, Tuesday 22th. CEs and SEs will be off on morning.
• JINR: NC
• KISTI: NTR
• KIT: Several ARC-CEs currently have issues communicating with computing nodes. As reported by LHCb (no GGUS ticket so far), this may cause jobs to start multiple times. In general, we urge VOs to switch over to our new HTCondor-CEs (CMS completed this shift already, ATLAS may follow very soon after today's updates to AGIS), while we still will try and fix ARC-CEs asap.
• NDGF: Downtime for dCache on Tuesday for Zookeeper upgrade
• NL-T1: Sara-matrix has a broken dCache pool node. Some files of LHCb are currently not available.
  • Question on ETA from LHCb (Concezio) answered by Onno: should be available by the end of the week.
• NRC-KI: NC
• OSG: NC
• PIC: NTR. (Sorry, Vidyo does not allow me to login! - J. Flix)
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
  • OTG0058893 - Batch service affected by network intervention
• CERN storage services: NC
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
  • Final availability reports for August 2020 sent around
  • CMS SAM3 is stuck since Thursday, we are still working on it
• MW Officer: NC
• Networks: NTR
• Security: NTR

AOB: