Week of 230703

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Panos (WLCG + Chair)
• remote: Christoph (CMS), Darren (RAL), David B. (IN2P3), David M. (FNAL), Douglas (BNL), Mark (LHCb), Onno (NL-T1), Peter (ATLAS), Xavier (KIT)

Experiments round table:

• ATLAS reports ( raw view) -
  • NTR

• CMS reports ( raw view) -
  • Problems when reading files from CMS EOS instance at CERN INC:3521021
    • Panos will check if the issue is handled by CERN IT.

• ALICE
  • KISTI was down from Monday evening to Wednesday morning because of a security incident
    • Services for ALICE were not compromised
  • Otherwise mostly business as usual

• LHCb reports ( raw view) -
  • Issues :
    • SARA:
      • Over the weekend, SARA-BUFFER went down and is currently not usable.
      • Ticket opened and converted to ALARM ticket this morning (GGUS:162617)
    • RAL:
      • Fix for jobs being killed due to memory limits is being tested on a section of the batch farm. If all OK, roll out to rest of the farm will happen tomorrow. (GGUS:162393)

• ASGC:
• BNL: NTR
• CNAF: NTR
• EGI:
• FNAL:
• IN2P3: Tape robot 2nd arm back in production after maintenance last week. Calibration will be checked by vendor on next Thursday, July 6th.
• JINR: NTR
• KISTI:
• KIT: NTR
• NDGF:
• NL-T1: The Surfsara dCache was in bad shape today. One of our two core domains, which are the communications hubs, was overloaded. Users got all kinds of errors. We tried to bring it down to continue running with the other core domain, but somehow we couldn’t get dCache stable. Even stopping all dCache components and starting them did not seem to help. In the end, we rebooted the node that had been running the overloaded core domain, and after that it seems better. dCache has been down from ~11:30 to ~14:00 CEST.
  • Mark reported that things look much better from the LHCb side, starting at ~14:00 CEST.
  • Peter remarked that ATLAS also has open tickets for this matter. Onno will look into them.
• NRC-KI:
• OSG:
• PIC:
• RAL: Antares will be down for ~20mins (4/7/23 11:00-11:20) for critical security patching (https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=34196 ).
• TRIUMF: NTR
• CERN computing services:
  • July 4th 09:00 ATLAS and CMS WLCG IAM database upgrades from mysql 5 to 8 OTG:0078123 (time verified with ATLAS and CMS)
  • July 10th 10h00 to 11h00: Grid CE downtime (ce503 ce516): OTG0078395 - GOCDB downtime
• CERN storage services:
• CERN databases:
• GGUS: NTR
• Monitoring:
• Middleware: BTR
• Networks: NTR
• Security:

AOB: