Week of 240429

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Maarten (ALICE), Panos (WLCG)
• remote: Andrew (TRIUMF), Borja (Chair, Monitoring), Brian (RAL), Carmen (CNAF), Christoph (CMS), Farrukh (FNAL), Giacomo (Computing), Ivan (BNL), Julia (WLCG), Peter L (ATLAS)

Experiments round table:

• ATLAS reports (raw view) -
  • NTR

• CMS reports (raw view) -
  • Decommissioning campaign for deprecated protocols is ongoing via GGUS tickets
  • Otherwise nothing major to highlight

• ALICE
  • NTR

• LHCb reports (raw view) -
  • NTR

Sites / Services round table:

• BNL: NTR
• CNAF: Network intervention on some worker nodes is planned on 2nd of May, they're currently in drain state. Jobs still running on those worker nodes on that date will be killed.
• EGI: NC
• FNAL:
  • Nothing major to report for operations. Our Alma9 migration is ongoing with an internal deadline of June 30.
  • GGUS ALARM ticket paging broken at Fermilab, the new release last week did not page the on-call rotation primary.
    • Maarten: Are you looking into the issue from your side?
    • Farrukh: Apparently the way GGUS communicates with SNOW is deprecated, it should be updated to use the SOAP API, trying to get people to look into it but not successfully yet.
    • Maarten: The best will be to follow it up with GGUS to get it sorted, I'll ping them now that I'm also aware.
• IN2P3: NTR
• JINR: NC
• KISTI: NC
• KIT:
  • Alice was suffering last week from few badly performing data servers in their xrootd cluster. Issue solved by stopping the execution a certain system call (sendfile) for xrootd. According to IBM, it shouldn't be a problem with our GPFS and kernel versions, yet our observations differ from that and a ticket to IBM has been issued.
  • After streamlining service endpoint names for all dCache SEs: Belle (GGUS:166269) observes problems with some transfers.
• NCBJ: NC
• NDGF: NC
• NL-T1: NTR
• NRC-KI: NC
• OSG: NC
• PIC: NC
• RAL:
  • gsiftp support at RAL will cease 03-06-2024 . May retire sooner if no users are still using them. ( this mainly applies to disk storage ECHO)
• TRIUMF: NTR

• CERN computing services:
  • schedd upgrades Thursday and Friday: OTG0149725 OTG0149727. Running jobs will not be affected, though users will not be able to query them during the upgrade.
• CERN storage services: NC
• CERN databases: NC
• GGUS: NTR
• Monitoring:
  • Distributed final SiteMon availability/reliability reports for March 2024.
• Middleware: NTR
• Networks: NC
• Security: NC

AOB: