Week of 240422

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Julia (WLCG), Maarten (WLCG, ALICE, GGUS)
• remote: Darren (RAL), Christian (NDGF), Daniele (CNAF), Dave (FNAL), Farrukh (FNAL), Ivan (BNL), Nils (Computing), Panos (Chair, CMS), Peter (ATLAS), Stephan (CMS), Xavier (KIT)

Experiments round table:

• ATLAS reports (raw view) -
  • NTR

• CMS reports (raw view) -
  • CMS encountered a problem with an ORACLE database (CMS_TRG_R) Saturday morning and needed on-call support from IT, which was not available. We were lucky that Kate was able to fix the problem in “hero mode” during the LHC interfill period, so CMS did not lose data, but we easily could have if the incident had lasted much longer.
    • Maarten: Online DBs are actually the only IT services that should offer an on-call piquet.
    • Stephan/Panos: CMS coordinators are aware of this and will follow up on the necessary communication channels.
  • significant drop in core usage early last week that came from opportunistic resources
  • several monitoring outages/interrupts the last week
  • we started decommissioning of GSIftp/gridFTP protocols with sites

• ALICE
  • after a Grafana upgrade last week, the SiteMon (SAM) GUI suffered usability regressions
    • both ALICE and CMS opened tickets about that
    • in particular, the logs now show a \r sequence at the end of each line
    • the MONIT team are looking into how that can be fixed

• LHCb reports (raw view) -

Sites / Services round table:

• BNL: NTR
• CNAF: Upgrading HTCondor cluster to HTCondor 23, ~1/2 of cluster is already upgraded with 4 CEs as well
• EGI:
• FNAL:
• IN2P3: NTR
• JINR:
• KISTI:
• KIT:
  • HPSS tape backend is in downtime since today 9 a.m. CEST (GOCDB:35274).
  • Alice was suffering last week from few badly performing data servers in their xrootd cluster. Those were removed from service for further investigations.
  • Streamlining service endpoint names for all dCache SEs: ATLAS (GGUS:166267), LHCb (GGUS:166268), Belle (GGUS:166269). The changes for CMS were applied by our in-house CMS contacts to the GridKa specific storage.xml already.
• NCBJ:
• NDGF: NTR
• NL-T1:
• NRC-KI:
• OSG:
• PIC:
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: NTR
• CERN storage services:
• CERN databases:
• GGUS:
  • A new release is planned for Wed this week
    • Release notes
    • A downtime has been scheduled from 06:00 to 08:00 UTC
    • Test alarms will be submitted as needed
• Monitoring:
• Middleware: NTR
• Networks:
• Security:

AOB: