Week of 240108

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Maarten (ALICE + WLCG), Steve (FTS)
• remote: Andrew (TRIUMF), Christoph (CMS), Concezio (LHCb), Darren (RAL), David B (IN2P3-CC), David M (FNAL), Ivan (BNL), Jens (NDGF), Julia (WLCG), Nils (computing), Peter (ATLAS + Lancaster), Xavier (KIT)

Experiments round table:

• ATLAS reports ( raw view) -
  • No significant issue over the Christmas/New Year break
  • Reminder to all sites about GGUS tickets about token support for storage submitted

• CMS reports ( raw view) -
  • Best wishes for 2024 to everyone!
  • Rather quiet operations during Xmas vacation time
    • Some site issues got addressed also during the holidays
    • No major issues

• ALICE
  • Best wishes for 2024 !
  • Normal activity levels on average in the last 3 weeks.
  • No major issues.

• LHCb reports ( raw view) -
  • Best wishes for 2024
  • smooth operations during the holidays, no major issues

Sites / Services round table:

• BNL:
  • Tokens enabled for BNL storage since this morning (GGUS:164654)
  • Downtime on 22.01.2024 for dCache upgrade (15:00 to 19:00 CEST)
• CNAF: GOCDB downtime (WARNING) 34757 & 34760 from 09/01 to 11/01 due to changes on the LHCOPN routing between CERN and CNAF.
• EGI:
• FNAL:
  • David M:
    • happy new year!
    • no incidents over the holidays
    • tomorrow a dCache upgrade is planned for the tape SE,
      while the disk SE should get done next week
  • Julia:
    • the new version will allow us to test the new monitoring workflow
      that we plan to use for DC24 in Feb
    • we still need the FNAL numbers for the Nov accounting validation
• IN2P3: NTR and best wishes for 2024.
• JINR:
• KISTI:
• KIT: No major issues since three weeks ago. One minor incident (1 ATLAS dCache pool offline) was handled by the on-call engineer on last Saturday.
• NCBJ:
• NDGF:
  • Jens:
    • happy new year!
    • no issues over the holidays
• NL-T1: Best wishes! Surf dCache is currently in scheduled maintenance for upgrade.
• NRC-KI:
• OSG:
• PIC:
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: NTR
• CERN storage services:
  • The CERN FTS team found that Alma 9 FTS test servers are failing to execute GridFTP transfers.
    The team will investigate whether or not re-enabling support for SHA1 encryption will solve this problem.
• CERN databases:
• GGUS: NTR
• Monitoring:
• Middleware: NTR
• Networks:
• Security:

AOB: