Week of 240115

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Maarten (ALICE + WLCG), Panos (CMS + WLCG + Chair), Julia (WLCG)
• remote: Andrew (TRIUMF), Concezio (LHCb), Daniele (CNAF), David M (FNAL), Henryk (NCBJ), Onno (NL-T1), Peter (ATLAS + Lancaster), Xavier (KIT)

Experiments round table:

• ATLAS reports ( raw view) -
  • NTR

• CMS reports ( raw view) -
  • A DC24 pretest for Italian sites was requested and completed last Friday.
    • From the CMS side everything was smooth but since CNAF realized that there was a problem with routing configuration, introduced by some very recent changes, the test will be repeated this week.
  • SAM test in many major sites stopped running during the weekend because token renewal was failing. The agent has been restarted this morning and things seem OK now.
    • GGUS:164917
    • It seems there is some instability in this system. Maarten will follow up to see if we can at least create some alarming for this issue.

• ALICE
  • High activity, no major problem

• LHCb reports ( raw view) -
  • processing and distributing 2023 heavy ion collisions
  • re-processing 2018 proton collision data, staging from tape nearly over
  • continuous simulation productions
  • no relevant issues

Sites / Services round table:

• BNL: NTR
• CNAF:
• EGI:
• FNAL:
  • There was a 4-hour degradation on Saturday.
  • Disk dCache has been updated to 9.2.
• IN2P3:
• JINR: NTR
• KISTI:
• KIT:
• NCBJ:
• NDGF:
• NL-T1: Last week we upgraded the Surf dCache to 9.2.
• NRC-KI:
• OSG:
• PIC:
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
• CERN storage services:
• CERN databases:
• GGUS: NTR
• Monitoring:
  • Distributed draft SiteMon availability/reliability reports for 2023 12
  • Henryk noted that he didn't receive those reports. To be checked if the federation responsible received them and also from the WLCG side Maarten will check with the project office to see who is receiving those reports.
• Middleware: NTR
• Networks:
• Security:

AOB: