Week of 231016

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Laurence (Computing), Maarten (ALICE)
• remote: Andrew (TRIUMF), Borja (Chair, Monitoring), Darren (RAL), Dave (FNAL), Henryk (LHCb), Ivan (ATLAS), Julia (WLCG), Panos (WLCG), Xavier (KIT)

Experiments round table:

• ATLAS reports (raw view) -
  • Performance issues (transfer timeouts) at Glasgow (GGUS:163552)
    • Allegedly the 2hr limit and 60s timeout(Glasgow) comes from FTS rather than src/dest.
    • Ivan: Is there someone from FTS connected to the meeting to have a look into the ticket?
    • Maarten: Better to give ticket visibility to the experts, try with the CC field so they are involved in the loop

• CMS reports (raw view) -
  • NTR

• ALICE
  • Low activity levels on average, significant raw data reco activities

• LHCb reports (raw view) -
  • Smooth running, 140k jobs

Sites / Services round table:

• ASGC: NC
• BNL: upcoming outage - A scheduled SDCC Facility wide network intervention
  • aiming at moving SciDMZ to new data center,
    • move SciZone perimeter FW and upgrading it to 200 Gbps of pass-through bandwidth capability
    • Completing the transition to the new 400 GbE enabled BNL perimeter
    • Date Tue Dec 19, 2023 06:00 – 21:00 BNL LT (15h window) (Dec 19,2023 11:00 UTC - Dec 20,2023 02:00 UTC).
  • This will result in SciDMZ & Service Block connectivity disruption for SDCC Data Center environment for the duration of that time window).
  • This will affect all SDCC WAN traffic including that of BNL ATLAS T1 dCache and BNL FTS systems
• CNAF: NC
• EGI: NC
• FNAL: NTR
• IN2P3: NTR
• JINR: NC
• KISTI: NC
• KIT: NTR
• NDGF: NC
• NL-T1: NC
• NRC-KI: NC
• OSG: NC
• PIC: NC
• RAL: There will be UPS testing undertaken at R89 on the weekend of the 21/22 October. Although the UPS’s will not be available, “raw” power will be, so the testing should be transparent to services.
  • Darren: Another outage to be announced at a later meeting
• TRIUMF: NTR

• CERN computing services:
  • WLCG IAM version upgrade on October 17th at 7:00 (OTG:0144943)
• CERN storage services: NC
• CERN databases: NC
• GGUS: NTR
• Monitoring:
  • Distributed final SiteMon availability/reliability reports for September 2023
• Middleware: NTR
• Networks: NC
• Security: NC

AOB: