Week of 240212

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Borja (Chair, Monitoring), Willy (Monitoring)
• remote: Andrew (TRIUMF), Andrew (Nikhef), Carmen (CNAF), Christoph (CMS), Concezio (LHCb), Dave (FNAL), Henryk (NCBJ), Julia (WLCG), Laurent (IN2P3), Michal (ATLAS), Panos (WLCG), Pepe (PIC), Xavier (KIT)

Experiments round table:

• ATLAS reports ( raw view) -
  • DC24 started
  • Issues:
    • "User timeout over" staging failures at BNL-OSG2_MCTAPE (GGUS:165182)
      • SRM contention
    • "Space manager timeout" transfer failures to BNL-OSG2_DATATAPE (GGUS:165297)
    • CERN-PROD and CERN-T0 squids down (GGUS:165231)
      • the manifests are being made compatible with alma9 - firewall rules were incorrect
    • HT-CondorCE failing ETF/SAM token checks
      • test end with "Job has failed with status: HELD"
      • crashed agent restarted

• CMS reports ( raw view) -
  • CMS ETF Token renewal again failing during the weekend
    • GGUS:165298 (and GGUS:165299 was made a slave ticket, because both address the same issue)
    • Christoph: Wondering if it might be somehow related to experienced ATLAS problems
    • Michal: Timeline seems different so not likely
  • About a third of the CMS disk endpoints are enabled for token based authentication
  • DC24 has started with T0->T1 export exercise

• ALICE
  • Low to normal activity on average.
  • No major issues.
  • Raw data replication to the tape SEs at T0 and T1 sites has started on Friday.
    • The operation should overlap with the whole of DC24.
      • This avoids a need for any fake data.

• LHCb reports ( raw view) -
  • DC24:
    • functional tests performed; we start tomorrow morning;
    • this week we exercise T0->T1 exports
    • tokens access configured for some Tier1s
  • usual activities, no major issues

Sites / Services round table:

• BNL: NTR
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3:Site will be in scheduled downtime on March 12th for quarterly maintenance
• JINR: NC
• KISTI: NC
• KIT: NTR
• NCBJ: NTR
• NDGF: NC
• NL-T1: Nikhef had a high failure rate for third party transfers at the start of the testing for DC24, see GGUS:165263. Our limits for the number of simultaneous third part transfers were too low (at 1000) and blocked new transfers at times. The limits have been increased to 10000.
• NRC-KI: NC
• OSG: NC
• PIC: NTR
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: NC
• CERN storage services: NC
• CERN databases: NC
• GGUS: NTR
• Monitoring: SiteMon final reports for January 2024 sent around
• Middleware: NTR
• Networks: NC
• Security: NC

AOB: