Week of 220905

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: Christoph (CMS), Petr (ATLAS), Jiri (ATLAS), Maarten (ALICE), Mark (LHCb), David (IN2P3), Julia (WLCG), Allesandro (CNAF), Darren (RAL), Marian (Network), David (FNAL), Jarka (Compute), Miro (DB, Chair)

Experiments round table:

• ATLAS reports ( raw view) -
  • xroot transfers fail with client on CentOS9 stream for sites where server certificate is signed by CA, which root certificate uses obsolete SHA1 algorithm
    • also problems with sites with old (< 5.4) XRootD when using CentOS9 Stream transfer client
    • short term solution: CentOS9 system can be switched to LEGACY mode
    • long term: CAs should update their root certificates
    • Question from Maarten: Can Legacy mode be enabled easily? Answer from Petr: Only if you have root privileges. But security teams might have an issue with it.
    • Question from Petr: Can a timeline be set? Answer from Maarten: This has to be discussed with Petr in the loop. Maarten will follow up.

• CMS reports ( raw view) -
  • Night from Monday to Tuesday last week some Singularity images in the OSG repo became unaccessible for about 3h
    • CMS SAM tests and production pilots depend on them

• ALICE
  • NTR

• LHCb reports ( raw view) -
  • Issues:
    • IN2P3:
      • Pilot submission problems (GGUS:158719)
    • NIKHEF:
      • Pilot submission problems (GGUS:158718). Investigations ongoing.

Sites / Services round table:

• ASGC: NC
• BNL: NTR (US Labor Day Federal holiday - lab closed)
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3: Quarterly maintenance scheduled on Tuesday Sept. 20th. Details will be provided the week before.
• JINR: NTR
• KISTI: NC
• KIT: NC
• NDGF: NC
• NL-T1: NC
• NRC-KI: NC
• OSG: NC
• PIC: NC
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
  • HTCondor tier-0 pool upgrade of Central Managers next Monday 12/9 10-14h: OTG:0072940
  • HTCondor shared pool upgrade of Central Managers next Thursday 15/9 10-14h OTG:0072944
  • VOMS DB upgraded: OTG:0072810
• CERN storage services: NC
• CERN databases: Forced upgrades of MySQL DBoD instances to supported versions (OTG:0072813 , OTG:0072817 , OTG:0072818)
• GGUS: NTR
• Monitoring:
  • Distributed draft SiteMon availability/reliability reports for August 2022
• Middleware: NTR
• Networks:
  • Reminder: ESnet upgrades at CERN will take place on the 12-13th of September (please check details at OTG:0071902, OTG:0071903)
• Security: NC

AOB: