Week of 221212

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Kate (DB, chair), Julia (WLCG), Maarten (ALICE, WLCG), Michal (ATLAS), Andrew (NL-T1), Daniele (CNAF), Andrew (TRIUMF), Douglas (BNL), Jan vE (Computing), Marian (network), Dave (FNAL), Borja (monitoring), Petr (ATLAS)

Experiments round table:

• ATLAS reports ( raw view) -
  • Issues:
    • "Read timed out" transfer failures from IN2P3-CC (GGUS:159828)
      • related to high load
    • "Unexpected server error" deletion failures at Taiwan-LCG2 (GGUS:159827)

• CMS reports ( raw view) -
  • NTR

• ALICE
  • moving forward with converting sites from single- to 8-core jobs

• LHCb reports ( raw view) -
  • NTR

Sites / Services round table:

• ASGC:
• BNL: Reminder Tape Downtime (HPSS version upgrade) Dec 19 05:00 UTC to Dec 20 02:00 UTC, Storage downtime (dCache upgrade) Dec 19, 2022 14:00 UTC to Dec 19, 2022 20:00 UTC
• CNAF: NTR
• EGI:
• FNAL: NTR
• IN2P3: NTR
• JINR:
• KISTI:
• KIT:
• NDGF: Emergency network outage tomorrow at IJS. Some atlas pools offline. Expected to last no more than 30 minutes. Details in gocdb.
• NL-T1: Reminder: this week SARA-MATRIX is having downtimes for dCache https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=33205 and tape https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=33177
• NRC-KI:
• OSG:
• PIC:
• RAL:
• TRIUMF: NTR

• CERN computing services:
  • In order to reduce power consumption over the winter months, the LxBatch HTCondor service will shutdown a container of compute nodes. In concrete terms, this will mean an overall reduction in capacity of around 31% of the available CPUs, and a reduction of 28% of the overall computing power.
    Status: draining now, expected available for shutdown on Thursday 15th December.
  • Installation of new Batch Workernodes is currently failing at CERN. It seems the mirror of the UMD-4 Yum repository is missing several packages, possibly as a result of an upstream change in signing key. A ticket with the CERN Linux team has been opened.
• CERN storage services:
• CERN databases: NTR
• GGUS:
  • A new GGUS support unit has been created under the WLCG category:
    CERN IAM services
    • It can be used to open tickets with normal priority for those services
    • Team and alarm tickets continue going through ROC_CERN,
      but can internally be reassigned in SNow as needed
    • High-priority tickets for CERN IAM services can be opened directly in SNow
• Monitoring:
• Middleware: NTR
• Networks:
  • LHCOPN: ES-PIC deployed new 100Gbps link. The old 10Gbps link was removed.
  • LHCOPN: Preparing an upgrade of the connection to NLT1 to 2x100G
• Security:

AOB: