Week of 220711

WLCG Operations Call details

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
  • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Andrew (TRIUMF), Ben (Computing), Borja (Chair, Monitoring), Daniele (CNAF), David (FNAL), Doug (BNL), Julia (WLCG), Maarten (ALICE), Mark (LHCb), Onno (NL-T1), Pinja (Security), Steven (Storage)

Experiments round table:

• ATLAS reports ( raw view) -
  • successful Run-3 start - everything works fine
  • T0 export to US/CA in rare cases exceptionally slow with throughput ~ 0.5MB/s (GGUS:157985)
    • default dCache transfer timeout 2h - in future we may ask sites to update transfermanagers.limits.transfer-time and rely on FTS to kill problematic transfers ( DMC-1278)
    • also discovered on EOSATLAS disknode with hw issues that caused few transfer failures (GGUS:157978)
  • message broker running at mb120.cern.ch:61013 was unreachable from Oxford site (failing jobs)
    • connections were blocked by CERN firewall - rule removed, but it is not clear to us why it was there
  • first ATLAS jobs successfully ran on CS9, asked FTS developers to provide gfal2 for CS9 / our CS9 ALRB environment
    • will become available in DMC testing repo in about ~ two weeks (after FTS 3.12 release)

• CMS reports ( raw view) -
  • NC

• ALICE
  • Run 3 started OK
  • Mostly business as usual

• LHCb reports ( raw view) -
  • Issues:
    • Issues transferring between PIC and CNAF/INFN.
      • Investigations on the PIC side seem to show routing problems to some IPs (GGUS:157955)
      • Ticket now opened against CNAF/INFN to see if they can verify this. (GGUS:158004)

Sites / Services round table:

• ASGC: NC
• BNL: NTR
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3: Update for last CVE done last week (on 6th and 7th July)
• JINR: NTR
• KISTI: NC
• KIT: NC
• NDGF: NC
• NL-T1: NTR
• NRC-KI: NC
• OSG: NC
• PIC: NC
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
  • Scheduling upgrade of tier-0 CEs to support token based submission on 28/7: OTG0072017
• CERN storage services: NTR
• CERN databases: NTR
• GGUS:
  • Last week's release has been postponed to Aug 3
• Monitoring: NTR
• Middleware: NTR
• Networks: NC
• Security: Advisory for a critical vulnerability sent last week. This is likely getting some more updates.

AOB: