Week of 210315

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
  • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Kate (DB, chair), Julia (WLCG), Maarten (WLCG, ALICE), Onno (NL-T1), Pavlo (ATLAS), Xavier (KIT), Claudia (CNAF), Borja (monitoring), Federico (LHCb), Christoph (CMS), Stephan (CMS), Darren (RAL), Daniel (security), Andrew (TRIUMF), David B (IN2P3), Marian (monitoring, network), Petr (ATLAS), Ville (NDGF), Dave M (FNAL), Pepe (PIC)

Experiments round table:

• ATLAS reports ( raw view) -
  • SARA-MATRIX GGUS:150757 reopened because of different CentOS8 / 1024bit keys issue ( FTS merge request for 2048bit keys, FTS-1700)
  • UKI-LT2-RHUL GGUS:150820 transfer failures for data recovery
  • INFN-T1 (StoRM) to AGLT2 (dCache) 87.3% big file transfers fails
  • TOKYO-LCG2 to CERN with 30% failure rate after moving to HTTP-TPC GGUS:151000
  • Australia-ATLAS troubles with non-LHC and now LHCONE CERN connectivity GGUS:150596
  • SARA will stop receiving ATLAS jobs in 2 weeks. No response on GGUS:145904

• CMS reports ( raw view) -
  • Good CPU usage and no major trouble
  • RAL disabled for production jobs in agreement with local CMS contacts
    • CPU efficiency observed to be very bad due to (remote) IO patterns
    • Mitigations are being worked on

• ALICE
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • MC and WG productions; User jobs.
  • Issues:
    • CERN: Migration to CTA is ongoing. Testing, re-opening the valves this afternoon if everything's OK.

Sites / Services round table:

• ASGC: nc
• BNL: NTR
• CNAF: Tier-1 will be at risk in 22/03-25/03 and 29/03 - 01/04 due to electrical works, downtime notices will be reported in a GOCDB link this afternoon.
• EGI: nc
• FNAL: NTR
• IN2P3: A tape maintenance will be done tomorrow at IN2P3-CC. SRM-nearline service will be in downtime for the whole day.
• JINR: NTR
• KISTI: nc
• KIT: The CERN CRL expired on Sunday evening on all CEs, which is why job submission failed for everyone. The issue was a consequence to a change in configuration management last week, which is fixed now.
• NDGF: ntr
• NL-T1:
  • The workaround for the crypto-policies issue reported last week was not properly installed on two of our nodes. It was fixed this morning. Please let us know if you see any more problems. Atlas ticket: GGUS:150757 LHCb ticket: GGUS:150928
  • Our tape backend will be in maintenance tomorrow morning. A fix will be installed that hopefully fixes stability issues we're having the last month or two. https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=30348
  • Another tape maintenance will be planned around March 28-30th (not yet in GOC).
• NRC-KI: nc
• OSG: nc
• PIC: NTR
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: nc
• CERN storage services: nc
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
  • Draft SiteMon reports for February sent around
• Middleware:
  • The HTCondor release with the improved ARC CE client has been delayed. New ETA TBC.
• Networks: NTR
• Security: NTR

AOB: