Week of 210118

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
    • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.
  • For now, Vidyo is a backup option.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: Miro (Chair), Alberto (Monitoring), Elena (CNAF), Francesco (CNAF), Maarten (ALICE), Michal (ATLAS), Onno (NL-T1), Ville (NDGF), Vincenzo (CNAF) Daniel (Security), Xavier (KIT), Pepe (PIC), David (IN2P3), Christophe (CMS), Darren (RAL), Dave (FNAL), Borja (Monitoring) , Andrew(TRIUMF), Julia (WLCG), Concezio (LHCb)

Experiments round table:

• ATLAS reports ( raw view) -
  • Issues:
    • "No write pools configured" transfer errors at FZK-LCG2 (GGUS:150234)
    • certificates issued by the provider in Switzerland became revoked, making Swiss sites fail (GGUS:150227, GGUS:150221)
      • the IGTF bundle was fixed and will be released on Monday
      • Updates on sites and also FTS servers needed
    • lost files at NDGF - declared lost by DDM ops
    • "The certificate has been revoked" transfer errors from Taiwan-LCG2 (GGUS:150239)
    • both of two WAN circuits broke at the same time.
      • "Connection terminated abruptly" transfer errors at Taiwan-LCG2 (GGUS:150177)
      • now, WAN circuit provider has finally got an emergency backup circuit

• CMS reports ( raw view) -
  • Good CPU utilization and no big troubles

• ALICE
  • Continued high activity
  • No major issues

• LHCb reports ( raw view) -
  • namespace migration at SARA-matrix tomorrow morning.

Sites / Services round table:

• ASGC: NC
• BNL: NC
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3: NTR
• JINR: NTR
• KISTI: NTR
• KIT: NTR
• NDGF:
  • Power outage at University in Bergen at last Saturday. The dCache nodes there were down. atlas.bluegrass.nsc.liu.se maintenance break on Tuesday.
• NL-T1:
  • The company SURFsara (formerly SARA) is now officially called SURF. The Grid site is still called Sara-matrix.
  • Sara-matrix will do the LHCb namespace migration tomorrow (19th). This should only affect LHCb users. There might be a small performance degradation for other users, regarding namespace operations like directory listings. https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=30033
• NRC-KI: NC
• OSG: NC
• PIC:
  • Tuesday 19th of January, PIC will be in a complete scheduled downtime due to the following interventions: dCache upgrade, HTCondor upgrade. PIC will be in OUTAGE from 19-01-2021 09:00 (PIC local time) until 19-01-2021 15:00 (PIC local time).
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: NC
• CERN storage services: NC
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
  • Final availability reports for December 2020 sent around
  • OTG0061473 Migration of WLCG Transfers monitoring (FTS and XRootD) to ES + CRIC
• Middleware: NC
• Networks:
  • GGUS:150048 - Issue with connectivity due to MTU (jumbo frames) between QMUL and TRIUMF; ticket was opened with GEANT and the issue was confirmed and is being followed up with Canarie and Internet2 (see ticket for more details).
• Security:
  • dCache vulnerability [EGI-SVG-2020-16939] is now public and the fixes are included in the release. Please check if the patch is installed.

AOB: