Week of 200210

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Maarten (ALICE), Miro (Chair), Olga (Compute), Diogo (Storage), Vincent (Security), Petr (ATLAS) , Pepe (PIC)
• remote: Andrew (NL-T1), Christoph (CMS), Darren (RAL) , Dave (FNAL), Elena (CNAF), Federico (LHCb), David (IN2P3), Xavier (KIT), Sang Un (KISTI), Andrew (TRIUMF), Vincenzo (EGI)

Experiments round table:

• ATLAS reports ( raw view) -
  • ATLAS S&C Week
  • Monitoring issues due to slow DB queries RQF1519692, looks fine for now
  • Ongoing reprocessing (will stage new data within 2-3 weeks)
  • We are planning to enable non-GridFTP TPC soon as a preferred protocol for few sites
    • XRootD and WebDAV transfers between production ATLAS storage sites

• CMS reports ( raw view) -
  • Backend database of FTS at CERN overloaded
    • (At least) partly caused by huge amount of user files to be transferred
    • Situation is improving following some actions by FTS and database on demand teams
  • Host/service certificate of voms2.cern.ch was prolonged only a few days before expiring
    • Caused some issue, when proxies were generated: INC:2300611
    • Request: Please re-new host/service certificate earlier, ideally a month before expiring
      • Maarten: normally the certificates are renewed on time; an issue with the renewal process prevented that on this occasion

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • Stripping campaign ongoing, occupying most of T0/1 capacity (no T2s)
    • Staging re-started last week, almost finished everywhere
  • Issues:
    • no significant issues

Sites / Services round table:

• ASGC: NC
• BNL: NC
• CNAF: NTR
• EGI: New CentOS 8 repository available
• FNAL: Affected by voms certificate issue reported by CMS
• IN2P3: NTR
• JINR: NC
• KISTI: NTR
• KIT:
  • Brief DNS outage (~ 2 min) on Thursday evening. Nearly all services recovered properly, except for the LHCb dCache dcap doors, which were handled by the on-call engineer within half an hour.
  • Still working on deployment issues with the new compute nodes: false routing tables, wrong IPv6 addresses configured, grid-security files missing, etc.
• NDGF: NC
• NL-T1: SURFsara had planned a dCache upgrade for today but this has been postponed to combine it with network maintenance. The new downtime will be in the week of 2 to 6 March. GOC:28371
• NRC-KI: NC
• OSG: NC
• PIC: Tomorrow Tuesday 11th we are enabling MTU 9000 in some VLANs. In principle this should be transparent.
• RAL: NTR
• TRIUMF:NTR

• CERN computing services: NTR
• CERN storage services:
  • Storage nodes offline due to datacenter network issue (EOSCMS, EOSATLAS, EOSPUBLIC, EOSPROJECT) (OTG0054796)
• CERN databases: NTR
• GGUS:
  • Last week's maintenance was essentially successful.
    • An issue with opening SNow tickets for CERN was cured Wed afternoon.
• Monitoring: NTR
• MW Officer: NTR
• Networks: NTR
• Security: NTR

AOB: