Week of 220530

WLCG Operations Call details

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
  • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Kate (chair, DB), Maarten (ALICE, WLCG), DavidB (IN2P3), Michal (ATLAS), Pinja (security), Daniele (CNAF), Andrew (TRIUMF), Christoph (CMS), Douglas (BNL), Federico (LHCb), Darren (RAL), Dave (FNAL)

Experiments round table:

• ATLAS reports (raw view) -
  • Issues:
    • mass queue blacklisting on Friday evening
      • Database issue suspected to be the underlying cause (OTG:0071193)
    • File fails to stage from NDGF-T1_DATATAPE for a month (GGUS:157366)
    • "no such file or directory" transfer failures from CERN-PROD_TZERO (GGUS:157301)
      • the ticket is without response from CERN for almost two weeks

• CMS reports (raw view) -
  • Last week job slot usage was found to be low at CC-IN2P3 from CMS
    • Likely not a site issue
    • Being investigated
  • Deletion of (test-)data at T1 and T0 tape endpoints from recent tape challenge
    • Communication with sites via GGUS

• ALICE
  • CVMFS problem affecting 2 sites in Japan and Thailand has been solved
    • The Stratum-1 service at ASGC had run out of disk space

• LHCb reports (raw view) -
  • NTR

Sites / Services round table:

• ASGC:
• BNL: ntr
• CNAF: the computing power of the CNAF farm will be reduced by about 40% until Monday, May 30th EOB: https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=32424
• EGI:
• FNAL: ntr
• IN2P3: NTR
• JINR: ntr
• KISTI:
• KIT: Reminder for site downtime on June 20th/21st
• NDGF:
• NL-T1: NTR
• NRC-KI:
• OSG:
• PIC:
• RAL: RAL will be observing the two Jubilee public holidays on the 2nd and 3rd June. Despite the majority of staff being on leave we will be running our “daily checks” plan from the 2nd – 6th June. In practice this means that where ALARM tickets we be dealt with as normal, standard GGUS tickets many not be actioned (or indeed seen) for up to 24hours.
• TRIUMF: NTR

• CERN computing services:
• CERN storage services:
• CERN databases: ADCR instance rebooted on Friday due to cluster configuration issue (OTG:0071193)
• GGUS: NTR
• Monitoring:
• Middleware:
  • A CE token support campaign will be launched for EGI sites
    • To get HTCondor and ARC CEs on the right versions and configurations
• Networks: NTR
• Security: NTR

AOB:

• NOTE: the operations meeting next Mon will be virtual .
• You may provide relevant incidents, announcements etc. for the operations record.