Week of 200504

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: Andrew (TRIUMF), Miro (Chair, DB), Christoph (CMS), Elena (CNAF), Gavin (Computing), Maarten (ALICE), Mark (LHCb), Xin (BNL), Pavlo Svirin (ATLAS), Xavier (KIT), Belinda (Storage), Brian (RAL), Pepe (PIC), David (IN2P3), Dave (FNAL)

Experiments round table:

• ATLAS reports ( raw view) -
  • stable running
  • Deployment of unified queues continuing
  • Running up to 60k for COVID

• CMS reports ( raw view) -
  • CPU usage a bit lower than usual: 150k cores production and 60k analysis
  • Folding@Home
    • Basically full HLT (about 50k cores)
    • A few thousand cores via CMS Global Pool - excluding countries with national efforts

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • Ongoing WG MC productions
    • We have ticketed all Tier 2 sites that are still running SL6 to ask them to upgrade to CC7. Not surprisingly, they have mostly said it'll have to wait until after quarantine!
  • Issues:
    • CERN: Issues with file access on EOS. Solved and ticket to be closed (GGUS:146673)
    • GRIDKA: Some ongoing file access issues. FZK are waiting on vendor. (GGUS:146379)

Sites / Services round table:

• ASGC: NC
• BNL: NTR
• CNAF: NC
• EGI: NC
• FNAL: NTR
• IN2P3: NTR
• JINR: NC
• KISTI: NTR
• KIT: NTR
• NDGF: NC
• NL-T1: NC
• NRC-KI: NC
• OSG: NC
• PIC: SD at risk for tomorrow Tue. 5/5 from 08h00 to 15h00 PIC time, to upgrade the dCache doors (to 5.2.20) and Enstore (to 6.3.4-2). There will be around 2h of reading from tapes n/a. Apart from that, the rest of interventions should be transparent for users.
• RAL: In process of upgrading site squids and planning stratum 1 upgrade.
  • Comment from Ops Meeting by Maarten. VOs with hardcoded connections not using aliases can be discovered during upgrade and subsequently corrected.
• TRIUMF: NTR

• CERN computing services: NTR
• CERN storage services:
  • CASTOR: all instances will be down tomorrow 5th of May, from 8h30 to 10h30 for a DB servers reboot campaign involving the name server DB and several stagers DBs ( OTG0056208).
• CERN databases:
  • Rolling restart of CASTOR related instances to resolve memory and swapping issues: OTG:0056208
  • Several production databases including some experiment standby databases unavailable due to Router migration: OTG:0056208
• GGUS: NTR
• Monitoring: NTR
• MW Officer: NTR
• Networks: NTR
• Security: NTR

AOB: N/A