Week of 210830

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
  • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: Alberto (monitoring), Andrew (TRIUMF), Chien-De (ASGC), Christoph (CMS), Claudia (CNAF), Daniel (security), Dave (FNAL), Gavin (T0 manager), Jens (NDGF), Joao (storage), Julia (WLCG), Maarten (ALICE + WLCG), Mark (LHCb), Mihai (FTS), Peter (ATLAS), Priyanshu (computing), Steven (FTS)

Experiments round table:

• ATLAS reports (raw view) -
  • CERN DC power cut, all services recovered about 2hrs afterwards, 4-6pm CEST
  • FTS DBoD issues over weekend, backlog of transfers now recovered
  • Related to FTS issue, PIC bandwidth congestion returns. We'll shelve this issue pending network upgrade later in year
  • FTS upgrade to mysql 8.x OTG0065688, done ok?
    • see below
  • Rucio LANDB set incident followup report at RucioAuthSvcInc20210820

• CMS reports (raw view) -
  • Power outage at CERN Thu (26.8.) in the afternoon
    • OTG:0065743, OTG:0065582#
    • Also central CMS services affected - Recovery relatively quickly

• ALICE
  • Very high activity, no major issues

• LHCb reports (raw view) -
  • Activity:
    • MC and WG productions; Staging Ongoing; User jobs.
  • Issues:
    • RAL: Still have transfer issues but progress is being made (GGUS:153532)
    • CNAF: Cannot access some files locally. Investigations ongoing (GGUS:153578)
    • If you Google 'GOCDB' you get directed to the pre-prod site and there's no discernible difference except the DTs aren't registered properly. Can banner be added please?
      • GGUS ticket opened after the meeting

Sites / Services round table:

• ASGC: NTR
• BNL:
• CNAF: NTR
• EGI:
• FNAL:
  • Dave: essentially NTR
• IN2P3:
• JINR:NTR
• KISTI:
• KIT: NTR
• NDGF: NTR
• NL-T1:
• NRC-KI:
• OSG:
• PIC:
• RAL:
• TRIUMF: NTR

• CERN computing services:
  • HTCondor pools affected by Power Incident, 26th August 2021 OTG0065748, GOCDB
• CERN storage services:
  • FTS3-Atlas incident on Saturday, 28th August 2021: OTG0065808
  • FTS3-Atlas database upgrade to MySQL8 on 10:00 Monday, 30th August 2021: OTG0065688
• CERN databases:
• GGUS: NTR
• Monitoring:
  • Alberto:
    • the power cut allowed us to see our Kafka setup work as designed:
      no data was lost, all was delivered as services came back
• Middleware: NTR
• Networks: NTR
• Security: Exploit for CVE-2021-33909 (Local privilege escalation) is now public. Please patch your Linux kernels.

AOB:

• Julia:
  • those T1 sites who did not yet provide the additional input that was
    requested to help prepare for the tape challenge in Oct: please do so
  • the Ops Coordination meeting on Thu has the tape challenge on its agenda