Week of 211220

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
  • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Kate (DB, chair), Julia (WLCG), Maarten (ALICE, WLCG), Xavier (KIT), Pinja (security), Michal (ATLAS), Andrew (TRIUMF), Mihai (FTS), Christoph (CMS), Claudia (CNAF), Nils (computing), DaveM (FNAL), Douglas (BNL), Steve (FTS)

Experiments round table:

• ATLAS reports (raw view) -
  • Issues:
    • 8 files lost at CNAF tape - all recovered
    • Getting inputs to or outputs out of RAL is painful
      • most failures on RAL PQs are transfer timeouts
  • Page for logging major incidents

• CMS reports (raw view) -
  • Datasets used by CMS SAM & Hammer Cloud test got accidentally removed from sites GGUS:155298
    • Subject says T1_RU_JINR & T2_DE_RWTH, but also other sites were affected
    • Root cause still being investigated
    • Site metrics are being corrected
  • Best wishes for a relaxing Xmas holiday and a good year 2022

• ALICE
  • Mostly business as usual
  • Thanks to all sites and experts!
  • Best wishes for 2022 !

• LHCb reports (raw view) -

Sites / Services round table:

• ASGC: NTR
• BNL: 4 hour down time expect for HPSS. likely 29-Dec. decision made in next few days.
• CNAF: NTR
• EGI:
• FNAL: NTR
• IN2P3:
• JINR: NTR
• KISTI:
• KIT: No issues to report. GridKa enters best-effort operation mode on Friday, 24th of December up until 3rd of January. Of course, on-call surveillance is in effect - as always.
  • Tape up and running since ~Tuesday lunchtime after the log4j patching
• NDGF:
• NL-T1:
• NRC-KI:
• OSG:
• PIC: NTR. Happy xmas and happy new year to everyone!
• RAL:
• TRIUMF: NTR

• CERN computing services: Back to normal capacity after intervention at Point 8 last Thursday.
• CERN storage services:
  • FTS: Updated Gfal2 v2.20.2 on all production instances (update procedure was transparent to the end-users). The update was put in prod more than 2 weeks ago.
• CERN databases: NTR
• GGUS:
  • The service was down from Dec 15, 17:00 UTC to Dec 16, 10:35 UTC
    • The cause was an unplanned outage of its DB that has been followed up
• Monitoring:
  • Distributed final SiteMon availability/reliability reports for November 2021
• Middleware:
• Networks: NTR
• Security: Log4shell: various updates in public sources

AOB:

Season's Greetings!

• THANKS for your help in making 2021 another successful year for WLCG !
  • Best wishes for the New Year !

• Next meeting: Mon Jan 10