Week of 191216

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Julia (WLCG), Kate (WLCG, chair), Maarten (WLCG, ALICE), Gavin (computing), Borja (monit), Vincent (security), Enrico (storage), Ivan (ATLAS)
• remote: Andrew (NL-T1), Marcelo (CNAF), Vladimir (LHCb), Darren (RAL), Xavier (KIT), Di (TRIUMF), Christoph (CMS), Caio (CMS), Xin (BNL), Dave M. (FNAL)

Experiments round table:

• ATLAS reports ( raw view) -
  • NTR

• CMS reports ( raw view) -
  • Preparing for Xmas break
    • Production will continue (ideally in full swing)
    • System mostly unattended - computing folks will occasionally check their mails
  • Many thanks to all for the rather productive year 2019 and have a good start in 2020

• ALICE -
  • Expectations for the end-of-year break:
    • steady MC production
    • raw data processing
    • analysis probably at a lower level than usual
  • Thanks to all sites and experts for another great year!
  • Season's greetings and best wishes for 2020 !

• LHCb reports ( raw view) -
  • Activity:
    • Running MC simulation, WG productions, user analysis.
  • Issues:
    • RAL: failing access by protocol to files in ECHO

Sites / Services round table:

• ASGC: nc
• BNL: NTR
• CNAF: Warning Downtime for Urgent intervention on one of the two redundant main electric powerlines Downtime 28176
  • We wish you all great holidays and an awesome 2020!
• EGI: nc
• FNAL: NTR
• IN2P3: nc
• JINR: NTR
• KISTI: nc
• KIT:
  • lhcbsrm-kit.gridka.de has been updated to dCache version 5.2 on last Wednesday without any issues. Now all of the GridKa dCache SEs are running on version 5.2 and are therefor TPC-ready.
  • The namespace migration for LHCb has been postponed to next year.
  • Today we've replaced the host certificates of the dCache doors service as xrootd endpoint for CMS. Now they have a different primary subject name, which matches the round robin cmssrm-kit.gridka.de. With that, even outdated xrootd clients (pre-4.9) can work with those hosts while gsi authentication is enabled (GGUS:143740).
  • arc-4.gridka.de has been in downtime since Wednesday and has been taken out of production in GOC-DB. Next year, we will replace it with a new machine.
  • GridKa will run on best effort over the holidays, as usual.
• NDGF: nc
• NL-T1:
  • Surfsara are in the final steps of recovering grid services after a security incident on 5th December forced them to offline their grid services.
  • A hardware failure at Nikhef on Saturday morning slowed IO to near zero on most of the VMs controlling our grid services. A failing multipath route between our VM host machines and their storage was identified as the cause and shutdown, restoring service late Saturday afternoon.
• NRC-KI: nc
• OSG: nc
• PIC: nc
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: Limited batch capacity today due to power intervention in P8
• CERN storage services: Successful EOS ATLAS intervention on Mon 16th morning, instance migrated to new hardware. All storage services will run in best effort during the end-of-year closure. Experiments are suggested to avoid running heavy tests or new activities that might compromise the stability of the systems.
• CERN databases: nc
• GGUS:
  • For the end-of-year break: GGUS is monitored by a system connected to the on-call service. In case of total GGUS unavailability the on-call engineer (OCE) at KIT will be informed and will take appropriate action. If GGUS is available but there is a problem with the workflow (e.g. ALARM to CERN doesn't generate email notification to the operators), then WLCG should submit an ALARM ticket, notifying site FZK-LCG2 (DE-KIT), which triggers a phone call to the OCE. As a last resort, the FZK-LCG2 emergency e-mail or telephone number published in the GOCDB can be contacted.
• Monitoring:
  • Some issue mailing the draft reports, they were officially sent the 11th (final reports for November will only be sent beginning next year)
• MW Officer: nc
• Networks: NTR
• Security: A report for the security incident at SURFsara mentioned will soon be sent as usual to site security contacts

AOB:

Season's Greetings!

• THANKS for your help in making 2019 a very successful year for WLCG!
  • Further challenges and opportunities await us in 2020...

• Next meeting: Mon Jan 6