Week of 220207

WLCG Operations Call details

• For remote participation we use Zoom: https://cern.zoom.us/j/99591482961
  • The pass code is provided on the wlcg-operations list.
  • You can contact the wlcg-ops-coord-chairpeople list (at cern.ch) if you do not manage to subscribe.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Andrew P (NL-T1), Andrew (TRIUMF), David (IN2P3), Borja (Chair, Monitoring), Christoph (CMS), Concezio (LHCB), Daniele (CNAF), David (FNAL), Doug (BNL), Julia (WLCG), Maarten (ALICE), Marian (Network), Martin (Computing), Michal (ATLAS), Pepe ( PIC), Pinja (Security), Ville (NDGF), Xavier (KIT)

Experiments round table:

• ATLAS reports ( raw view) -
  • Issues:
    • "Connection timed out" deletion errors at RAL-LCG2 (GGUS:155856)
      • deletion of small files -> deletion tried to delete as fast as possible to free space
      • DDM ops reshuffled the deletion queue to delete some bigger files

• CMS reports ( raw view) -
  • Migration to tokens in the Global HTCondor Pool has started
    • OSG sites are being targeted first
    • GGUS tickets (for communication) will be opened as needed
  • No major issues otherwise

• ALICE
  • NTR

• LHCb reports ( raw view) -
  • running at 150k jobs without significant issues
  • trying to progress on a couple of long-standing tickets

Sites / Services round table:

• ASGC:
• BNL: IPv4 saturation
  • Doug: Why so many IPv4 transfers, shouldn't they be IPv6? is FTS taking a wrong default?
  • Maarten: Better to check by opening a ticket with Network WG first, otherwise open one for FTS to check for the correctness of your theory

• CNAF: February 8th from 6:00PM to 12:00AM, transparent hw intervention on the redundant general-IP router is scheduled. LHCONE/OPN will not be affected: https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=31717
• EGI: NC
• FNAL: NTR
• IN2P3: Site will be in downtime on March 15th for quarterly maintenance. Nothing else to report.
• JINR: NC
• KISTI: NC
• KIT: dCache servers restarted on Friday
• NDGF: There were DNS issues in couple of clusters over the weekend. They have now been fixed.
• NL-T1: February 15th, SARA-MATRIX dCache upgrade to 7.2. https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=31801
• NRC-KI: NC
• OSG: NC
• PIC: NTR
• RAL: NTR
• TRIUMF: Reminder: Outage Feb 9 UTC:17:00 network, storage, os updates. GOCDB:https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=31705

• CERN computing services: NTR
• CERN storage services: NC
• CERN databases: NC
• GGUS: NTR
• Monitoring:
  • Distributed draft SiteMon availability/reliability reports for Jan 2022
• Middleware: NTR
• Networks: NTR
• Security: Updated information sent about a critical vulnerability.

AOB: