Week of 190304

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Miro (DB, Chair), Maarten (ALICE), Michal (ATLAS), Mark (LHCb), Vincent (Security), Borja (Monitoring), Marian (Network), Gavin (Computing), Andrea (Storage)
• remote: Marcelo (CNAF), Onno (NL-T1), John (RAL), Dmytro (NDGF), Di (TRIUMF), Dave (FNAL), David (IN2P3)

Experiments round table:

• ATLAS reports ( raw view) -
  • Activities:
    • normal activities
  • Problems
    • CERN
      • transfers to CERN-PROD_LOCALGROUPDISK failed with "500 mkdir() fail" (GGUS:139796) - atlas003 user did not have proper permissions - permission fixed
      • timeouts in transfers to CERN (OTG:0048534) - EOSATLAS degradation
      • jobs failing with "None of the given files have been uploaded" + transfer timeouts (GGUS:139993, OTG:0048587) - EOSATLAS degraded
      • "No space left on device" transfer failures (GGUS:140025, INC:1931380) - accounting shows there should be O(100)TB of free space - There is not enough capacity added and the instance groups run out of space (related to Wigner draining campaign)
    • BNL
      • jobs at BNL_PROD_UCORE fail with "The available CRL has expired" (GGUS:140016) - fixed
      • transfers to several US cloud sites fail with "The available CRL has expired " (GGUS:139989) - fetch-crl was not working on FTS hosts at BNL - fixed
    • RAL
      • lcgvo-frontier05.gridpp.rl.ac.uk unavailable - logrotate wasn't running and the disk filled up - fixed
    • IN2P3-CC
      • ccin2p3-ccfrontier03 down (GGUS:139917) - I/O issue on back-end storage of databases - fixed
      • low throughput to IN2P3-CC (GGUS:139756) - problem with packet loss - ticket closed with "Known problems with the network"

• CMS reports ( raw view) -
  • Note: CW will be traveling on Monday, if there are delays CW might miss the meeting
  • Good resource utilization ~230k cores
    • Usual mix: ~75% production, 25% analysis
  • No major issues

• ALICE -
  • Lowish activity during the weekend due to lack of MC productions
    • Fixed this morning
  • EOS intervention concluded without problems

• LHCb reports ( raw view) -
  • Activity
    • User jobs and MC production
  • Issues
    • CERN: Some ongoing EOS issues both writing and reading GGUS:139927
    • CERN/VOMS: Proxy renewal for SAM tests has stopped working. VOMS team investigating GGUS:139920 (Update: This looks like it is now fixed!)

Sites / Services round table:

• ASGC: NC
• BNL: issue with updating CRLs (see ATLAS report above), some crl_url become not reachable from BNL. Fixed by switching to OSG cvmfs repo for CRLs, while continuing investigation on the crl_url access issue.
• CNAF: Cloud team has tested SQUID 4.4 and have not seen any issue. They want to upgrade the farm but according to documentation the max supported version is 2.x. We would like to know what other sites are doing about it and what WLCG thinks about it.
  • Note from chair - Question clarified in follow-up e-mail by Maarten: "The baseline table mentions the minimum acceptable version per product. If a newer version works fine for all parties concerned, use it as you wish."
• EGI: NC
• FNAL: NTR
• IN2P3: About network problem, a ticket has been opened to RENATER. See ATLAS report above.
• JINR: The number of running jobs lowered for a couple of days because of kernel upgrade and reboot of WNs.
• KISTI: NC
• KIT: NTR
• NDGF:
  • NT1 would like to request recomputation of Av/Re metrics starting June last year. Reasons are bugs in the probes and monitoring software from CERN side, that have been hammering us all this time, impacting our metrics. See GGUS:139979
  • Note from chair - November 2018 can be considered for recomputation
• NL-T1:
  • SARA-MATRIX downtime last week went reasonably well. dCache pools not 100% stable, we have a cron job that checks the pools and restarts them if necessary. Mainly Atlas affected. Investigating.
• NRC-KI: NC
• OSG: NC
• PIC: NC
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
• CERN storage services:
  • FTS: FTS ATLAS and CMS/LHCb instances upgrade to FTS v3.8.3, tomorrow from 14:00 to 17:00 CET (OTG:0048514)
• CERN databases:
  • LCGR database cluster reconfiguration completed OTG0048557
• GGUS: NTR
• Monitoring:
  • Draft availability/reliability reports for February 2019 sent around
  • INC:1927596, INC:1927592 SAM/ETF LHCb tests were failing due to a missing LHCb credential after LHCb service certificate was changed. A general recomputation will be performed for all sites starting 26/02 (from 26/02 - 04/03 the results will be marked as unknown).
• MW Officer: NTR
• Networks: NTR
• Security: NTR

AOB: