Week of 190513

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Miro (Chair, DB), Dmitry (LHCb), Alberto (Monitoring), Eddie (Storage), Ben (Compute), Julia (WLCG)
• remote: Jens (NDGF), Alexander (NL-T1), Xin (BNL), Di (TRIUMF), David (IN2P3), Dave (FNAL)

Experiments round table:

• ATLAS reports ( raw view) -
  • NTR
  • A gentle reminder of the CentOS7 site deployment deadline - 01.06.2019. Please update your site status here.
  • No one present in person - An ATLAS Work Flow Management Technical Interchange meeting in Ljubljana. If any comments / questions, please add them here and they will be replied later over e-mail.

• CMS reports ( raw view) -
  • NTR

• ALICE -
  • Apologies: ALICE ops experts are traveling to the yearly Tier-1/Tier-2 workshop, May 14-16, Bucharest
  • NTR

• LHCb reports ( raw view) -
  • Activity
    • User jobs, MC productions, and staging this week
  • Issues
    • CERN:
      • since Friday communication problems between lbvobox'es and CNAF CE (job submission gets stuck) INC:1983869
      • observe low efficiency of transfers from CERN WNs to outside SEs (GGUS:141112) (The problem starts from ~8th Apr, but the amount of such transfers is very low, so they are not visible in the overall plots)

Sites / Services round table:

• ASGC: ASGC WN are all CentOS7 already.
• BNL:NTR
• CNAF: NC
• EGI: NC
• FNAL: Reported upgrade of dCache
• IN2P3: NTR
• JINR: NTR
• KISTI: NC
• KIT: Looking into why the CAM SAM test for ARC-1 is failing since six hours.
• NDGF: dCache 5.1 upgrade postponed due to staff on conference and the upcoming dCache Workshop. New try in two weeks.
• NL-T1: Upgrade of the tape back-end at SURFsara starting on Monday 20/5 15:00 UTC until Tuesday 21/5 15:00 UTC. Reading and writing to tape will not be possible during this upgrade.
• NRC-KI: NC, Migration to CC7 start reported by Dmitry.
• OSG: NC
• PIC: NC
• RAL: NTR
• TRIUMF: All old WNs at old data centre had been migrated to SL7 after power outage downtime was over.

• CERN computing services:
  • OTG:0050233 - 4 more CEs will be defaulted to CC7 resources (that are now ~2/3 of the CERN batch pool).
  • OTG:0050204 - CERN MyProxy server authorized_renewer DN cleanup to reduce exposure (not in DNS OR not used in the last month). 771 DNs -> 112 DNs.
    • 5 DNs had to be re-added (mostly CMS).
• CERN storage services:
  • FTS:
    • FTS CMS/LHCb down due to DB migration (OTG:0050270), ~10 min on the 16th of May during the time window 8:30 to 10:30 CEST
    • FTS Pilot upgrade to 3.9.0 ( OTG:0050314). The service will be down during the upgrade on the 15th of May from 10:00 to 12:00 CEST. Clients will need to schedule the transfers to another FTS instance for the duration of the upgrade.
• CERN databases: NTR
• GGUS: NTR
• Monitoring: A/R reports for April 2019 distributed by the WLCG Office. Feedback by 17 May.
• MW Officer: NTR
• Networks: NTR
• Security: NTR

AOB: