Week of 191118

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Borja (Chair, Monitoring), Cristian (Storage), Dima (LHCb), Ivan (ATLAS), Julia (WLCG), Kate (WLCG), Maarten (ALICE), Olga (Computing)
• remote: Alexander (NL-T1), Christoph (CMS), Darren (RAL), Dave (FNAL), Di (TRIUMF), Marcelo (CNAF), Pepe (PIC), Sang-Un (KISTI), Xavier (KIT), Xin (BNL)

Experiments round table:

• ATLAS reports ( raw view) -
  • Normal Operations
  • An issue concerning DPM read access via WebDAV is being followed up with the DPM team.

• CMS reports ( raw view) -
  • CMS HLT could not reach CMS EOS instance at CERN GGUS:144051

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • Smooth running at ~120K jobs: MC simulation, WG productions, user analysis.
    • have staged 2015,2016,2017 data for the re-strippign campaign, awaiting for validation by physics groups
  • Issues:
    • no significant issues

Sites / Services round table:

• ASGC: NC
• BNL: HPSS tape system will not be available on Nov 19 from 9am to 5pm (EST), due to tape library and CORE server relocation. dCache SE will still be available.
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3: Ticket about Singularity for LHCb solved.
• JINR: NTR
• KISTI: NTR
• KIT:
  • One LHCb disk pool node was rebooted on Sunday evening.
  • We'll offer to update the ATLAS dCache SE - atlassrm-kit.gridka.de - to version 5.2 on Wednesday next week through our VO contact.
  • Hopefully, we've finally resolved all issues with ARC-CE-4 by now. In any case, we've implemented measurements to counter the same issues seen so far through Puppet.
• NDGF: NC
• NL-T1: On 19-11-2019 SURFsara has a downtime due to a minor dCache upgrade and maintenance on the tape library
• NRC-KI: NC
• OSG: NC
• PIC: Understanding the origin of some files corrupted in our storage system for ATLAS. There is a GGUS that follows this: GGUS:143878
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
  • OTG:0053358 Thu 21st November, 11.00: We'll turn off setuid Singularity on the CERN batch service - unprivileged user namespaces are already enabled.
  • OTG:0053356 Condor scheduler access issue this morning.
• CERN storage services: NTR
• CERN databases: NC
• GGUS: NTR
• Monitoring: NTR
• MW Officer: NC
• Networks: NTR
• Security: NTR

AOB: