Week of 181112

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Julia (WLCG), Kate (WLCG, DB, chair), Maarten (ALICE, WLCG), Ivan (ATLAS), Frank B (ATLAS), Gavin (comp), Paul (storage), Borja (monit), Joao (storage), Rob (LHCb)
• remote: Xavier (KIT), Marcelo (CNAF), Sang-Un (KISTI), Darren (RAL), Alexander (NL-T1), David B (IN2P3), Ville (NDGF), Dave M (FNAL)

Experiments round table:

• ATLAS reports ( raw view) -
  • Production
    • Continuing switching to unified queues and Harvester for fully dynamic sites (ongoing for FR, CA, US and NL cloud)

Ville (NDGF) reported issues with Rucio, Ivan (ATLAS) is following up, as no issues are reported by ATLAS monitoring.

• CMS reports ( raw view) -
  • CEPH problems at Wigner badly affected CMS Frontier service
    • Wigner machines got isolated and temporaily replaced by CERN located integration machines
    • OTG:0046831 and OTG:0046832
  • Overall good CPU utilization
    • Production: 160k cores
    • Analysis: 50k cores

Xavier (KIT) reported that the availability reports for CMS are not working since the weekend. This is to be checked.

• ALICE -
  • Lowish activity on average due to temporary lack of MC productions

• LHCb reports ( raw view) -
  • Activity
    • Data reconstruction for 2018 data
    • User and MC jobs
  • Site Issues
    • CERN: spike in failed jobs on Sunday, currently investigating, FTS delegation issue (GGUS:138063)
    • SARA,IN2P3 FTS3 data transfer problem SARA <=> IN2P3 (GGUS:137967) (GGUS:137972)
    • RAL: FTS issues (server removed from Configuration) (GGUS:137822)
    • IN2P3: Decreased transfer efficiency (GGUS:137918)

Sites / Services round table:

• ASGC: Having a downtime for DPM upgrade to 1.10. It's scheduled on 19th of Nov, from 1AM to 10AM in UTC time. (already put downtime in GOCDB.)
• BNL: nc
• CNAF: Warning Downtime declared for tomorrow, extraordinary maintenance in Disk storage affecting Atlas, Alice and CMS. Should be transparent.
• EGI: nc
• FNAL: NTR
• IN2P3: IN2P3-CC will be in maintenance on December 4th, a Tuesday. As usual details will be available one week before the event. During this maintenance the last SL6 WN will be upgrade to CentOS7.
• JINR: NTR
• KISTI: NTR
• KIT:
  • Announced a downtime for updating dCache for LHCb on Tuesday, 20th of November: GOC-DB entry.
• NDGF: Some problems with the rucio today, looks like something wrong in the cern end. HPC2N has lost power at 14:00. Some power restored at 14:45 but something wrong still with the cluster power.
• NL-T1: NTR
• NRC-KI: nc
• OSG: nc
• PIC: NTR
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
  • OTG:0046717 : ~1 hour VOMS service downtime on Wednesday 14th November for DB intervention.
• CERN storage services: From Sunday Nov, 11th at ~13h30 to this morning at 10h, LHC EOS instances have intermittent problems talking to storage nodes in the Wigner computer center. A fraction of data stored in Wigner and new files directed there might report errors or timeouts. OTG:0046849
• CERN databases: Wigner standby DBs affected by the network issues in Wigner
• GGUS: NTR
• Monitoring:
  • Draft reports for the October availability sent around
• MW Officer: nc
• Networks: nc
• Security: NTR

AOB: