Week of 150831

WLCG Operations Call details

• At CERN the meeting room is 513 R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Web
• Whenever a particular topic needs to be discussed at the daily meeting requiring information from site or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Monday

Attendance:

• local: Maria Alandes (chair&minutes), Maarten Litmaath (ALICE), Fernando Meireles (Grid&Batch)
• remote: Andrzej Olszewski (ATLAS), Onno Zweers (NL-T1), Sang Un Ahn (KISTI), Michael Ernst (BNL), Salvatore Tupputi (ATLAS), Dimitri (KIT), Rolf Rumler (IN2P3), Antonio Falabella (CNAF), Christoph Wissing (CMS), Lisa Giacchetti (FNAL), Di Qing (TRIUMF), Pepe Flix (PIC), Kyle Gross (OSG)

Experiments round table:

• ATLAS reports (raw view) -
  • T0/Central services
    • frontier degradation in Kibana (ELOG:54542)

Salvatore adds that there will be a new filter for the critical profile for ATLAS WLCG SAM tests so that only production endpoints will be tested and taken into account for site availability metrics. This will be available from the SAM3 interface.

• CMS reports (raw view) -
  • Rather large submission of MC and DIGI-RECO last week
    • Sustained ~100k production jobs since Friday
    • ~120k jobs in total in the CMS Global Pool
  • Some (short) DBS3 outages
    • DBS3 backend nodes run out of swap space
    • Access patterns analysed by experts

• ALICE -
  • very high activity
    • up to 90k jobs on Sat
    • up to 11k5 jobs at RAL on Sun
  • CERN: CVMFS Stratum-0 interface machine down Fri afternoon, fixed late afternoon, thanks! (INC:0846886)
  • CERN: still issues with accessing CASTOR for reading or writing raw data files.
    • The CASTOR team applied mitigations during the weekend, thanks!
    • Looking forward to the real fix...

• LHCb reports (raw view) -
  • Data Processing:
    • Drop of MC jobs at T1 sites on the week-end, solved this morning and restarting now. DataReconstruction at T1s.

Sites / Services round table:

NA = Not Available NTR = Nothing to report

• ASGC: NA
• BNL: NTR
• CNAF: Due to a fire causing problems with one electrical supply line that happened last Thursday, the computing centre is running at lower capacity (around 30% less of the pledged capacity) Only one power line is working at the moment and for this reason not all the chillers are on. LHCb disk servers have also been affected and they are currently offline. Checks are ongoing. It is expected that all is back to normal in few days. For more details, please check GOCDB
• FNAL: NTR
• GridPP: NA
• IN2P3: NTR
• JINR: NA
• KISTI: NTR
• KIT: NTR
• NDGF: NA
• NL-T1: A DNS migration is scheduled for next weekend.
• NRC-KI: NA
• OSG: NTR
• PIC: NTR
• RAL: NA
• TRIUMF:
  • Site-wide power outage due to extreme wind/storm last Sunday. More details in GOCDB
  • Tier-1 core network switch replacement and upgrade on Wednesday 02.09 at 16h00 UTC. All services will be unavailable for 6h. More details in GOCDB
• CERN batch and grid services: NTR
• CERN storage services: NA
• Databases: NA
• GGUS:
  • Long standing ggus ticket opened from LHCb: GGUS-114940. Any progress?
• Grid Monitoring: NA
• MW Officer: NA

AOB: None

Thursday

Attendance:

• local: Maria Alandes (chair&minutes), Maarten Litmaath (ALICE), Stefan Roiser (LHCb), Oliver Gutsche (CMS), Andrea Manzi (MW Officer), David Cameron (ATLAS)
• remote: Andrzej Olszewski (ATLAS), Andrew Pickford (NL-T1), Michael Ernst (BNL), Salvatore Tupputi (CNAF), Thomas Hartmann (KIT), David Bouvet (IN2P3), Pepe Flix (PIC), Rob Quick and Chris (OSG), Jens (NDGF), Gareth Smith (RAL)
• apologies: Di Qing (TRIUMF), Fernando Meireles (Grid&Batch)

Experiments round table:

• ATLAS reports (raw view) -
  • Production using full capacity of resources
    • 200,000 slots, 50/50% on single/multi-core
    • analysis share requested to decrease to 25% at T2 sites, currently running within the share
  • T0/Central services
    • Temporary problems in Tier0 exports to CERN endpoints
    • Reminder for sites to update FTS servers by mid-September

• CMS reports (raw view) -
  • Exceptional performance of MC and DIGI-RECO
    • period of 120k running jobs over for now, workflows in tails
    • expect more workflows soon

• ALICE -
  • high activity

• LHCb reports (raw view) -
  • Data Processing:
    • Dominated by MC and user jobs
  • T0
    • identified another slow worker node (GGUS:116023)
    • dbod 9 Sep downtime - asking for move to cluster installation before
    • lbvobox downtime 11 Sep - is it possible to do live migration of a VM?
  • T1
    • RAL: ARC-CE submission working again
    • CNAF OUTAGE downtime declared Friday evening for LHCb storage

There is a discussion about the poor performance of the batch nodes, as CMS and ATLAS have experienced similar issues. Stefan confirms PES is following up and in general it is reminded that this sort of issues are followed up and discussed in the CERN-IT Job Efficiency task force in which all experiments participate and whose next meeting is this Fri Sep 4.

It is also suggested that LHCb should open a GGUS ticket to migrate the lbvobox to a new VM.

Stefan reminds T1s that storage downtimes should be declared in GOCDB explicitly because LHCb has an automatic mechanism to ban storage services based on downtime information.

Sites / Services round table:

• ASGC: NA
• BNL:
  • It is confirmed that the FTS server will be upgraded before mid September as requested by ATLAS.
  • 8 PB of disk storage are currently phased into production, mostly to replace obsolete disk storage hardware. The process involves movement of data on old hardware to the new storage systems, which generates high load on the old and the new systems, causing some file copy operations executed by the PanDA pilot to time out. BNL staff is rebalancing the load in such cases to minimize these effects. ATLAS mentions that they haven't noticed anything.
• CNAF: Still running around 10% under pledged capacity. The missing power supply will be restored within 5 or 6 weeks. Storage resources are connected with double power supply so they shouldn't been affected. Although LHCb storage is still unavailable as stated in GOCDB More updates of the status in upcoming meetings.
• FNAL: NA
• GridPP: NA
• IN2P3: NTR
• JINR: NA
• KISTI: NA
• KIT:
  • The number of job slots in the machine job features configuration were not correct, but this has been fixed.
  • dCache gridFTP doors had been updated fixing the recent vulnerability.
• NDGF: dCache update also done.
• NL-T1: NTR
• NRC-KI: NA
• OSG: NTR
• PIC: NTR
• RAL:
  • After an update to SLC6 of the disk servers, some routing information was wrong and there have been some transfers failures that should be fixed now.
  • There was a network reconfiguration today that took longer than expected.
  • Plans to upgrade FTS will be reported at a next meeting
• TRIUMF: Successfully replaced TRIUMF Tier1 core switch, however it took 5 more hours than planned, thus all Tier1 resources were not available about 11 hours in total.

• CERN batch and grid services:
  • Argus filer migration: intervention successful.
• CERN storage services: NA
• Databases: NA
• GGUS:
  • KIT network intervention from 05-Sep-15 06:00:00 to 16:00:00. The intervention should be transparent
• Grid Monitoring: NA
• MW Officer: NTR

AOB: