Week of 180108

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Julia (WLCG), Maarten (ALICE, WLCG), Kate (DB, chair), Vincent (security), Andrea M (FTS, MW), Ivan (ATLAS), Paul (storage), Stefan (LHCb), Gavin (comp)
• remote: Di Qing (TRIUMF), Dave M (FNAL), Gareth (RAL), Jens L (NDGF), John (RAL), Ken (CMS), Marcelo (CNAF), Onno (NL-T1), Pepe (PIC), Sang Un (KISTI), Xavier (KIT), Xin (BNL), David B (IN2P3), Kyle (OSG)

Experiments round table:

• ATLAS reports ( raw view) -
  • Workflow Management
    • Relatively smooth operation
    • 20-25th December: evgen very small job with a lot of outputs, created serious troubles to FTS.
    • Currently - not many score / evgen. Under investigation
    • Reprocessing finished very smoothly.
  • Data Transfers
    • Relatively smooth operation
  • Meltdown and Spectre
    • Very quick tests with ATLAS simul/reco, degradation for reco is about 5% (Desktop, no VM, using singularity, Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz4 core/8 HT, 8-core job). Under further investigation.

Vincent asked if the degradation was also measured on VMs. Ivan confirmed. It is to be checked if both the VMs and hypervisors were patched. Maarten remarked all workloads would need to be looked at after all components are patched.

• CMS reports ( raw view) -
  • Happy new year everyone! I guess things went about as well as might be expected over break. The clearest metric that I have is that we have very nearly completed our end-of-year re-reconstruction of the 2017 data.
  • This was helped by incorporating HLT resources, 31K CPUs.
  • Sites stayed up as well as we might have hoped. Six T2 sites made their way into the CMS "waiting room" over the break.
  • Grr, the e-log went down on Friday night, GGUS:132732, ticket didn't get addressed over the weekend.

A discussion on e-log responsibility took place, it is to be checked by CMS if it is it's internal responsibility as it runs on a CMS VOBox. Maarten reminded that TEAM tickets (and of course ALARM tickets) typically will be treated with higher priority than ordinary tickets.

• ALICE -
  • Best wishes for 2018!
  • Very high activity on average in the last 3 weeks: 150k concurrent jobs
    • Thanks to the sites!
    • Records on Dec 25: 175k on the grid, 100k at CERN
    • Mostly MC and significant amounts of reco and analysis trains
  • Central services: annual power cut tests on Sat Jan 6 had some fallout
    • A very large number of jobs failed during ~3 hours
    • Normal operations were restored by noon

• LHCb reports ( raw view) -
  • Activity
    • Running at maximum possible amount of resources, including fully available HLT farm during YETS
    • 2016 data restripping running full steam. Approx 1/2 of data processed (without CNAF) during YETS
    • Monte Carlo productions using remaining resources.
  • Meltdown & Spectre, performance hit after fix expected to be less critical for data processing and monte carlo jobs (accounting for vast majority of work carried out).
    • Need to patch voboxes, waiting for instructions from CERN
  • Site Issues
    • CERN/T0
      • ALARM ticket (GGUS:132628) for EOS transfer problems fixed internally by LHCb
    • T1
      • RRCKI problems with FTS transfers currently under investigation

Xavier asked about LHCb SAM test unavailability at KIT and elsewhere on 23rd of Dec. Stefan explained it was due to a VOMS AUP re-signing issue that got sorted later that day.

Sites / Services round table:

• ASGC: nc
• BNL: running smoothly in the last 3 weeks. To help ATLAS with reprocessing campaign, extra CPU slots (~15kHS06) were added to ATLAS farm over the break.
• CNAF:
  • Energy is almost fully recovered
  • Waiting batteries for UPSs to be delivered this week
  • During this month services will be tested and brought up
  • The tapes were collected during the last week of December. We expect news from the lab during this week.
  • We need to extend the Downtime by 2 weeks (Till 1st February)
  • Happy New Year from CNAF, and thank you all for the great support and patience!
• EGI: nc
• FNAL:
  • Calm during shutdown.
  • FTS upgrade will take place on the 10th
• IN2P3: ntr
• JINR: nc
• KISTI: ntr
• KIT:
  • Issues for ATLAS last weekend with many jobs failing xrootd data stage-in (GGUS:132728). Likely this was caused by one ATLAS user, which could not be identified, because before today the xrootd transfer protocol had been unauthenticated for ATLAS at GridKa.
  • One tape drive was malfunctioning during the Christmas period and was ultimately set offline on Jan 2nd. Thanks to that tape drive, about 140 tape cartridges were set unavailable, hence their data could not be read for some time.
  • The host certificate for the CMS VO box was lost due to our automated workflow producing an illegal certificate file. That was fixed on Friday, Jan 5th, and we will improve our workflow to avoid such issues in the future.
• NDGF: Meltdown/spectre fixing is ongoing
• NL-T1:
  • Sara-matrix down tomorrow to patch against Meltdown and Spectre. https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=24625
• NRC-KI: nc
• OSG: ntr
• PIC: ntr
• RAL: An Atlas disk server at RAL failed on Friday. It had multiple disk failures. We decided this morning that it was not possible to recover it so we have to declare the files on it as lost.
• TRIUMF: Two interruptions in last three weeks
  • From December 27 15:30 (UTC time) to December 29 4:30 (UTC time) a long scheduled downtime due to the 5 year service on the site 12.47 kV Electrical Distribution System, the service is done only every 5 years
  • On December 30 from 16:25 to 19:10 (UTC time) the routing table of the regional exchange router getting overwhelmed with full internet routing from one of peers, this caused all the WAN connections of our site lost about two and half hours.

• CERN computing services:
  • Determining impact of Meltdown/Spectre wrt compute, services and hypervisors. Performance impact of the fix is being studied for various payloads. Fix will be applied, as appropriate.
• CERN storage services:
  • Update of EOSATLAS to Citrine tomorrow
  • Castor Alice under investigation
  • FTS:
    • from 23/12 to 29/12 the number of scheduled transfers on the FTS cluster has been low due to FTS Scheduler DB queries taking longer than usual. The cause of this DB slowdown has been identified ( issues on backup process and ATLAS monitoring view when the Service is under heavy load) and we will improve the components in the next release. The issue has been fixed on the 29/12 by cancelling part of the ATLAS transfers and disabling the Web Monitoring and the following days the Service has ran quite smoothly ( 2.5 M transfers scheduled per day on the 30/12 and 31/12 new record!) so it managed to chew the transfers queue steadily.
• CERN databases:
  • Several incidents related to database storage issues. (DB group is following up with the storage vendor). Many databases were affected during incidents, the list can be checked in the following SSB entries:
    • Thu, Jan 04, 2018 01:30 to Thu, Jan 04, 2018 01:35 - OTG0041657
    • Sat, Dec 30, 2017 20:00 to Sat, Dec 30, 2017 20:35 - OTG0041630
    • Tue, Dec 26, 2017 08:32 to Tue, Dec 26, 2017 09:00 - OTG0041621
    • Thu, Dec 21, 2017 21:45 to Thu, Dec 21, 2017 22:30 - OTG0041595
    • Sun, Dec 17, 2017 05:15 to Sun, Dec 17, 2017 05:45 - OTG0041497
• GGUS:
  • Downtime Thu Jan 11 07:00-08:00 UTC for installing kernel patches
• Monitoring: NC
• MW Officer: NTR
• Networks: Perfsonar dashboards are not showing the correct information - developers are aware.
• Security: New vulnerabilities (CVE-2017-5753/CVE-2017-5715/CVE-2017-5754, alias "Meltdown" and "Spectre"): https://access.redhat.com/security/vulnerabilities/speculativeexecution
  • Packages (linux + linux-firmware + microcode_ctl & qemu-kvm for hypervisors) released by RedHat & Centos
  • Impact on performances still unknown (rumors of it being lower than initial reports)

AOB: