Week of 160718

WLCG Operations Call details

• At CERN the meeting room is 513 R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Web
• Whenever a particular topic needs to be discussed at the daily meeting requiring information from site or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Tier-1 downtimes

Experiments may experience problems if two or more of their Tier-1 sites are inaccessible at the same time. Therefore Tier-1 sites should do their best to avoid scheduling a downtime classified as "outage" in a time slot overlapping with an "outage" downtime already declared by another Tier-1 site supporting the same VO(s). The following procedure is recommended:

1. A Tier-1 should check the downtimes calendar to see if another Tier-1 has already an "outage" downtime in the desired time slot.
2. If there is a conflict, another time slot should be chosen.
3. In case stronger constraints cannot allow to choose another time slot, the Tier-1 will point out the existence of the conflict to the SCOD mailing list and at the next WLCG operations call, to discuss it with the representatives of the experiments involved and the other Tier-1.

As an additional precaution, the SCOD will check the downtimes calendar for Tier-1 "outage" downtime conflicts at least once during his/her shift, for the current and the following two weeks; in case a conflict is found, it will be discussed at the next operations call, or offline if at least one relevant experiment or site contact is absent.

Links to Tier-1 downtimes

ALICE	ATLAS	CMS	LHCB
	BNL	FNAL

Monday

Attendance:

• local: Luca (SCOD+Storage), Julia A, Maarten (ALICE), Ivan (ATLAS), Maria A (WLCG+CMS), Vincent (Security), Andrei (DB)
• remote: Daniele (CMS), Eric (LHCb), David (FNAL), Di Qing (TRIUMF), Dmytro (NDGF), Rolf (IN2P3), Jose (PIC), Onno (NL-T1), Sang-Un (KISTI), Tiju (RAL), Vincenzo (EGI)

Experiments round table:

• ATLAS reports (raw view) -
  • Current Issues:
    • CERN-RAL network is saturated (Netstat)
      

      CERN to RAL transfers
      • The LHCOPN link is saturated at 1.5 GB/s
      • Mostly by CMS
    • EOS stageout problems (GGUS ALARM ticket)
  • Activities:
    • T0 Spillover processing & derivation production ongoing.
  • Solved Issues:
    • EOS
      • 11.07.2016 - Namespace restart / SRM stuck (GGUS ALARM)

• CMS reports (raw view) -
  • T0-outbound traffic: after special meeting and debug/reconfigurations by EOS/FTS/network experts, rates went up to unprecedented levels. Digestion of the backlog in progress. As a positive note, during last weekend the AOD/MINIAOD backlog to PhEDEx Disk nodes was completely reabsorbed, only a few TBs left to transfer.
    • Thanks for the professional and effective help to everyone involved - CMS suggests we keep stable weekly meeting (with EOS at least, and for the moment at least) to capitalize and keep progresses stable in these heavy data taking times
  • P5 - Tier-0 transfer issues on July 15, started at ~16:00, per-file transfer rate dropped causing plenty of timeouts. No local file system issues, as tests confirmed. Over last month, rate was excellent, up to ~ 4GB/s.
    • GGUS:122920 (ALARM, July 15th, 18:30 - decided for ALARM as it may have affected data taking)
    • quick response, EOS and network experts on it, debugged to be a Geant link to Wigner down, ATLAS also affected
    • reconfigured EOS to cope with it, rates back up to cruise values at ~21:40
    • CMS confirmed good rates, backlog accumulated at P5 has been digested through the night/morning after (confirmation at the Run meeting on Saturday)
    • On Monday July 18th, confirmation that the link came back during the weekend with no issues, systems reconfigured to its initial settings (work with balanced load on Meyrin and Wigner), all back OK
  • inability to contact ElasticSearch on July 15th
    • GGUS:122917 (TEAM, then raised to ALARM - decided for ALARM for the reason explained below)
    • impacting e.g. https://meter.cern.ch/public/_plugin/kibana/#/dashboard/temp/CMS::CMS (i.e. CSP blind to critical services)
    • followed up by IT monitoring in https://cern.service-now.com/service-portal/view-outage.do?n=OTG0031725
    • investigations confirmed that due to a problem in the backend ES instance, metric data were arriving with some delay in meter
    • root cause found to be that some data nodes had memory problems causing the ES process to crash.
    • Problem solved on Friday evening itself. No data lost. Case closed.
  • T1 ingesting plenty of RAW+RECO data, so far so good - apart from Fermilab that needed massive clean-up to be accelerated to gain storage space, and IN2P3 needing to keep an eye on FTS-level configuration, apart from this all busy but under control

• ALICE -
  • Very high activity on average
    • New record today: 110636 concurrent jobs!

• LHCb reports (raw view) -
  • Activity
    • Monte Carlo simulation, data reconstruction/stripping and user jobs on the Grid
  • Site Issues
    • T0: NTR
    • T1: NTR

Sites / Services round table:

• ASGC:
• BNL:
• CNAF:
• FNAL: NTR
• GridPP:
• IN2P3: NTR
• JINR:
• KISTI: In scheduled downtime, everything smooth
• KIT:
• NDGF: NTR
• NL-T1:
  • Downtime tomorrow: a core router will be moved to Surfsara's new datacenter. LHCOPN and LHCONE connections will be unavailable.
  • Atlas disk poolgroup is almost full. The cause may be slow deletion of files by the dCache Cleaner component. We've done tuning and the number of deletions seems to have gone up a lot; but we'll keep an eye to see whether more tuning is needed. Additionally we found an out of memory issue; we'll assign more memory to the Cleaner during tomorrow's downtime.
    • The tuning seems to be effective: space is becoming available.
    • The tuning done: setting concurrent tape deletions per pool from 1 or 0 to 10. This seems to speed up disk deletions too.
    • dCache spreads tape deletions over all poolgroups, regardless of the VO, as long as the tape backend is the same. This works well, but it implicates that tuning should be done over all tape pools and not only pools of the involved poolgroup.
  • A service incident report about the recent network outage has been posted on https://twiki.cern.ch/twiki/bin/view/LCG/WLCGServiceIncidents#Q2_2016.
• NRC-KI:
• OSG: NTR
• PIC: NTR
• RAL: NTR
• TRIUMF: NTR

• CERN computing services:
• CERN storage services:
  • One link to Wigner (GEANT) was down from Friday afternoon (~17:00) and back Sunday morning (~7:00) and the other link was saturated. To avoid charging even more the remaining link EOSATLAS and EOSCMS Wigner machines were set to readonly since this issue was impacting experiments stage-in. All capacity was set to rw on Monday morning (~9:30) when the link appeared stable again.
  • EOSATLAS namespace was deadlocked and then restarted around 11:40 this morning since was not accepting any new connections.

• CERN databases: NTR
• GGUS: From Maria D. (Luca please read this for me- Grazie) Release on 27/7 with important changes on the servers - details next Monday.
• Monitoring:
• MW Officer:
• Networks:
• Security: NTR

AOB:

• Please take note of the decommissioning of the old EGI downtime notification system:
  • EGI broadcast
  • Existing subscriptions will not be automatically imported into the new system