Week of 170327

WLCG Operations Call details

• At CERN the meeting room is 513 R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Web
• Whenever a particular topic needs to be discussed at the daily meeting requiring information from site or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Tier-1 downtimes

Experiments may experience problems if two or more of their Tier-1 sites are inaccessible at the same time. Therefore Tier-1 sites should do their best to avoid scheduling a downtime classified as "outage" in a time slot overlapping with an "outage" downtime already declared by another Tier-1 site supporting the same VO(s). The following procedure is recommended:

1. A Tier-1 should check the downtimes calendar to see if another Tier-1 has already an "outage" downtime in the desired time slot.
2. If there is a conflict, another time slot should be chosen.
3. In case stronger constraints cannot allow to choose another time slot, the Tier-1 will point out the existence of the conflict to the SCOD mailing list and at the next WLCG operations call, to discuss it with the representatives of the experiments involved and the other Tier-1.

As an additional precaution, the SCOD will check the downtimes calendar for Tier-1 "outage" downtime conflicts at least once during his/her shift, for the current and the following two weeks; in case a conflict is found, it will be discussed at the next operations call, or offline if at least one relevant experiment or site contact is absent.

Links to Tier-1 downtimes

ALICE	ATLAS	CMS	LHCB
	BNL	FNAL

Monday

Attendance:

• local: Julia A. (WLCG), Kate (chair, WLCG, DB), Stefano B. (CMS), Marcelo (LHCb), Marian (network), Hervé (storage), Gavin (computing), Vincent (security)
• remote: Dimitri (Kit), Dmytro K (NDGF), John K (RAL), Luca L (CNAF), Onno (NL-T1), Xin Zhao (BNL), Fa Hui (ASGC), Jose Enrique G.N. (Atlas), Victor Z (JINR), David M (FNAL), Vincenzo S (EGI)

Experiments round table:

• ATLAS reports ( raw view) -
  • Production running stably with 300k - 340k job slots.
  • Tier-0 Frontier load was traced to too many accesses by the reprocessing cosmics runs, a fix was applied.
  • Data reprocessing started this Sunday, now running (40k slots). The share has been increased.
  • FTS is still not handling yet the timing of the file transfer proportional to the file size, still 4k seconds. Fixed in FTS pilot mentioned just because sites saw problem.
  • AGIS Issue with downtime cancellation. There was a bug in the method to cancel downtime and manual online did not work. Fixed now. Different topic is the policy the "switcher" is applying, ADC will present to sites again in the next weeks to remind, and review it.

• CMS reports ( raw view) -
  • two relevant issues:
    • Oracle upgrade did not work as expected: Firewall had to be disabled, CNAF and Wisconsin could not be made to work for 2 days due to a subtle problem with jumbo network packets. INC:1323920
    • EOS unresponsive Friday due to GSI overloaded by HLT, full investigation still ongoing (?). GGUS:127297
  • CMS detector starting commissioning with cosmic rays
  • Very, very little load at the moment. Expect Production to ramp up this week, but not analysis.

Julia informed that network expert will be asked to attend WLCG coordination meeting next week. Kate has explained network configuration has changed as DB traffic to storage is using public network in the new HW configuration, no private connection to storage any more. The intervention to redeploy FW will be announced separately.

• ALICE -
  • High activity
  • Not much to report, at least until this morning
  • CERN: EOS namespace restart on Thu did not have a big impact

• LHCb reports ( raw view) -
  • Activity
    • MC Simulation, Stripping
    • Staging campaign for Stripping27, Stripping28 and Stripping24b should take 6 to 7 weeks with peaks of staging.
  • Site Issues
    • T0:
      • CERN-HIST-EOS and FAILOVER communication error (GGUS:127308) and (GGUS:127309). Seems solved.
    • T1:
      • RAL: disk server gdss780 is currently unavailable.
      • CNAF: Added an additional drive for staging.
      • PIC: LTO5 drive is supposed to be replaced today. could be slower than usual.
      • FZK: FTS transfers fail (GGUS:127301). Under investigation.

Marcelo asked if there is a possibility in the sites to improve staging (tape to disk) for LHCb workload - not stable, with small peaks. Julia has asked if it is configuration related. Julia will contact ATLAS about staging test they have done, to check if they can provide any input.

Sites / Services round table:

• ASGC: IPv6 has been enabled for storage elements. Transfer tests are planned.
• BNL: dCache upgrade rescheduled for this week (see downtime page for more details). A lot of T1 CPU cycles went idle over the weekend, because ATLAS started to drain jobs ahead of downtime. As mentioned in the ATLAS report, ATLAS will revisit the policy.
• CNAF:
  • First tests with IPv6 succeeded, CE plus UI. Still globus-url-copy is not working on IPv6.
• EGI: ntr
• FNAL: ntr
• IN2P3: nc
• JINR:
  • The problem of low load is still under processing.
    • Data transfer timeout from FNAL to JINR leads to rejection of jobs execution.
    • Stage-in to tape at FNAL is OK, but data is staying in disk buffer for only one hour,so request to transfer the data results in reading it from tape again which causes user-timeout,
• KISTI: nc
• KIT:
  • We consider to switch off the resource BDIIs for the dCache SEs. ATLAS, CMS and LHCb: Please inform us asap whether you depend on any of the information published through BDII on the storage elements.

Julia: ok to switch BDII off, experiments will be contacted nonetheless.

• NDGF: ntr
• NL-T1:
  • SURFsara now has a test dCache instance dual stack IPv6. Addresses have been added to http://hepix-ipv6.web.cern.ch/testbed-nodes. We've noticed that some storage clients don't work well with IPv6, so we hesitate to put this in production. Is there a test/implementation procedure? SCOD will check with experiment liaisons if they want to test first
• NRC-KI: nc
• OSG: Issue with ticket exchanged has been explained - 2 databases were out of sync. Now fixed. Friday SLA for BDII will expire, it will be switched off next week.
• PIC: nc
• RAL: Upgraded LHCb SRM to 2.1.16 last Thursday. Plan to do Atlas SRMs tomorrow and CMS and Alice on Thursday (30/03/2017).
• TRIUMF: NTR

• CERN computing services:
  • CERN VOMS service update (security) next Tuesday 4th April.
• CERN storage services:
  • Nothing to report
• CERN databases:
  • ATLARC has been migrated to new HW today
  • CMSR issue - SIR draft has been circulated in CMS. https://drive.google.com/file/d/0Bxca_eoT6OBaYU1hckhDdURYNjQ/view?usp=sharing Kate has assured it is possible to rollback after the HW migration.
  • CMSONR will be migrated to new HW next Tuesday
• GGUS:
  • Release + alarm tests Wed this week
• Monitoring:
  • NTR (and sorry for missing the meeting)
• MW Officer: ntr
• Networks:
  • Together with ATLAS, tested network performance of BNL/ASGC and managed to achieve 500MB/s sustained for several days.
• Security: ntr

AOB: