Week of 170123

WLCG Operations Call details

• At CERN the meeting room is 513 R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Web
• Whenever a particular topic needs to be discussed at the daily meeting requiring information from site or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Tier-1 downtimes

Experiments may experience problems if two or more of their Tier-1 sites are inaccessible at the same time. Therefore Tier-1 sites should do their best to avoid scheduling a downtime classified as "outage" in a time slot overlapping with an "outage" downtime already declared by another Tier-1 site supporting the same VO(s). The following procedure is recommended:

1. A Tier-1 should check the downtimes calendar to see if another Tier-1 has already an "outage" downtime in the desired time slot.
2. If there is a conflict, another time slot should be chosen.
3. In case stronger constraints cannot allow to choose another time slot, the Tier-1 will point out the existence of the conflict to the SCOD mailing list and at the next WLCG operations call, to discuss it with the representatives of the experiments involved and the other Tier-1.

As an additional precaution, the SCOD will check the downtimes calendar for Tier-1 "outage" downtime conflicts at least once during his/her shift, for the current and the following two weeks; in case a conflict is found, it will be discussed at the next operations call, or offline if at least one relevant experiment or site contact is absent.

Links to Tier-1 downtimes

ALICE	ATLAS	CMS	LHCB
	BNL	FNAL

Monday

Attendance:

• local: Petr (ATLAS), Gavin (IT-CM), Marcelo (LHCb), Xavier (SCOD), Andrea M. (MW), Herve (IT-ST), Kate (IT-DB), Maarten (ALICE), Alberto (Monitoring), Marian (Network), Vincent (Security)
• remote: John (RAL), Francesco (INFN), Di (TRIUMF), Chema (CMS), Onno (NL-T1), Victor (JINR), Xin (BNL), David B. (IN2P3), David M.(FNAL), Vincenzo (EGI), Pepe (PIC), Sang Un (KISTI)

Experiments round table:

• ATLAS reports ( raw view) -
  • production stable with ~ 250k cores used in average
    • on Friday monitoring was changed - transferring, holding and merging stage was treated as "running" in ATLAS Job Dashboard (reverted)
    • INFN T1 - jobs got stuck while accessing non-existing file from /afs/.cern.ch/sw/lcg/contrib/gcc/4.9.3/x86_64-slc6/lib64/libstdc++.so.6.0.20 (AFS removed from WN, not obsolete ATLAS release - will be fixed)
    • Hammercloud jobs failed because of missing input file (was part of removed dataset) - blacklisting production sites
  • storage - old installation of DPM configured to use only RC4 cipher (not compatible with CC7)
  • FTS server in BNL (GGUS:126082)
  • automatic mod_ssl upgrade breaks apache configured by puppet (add "missing" ssl.conf that breaks our configuration)
  • ATLAS Sites Jamboree (January 18-20).

• CMS reports ( raw view) -
  • no major issues to report
  • Very high and steady utilization of CPU resources (>200 kcores) both for production and analysis tasks in preparation for the major winter conference (Moriond 2017)

• ALICE -
  • A problematic MC production using way too much memory is being investigated
  • CERN: condorce01 down Thu evening (GGUS:126049)

• LHCb reports ( raw view) -
  • Activity
    • Mainly running simulation on grid only resources, HLT back and running ~10K jobs. ~67K jobs total
  • Site Issues
    • T0:
      • EOSLHCB "very slow" via SRM last week. Back to normal (GGUS:126037)
    • T1:
      • RAL: Storage in Downtime today from 10:30 to 12:30
      • CNAF: announced a Downtime on 13th and 14th February for changing of core switch

Sites / Services round table:

• ASGC: -
• BNL: FTS server issue (GGUS:126082) caused by running out of inode limit, fixed. Next Monday (01/30), T1 network will undergo migration, which should mostly be transparent to users/VO jobs.
• CNAF: We planned a network intervention on 13-14/2 (no overlap with other Tier1 found in the calendar). We are going to insert a down in the gocdb. The down will start accordingly to the preference expressed by LHC experiment: 1) to schedule a down starting on Friday 10/2 in order to put in draining all the queues; 2) to schedule a down starting on Monday 13/2, queues will be not drained and remaining jobs will be killed just before of the intervention. Input from experiments are needed to take a decision.
• EGI: ntr
• FNAL: ntr
• GridPP: -
• IN2P3: ntr
• JINR: All OK, ntr
• KISTI: ntr
• KIT: -
• NDGF:
• NL-T1: ntr
• NRC-KI: -
• OSG: BDII server down last week. Monitoring alert received. Service was restarted.
• PIC: ntr
• RAL: Series of downtimes are planned / announced to upgrade Castor to 2.1.15 and to patch for CVE 7117. Last week's LHCb Castor upgrade had some problems which be believe were due to Castor tuning issues.
• TRIUMF: ntr

• CERN computing services: nothing to add
• CERN storage services:
  • Upgrade EOS next week (in preparation of 2nd part of Run II), 2h downtime expected (See Release Notes )
    • EOSLHCB: 31st Jan 7:30 to 9:30 CERN Time
    • EOSCMS: 1st Feb 7:30 to 9:30 CERN Time
    • EOSALICE: 7th Feb 7:30 to 9:30 CERN Time
    • EOSATLAS: 8th Feb 7:30 to 9:30 CERN Time
• CERN databases: ntr
• GGUS:
  • New "release" Wed this week with alarm tests as usual
    • No changes to GGUS this time, but OS patches need to be installed
    • Downtime Jan 25 07:00-08:00 UTC
• Monitoring:
  • December report ready to be generated. The only issue found is solved: INC:1252018
• MW Officer:
  • A new version of dCache 2.13 has been released last week ( 2.13.51, https://www.dcache.org/downloads/1.9/release-notes-2.13.shtml). It fixes a problem in jglobus discovered in the context of the RFC Proxy TF, which prevented RFC proxies of certificates belonging to certain CAs to work with dCache 2.13 (from v 2.14 jglobus has been removed from dCache). The same issue is still present in Bestman though. (GGUS:124650)
• Networks: ntr
• Security: ntr

AOB:

• Reminder: Ops coordination meeting this week.