WLCG Operations Coordination Minutes - March 20, 2014

Agenda

• https://indico.cern.ch/event/308833/

Attendance

• Andrea Sciaba (chair), Nicolo Magini (secretary)
• local: Maria Alandes, Simone Campana, Marcin Blaszczyk, Maria Dimou, Oliver Keeble, Stefan Roiser, Alesandro Di Girolamo, Felix Lee, Daniele Spiga, Maarten Litmaath, Pablo Saiz
• remote: Valery Mitsyn, Yury Lazin, Maite Barroso, Daniele Bonacorsi, Thomas Hartmann, Christoph Wissing, Massimo Sgaravatto, Antonio Perez, Alessandra Forti, Alessandro Cavalli, Frederique Chollet, Pepe Flix, Rob Quick, Michel Jouvin

News

• Simone Campana is the new ATLAS Distributed Computing coordinator and Andrea Sciabà and Maria Alandes are the new WLCG operations officers
• Alessandra announced that the last T2 migrated to SL6 (in this case, CentOS 6.x). The only remaining site is CERN, which has moved 62% of the resources to SL6
• Alessandra started a web page for batch system comparison as decided at the latest pre-GDB meeting
• GOCDB now supports custom tags that can be attached to sites and services. This can be extremely useful for operations (e.g. they can be used to expose the VOs supported by a service) but it might require some previous agreement on the names of tags to introduce
• It has been proposed to move the planning meeting from April 3 to April 17 to prevent a conflict with the HEP software collaboration meeting that will take place on April 3
• Sites are reminded that CVMFS 2.0 is reaching end-of-life at the end of March and they should upgrade their CVMFS clients as soon as possible, as it's a prerequisite for the upgrade of the Stratum0 repositories, planned for June
• Experiments are reminded that the plan is to migrate to the GFAL2/FTS3 clients and decommission the GFAL/FTS2 clients during the summer
• The April 1 deadline for deploying the latest version of perfSONAR and opening the firewall to the IPs needed for the central monitoring is very close and many sites are not OK: please take action!
• Proposed dates of future meetings:

• Alessandro asks if OIM also supports custom tags like GOCDB. Rob answers that OIM already has VO support tags, other usecases to be identified.

Middleware news and baseline versions

https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions

• perfSONAR baseline increased to 3.3.2 as requested by TF

Tier-1 Grid services

Storage deployment

• New T1 RRC-KI-T1 in Russia supporting ALICE, ATLAS and LHCb added to the table

FTS deployment

LFC deployment

Oracle deployment

• Note: only Oracle instances with a direct impact on offline computing activities of LHC experiments are tracked here
• Note: an explicit entry for specific instances is needed only during upgrades, listing affected services. Otherwise sites may list a single entry.

• Need confirmation from IN2P3 about Oracle upgrade in March
• LFC and DIRAC bookkeeping were successfully validated on Oracle12

Other site news

Wigner job efficiency state of affairs

• some significant differences in job efficiency (CPU/wall-clock) are observed when the Wigner and Meyrin computer centers are compared
• the effects differ per experiment and per workflow
  • analysis jobs typically are affected, up to ~20%
  • simulation typically is not
• the investigation is complex because of multiple factors:
  • network latency and performance between Wigner and Meyrin
    • in particular for accessing EOS at CERN (for the time being the LHC experiments data are only in Meyrin)
    • mitigations are being looked into
  • Wigner and Meyrin have different CPU type mixes (AMD vs. Intel)
  • Wigner nodes are all virtualized, while Meyrin also has physical nodes
  • Wigner nodes run only SLC6, Meyrin has SLC5 and SLC6
  • for each experiment both centers receive job mixes that may not be easy to disentangle
    • we cannot simply reserve a lot of capacity for dedicated tests, because the resources are needed for real work!
  • the investigations require a number of experts to be involved, and such people are generally quite busy already
• to be continued...

• capacity numbers determined on March 11:
  • Wigner: ‍ 7976 non-dedicated slots,   57613 HS06
  • Meyrin: 32236 non-dedicated slots, 288042 HS06
  • SLC5: 118897 HS06
  • SLC6: 226781 HS06

• Cristoph asks how to target intentionally Meyrin vs Wigner in job submission. Maarten answers that it is not easy, but possible tweaking the experiment frameworks. As it is not the submission mode intended by IT, the message is that there is no reason to make difference for real production, only for debugging.

Data management provider news

Experiments operations review and Plans

ALICE

• T1-T2 workshop, March 3-7, Tsukuba, Japan
  • admins please have a look at the relevant presentations, thanks!
• The productions and analysis for Quark Matter 2014 are progressing at steady pace and according to plan.
• CERN
  • Wigner job efficiencies
    • no change
    • next meeting Apr 4

ATLAS

• Rucio: migration from LFC to RFC is progressing without problems, as of today we have migrated one entire cloud plus few more sites. In the next weeks we will increase the speed of how many sites at once we migrate.
• JEDI for analysis: few users tested without problems. In the next weeks few fixes/improvements in the monitoring will be done and we can then go in full production.
• Activity as usual: MC production run 30% MCORE.
• slow transfers have been observed between various SRC and DST. 3 main different issues: UKI-SOUTHGRID-CAM-HEP -> BNL, NIKHEF->TRIUMF, NDGF->TAIWAN. For CAM-BNL transfers with FTS3 CERN were 0.4MB/s, while with BNL FTS2 20MB/s. But then we noticed that lcg-cp with >3 streams, srmcp, and also FTS2 RAL managed transfers have the same performances. We also noticed that a very similar problem was observed already in the past http://bourricot.cern.ch/dq2/ftsmon/multi_spacetoken_view/UKI-SOUTHGRID-CAM-HEP/BNL-OSG2/2013-07-01/2014-03-20/48/0/1/ .
• regarding the Geneva-Wigner measurements for ATLAS: the impact on production activity is not really visible, on analysis we observe 20% efficiency degradation. The overall total impact on the ATLAS resources is of the order of few % (having ATLAS approx 90-10 prod vs analysis share), but it's highly visible because it's affecting user jobs.
  • we are now in a situation in which we are able to study and measure the issues and their evolution with time
  • network experts have been included in the loop
  • mitigation measures have been taken by some experiment (e.g. for ATLAS activation of TTReeCache when possible for analysis activity is ongoing)
  • for the future we also believe that once the data will be spread between Wigner and Meyrin the geolocation of the client should mitigate even further the observed issues because the "closer" copy of the file will be read.

• Comment that the "slow" links seem to be only transcontinental

CMS

• FTS2 decommissioning at CERN by August 1st?
  • CMS will come to an answer next week (during C&O workshop)
• FTS3
  • Sites should switch their Phedex “Debug” agents to use RAL FTS3
• Brief production and processing overview
  • HeavyIon rereco pass lunched last week
    • Quite memory (3-4GB) and cpu (96h jobs) demanding
    • Also using HLT-Cloud
  • Phase II upgrade MC
  • Soon 13 TeV MC digi/reco
• CPU efficiency for jobs running in Wigner center
  • One reported case with rather bad CPU efficiency
    • Quite special work flow
    • Limited statistics
  • More general analysis ongoing
• CVMFS Migration
  • Finalizing to use CMSSW via CVMFS at CERN
    • Expect to switch by April 1st
• Multicore processing
  • Switching a large fraction of FNAL resources
  • Continue testing at other sites, primarily Tier-1s
• CMS Spring Computing & Offline week at CERN * March 24-28

• Maria Dimou reminds to inform CMS users about deployment of new features for CMS in GGUS on Wed 26th. The target for disabling the Savannah-to-GGUS bridge functionality in GGUS is one month later. Cristoph says that CMS Computing Operations workflows will be moved from Savannah to GGUS piecewise.

LHCb

• Spring '14 Incremental stripping campaign
  • 2/3 of the data have been processed, estimating another 2 weeks of processing
  • Several sites have finished staging and processing of data
• WMS decomissioning campaign for remaining few small sites resurrected
  • Some problems found with parsing of BDII information by Dirac
• FTS3 clients compilation in AA context currently on hold b/c of C++98/11 libraries (AA / middleware)

Ongoing Task Forces and Working Groups

Tracking tools evolution TF

• GGUS release on the 26th of March. The release will bring the possibility to notify multiple sites with one ticket, Shibboleth support, and implement several CMS specific requests. The service will be in downtime from 7:00 to 10:00 UTC

FTS3 Deployment TF

• Understanding slow transfers as reported by ATLAS. We have to study the behaviour of the optimizer in these "strange" situations.
• The FTS3 server was not using the correct user proxy to transfer their files if the user has used several proxies (each one with different role) for submitting FTS jobs to the server. Developers have implemented the fix and applied it in the FTS3 pilot server for testing.

gLExec deployment TF

• 79 tickets closed and verified, 16 still open
• Deployment tracking page

Machine/Job Features

• patch release for the mjf.py client made available for CERN batch nodes. Deployment by PES

Middleware readiness WG

• 3rd meeting of the WG held on March 18. Watch the Agenda link for the minutes which are being prepared.
• Next (4th) meeting will take place on Thursday May 15 at 10:30 CEST. Items in that agenda will be:
  • The different Readiness Verification approaches across experiment VOs.
  • The refinement of the WLCG MW Officer role.
  • Issues of exact versions run at the Volunteer sites.

Multicore deployment

• NTR

perfSONAR deployment TF

• We have added 3 new sites in the LHCONE backbone and a new site in Finland is "in-progress", waiting on a decision about which mesh to put them in.
• In terms of "quality" we have seen some definite improvements in the CA, FR, USATLAS, USCMS and UK clouds.
• We still have quite a few issues with getting tests to run and return the needed metrics. This is caused by firewalls, down services or sites not yet using the mesh-config. We have a target of April 1, 2014 to have all sites instrumented.
• Apologies to the sites which have been waiting for an answer in the tickets assigned to them

• Less than 10 sites have not yet installed perfSONAR; but many sites are still running old releases (especially Tier-1s)

SHA-2 Migration TF

• SHA-2 user certificates are being used by all 4 experiments
• campaign to get our future VOMS servers recognized across WLCG has been launched on Mon March 17
  • EGI broadcast
  • OSG operations team is involved

• SAM tests could be used to discover if new VOMS servers are recognized, though not optimal
• The action about VOMS servers is not SHA-2 specific, so it can be decoupled and moved to the action list. Since there are no remaining actions concerning SHA-2, it was agreed to review the TF status at the planning meeting on April 17th and possibly declare it concluded.

WMS decommissioning TF

• CERN WMS instances for experiments will start getting drained as of April 1
• SAM instances have their own timeline

• Andrea comments that the timeline for the SAM WMS decommissioning is the validation of the new SAM probes with direct CREAM and CondorG submission, which is on schedule for June

xrootd deployment TF

• NR

IPv6 validation and deployment TF

• ATLAS first goal to be able to run Panda Tasks on IPv6 WN.
  • Panda Client uses http (curl) so software should work.
  • Development Panda instances at CERN is being made dual stack.
  • Development Pilot Factory instance at CERN is being made dual stack
• Frontier/Squid test box has also been made dual stack.

HTTP proxy discovery TF

* NR

Action list

1. Investigate how to separate Disk and Tape services in GOCDB
   • proposal submitted via GGUS:93966
   • in progress - ticket updated, current solution to be validated.
     • Some of the T1 sites are adding SRM.nearline entries as desired.
     • Downtime declaration tests to be done.
     • Experiences to be reported in the ticket.
2. Agree with IT-PES on a clear timeline to migrate OPS and the LHC VOs from VOMRS to VOMS-Admin
   • in progress
     • voms-admin test cluster is available
     • experiment VO managers have given feedback
     • VOMS service managers are in contact with developers to get major issue(s) fixed
     • https://twiki.cern.ch/twiki/bin/view/GridServices/VomsAdminTestInstance

• About the GOCDB action:
  • Stefan is coordinating for LHCb Tier-1s, though DIRAC is not ready yet.
  • For ATLAS and CMS, the statement is still:
    • no problem if Tier-1s declare new "SRM.nearline" services, but not needed yet. Will be evaluated in the future.
    • Tier-1 "SRM" services should not be in "OUTAGE" if only tape is affected. Declaring "AT RISK" downtime is fine.

• About VOMS-Admin, Maarten comments that several fixes were applied reducing the discrepancies between HR DB and VOMS DB, but not yet fully fixed. No significant problems reported by VO Managers with GUI or API so far.

AOB

-- NicoloMagini - 18 Mar 2014