Week of 201012

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: David (IN2P3), Jens (NDGF), Brian (RAL), Andrew (NIKHEF), Maarten (ALICE), Pavlo (ATLAS), Julia (WLCG), Concezio (LHCb), Ankur (computing), Pepe (PIC), Dave (FNAL), Ben (compute), Elena (CNAF), Xavier (KIT)

Experiments round table:

• ATLAS reports ( raw view) - NTR

• CMS reports ( raw view) - NC

• ALICE -
  • CERN migration from CASTOR to CTA: many thanks to IT-ST !
  • Lower activity today because of intervention on the task queue DB
  • F@H stopped on the grid since Monday evening last week

• LHCb reports ( raw view) -
  • smooth running without any major problems
  • migration from CASTOR to CTA: some EOS-->CTA tests will be performed this week

Sites / Services round table:

• ASGC: NC
• BNL: NC
• CNAF: Scheduled Downtime for storm-cms @ CNAF from 12/10 at 20:00 to 13/10 at 14:00 CEST to upgrade storm-cms (SO, GPSF and StoRM backend end frontend) + HSM. GOCDB link
  • Question from Julia about target version .There seem to be problems with 1.11.18 version. Maybe it's possible to wait for new version? Elena to check.
• EGI: NC
• FNAL: NTR
• IN2P3: NTR
• JINR: NC
• KISTI: NC
• KIT:
  • Last week's downtime was successful in that we've accomplished all goals, but not everything went smoothly...
    • All our network infrastructure devices are now IPv6 enabled.
    • GPFS and dCache have been updated.
    • All ARC-CEs are decommissioned now.
    • The downtime was extended for ATLAS (GOCDB:29546), because their dCache pools refused to start up properly. We needed support from dCache.org to find the reason and by noon on Oct 8th the SE was online again.
    • Only after LHCb had reported it (GGUS:148964) did we realize, that the new IPv6 addresses included in the SRM DNS round-robins, e.g. lhcbsrm-kit.gridka.de, were missing their respective PTR records at first.
    • ATLAS' as well as CMS' job submission framework did not pick up on the end of the downtime initially and we did not receive jobs right away.
  • Reported 97 lost files for tape to CMS (GGUS:148998)

• NDGF: Upgrade of dCache to 6.2.6 last week was successful but not without problems. An authentication issue for Alice Xrootd was detected just after the upgrade (this was not found in the pre production system). The dCache team quickly solved this and we could roll out a fix on all headnodes and pools within the 3 hour downtime window.
• NL-T1: NTR
• NRC-KI: NC
• OSG: NC
• PIC: NTR
• RAL:
  • Announcing decommissioning timeline for myproxy service: (GGUS:148975)
  • CMS jobs returning after CMS AAA failures last week. Investigating at site
• TRIUMF: NTR

• CERN computing services:
  • Another week another CentOS7 upgrade for CEs: OTG0059642 ce509 & ce510. Also in gocdb.
• CERN storage services: NC
• CERN databases: NTR
• GGUS: NTR
• Monitoring: NC
• MW Officer: NC
• Networks: NTR
• Security: NC

AOB:

• (Xavier/KIT): Is it intended that sites loose availability on monit-grafana, even though a downtime was properly scheduled through GOCDB?
  • To be checked when full data is available and inconsistencies reported via ticket to monitoring team.