Week of 200316

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• remote: Kate (WLCG, DB, chair), Julia (WLCG), Maarten (WLCG, ALICE), Onno (NL-T1), Elena (CNAF), Gavin (computing), Vincent (security), Darren (RAL), Dave M (FNAL), Christoph (CMS), David B (IN2P3), Andrew (TRIUMF), Cristi (storage), Pepe (PIC), Xin (BNL), Renato (LHCb)

Experiments round table:

• ATLAS reports ( raw view) - NTR

• CMS reports ( raw view) -
  • CPU utilization a bit lower than usual: ~180k for production, 60k for analysis
    • Have some processing constrains to run I/O intense work at sites that host input data (pileup for 'classical' event mixing in particular)
  • Otherwise no major issues

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • Stripping campaign ongoing, occupying most of T0/1 capacity (no T2s)
    • Staging for 2016 is almost finished. Staging for 2017 is ongoing.
  • Issues:
    • NTR

Sites / Services round table:

• ASGC: nc
• BNL: NTR
• CNAF: NTR
• EGI: nc
• FNAL: NTR
• IN2P3:
  • Following stage 3 declaration in France, IN2P3-CC has switched to best effort mode. It will strive to ensure the full continuity of service in the respect of the sanitary rules and following the instructions given by CNRS. As a result, the maintenance scheduled tomorrow is postponed. Downtime in GOCDB will be updated accordingly by the end of the day.
  • Issue:
    • ALICE opened a ticket for our ALICE voboxes (GGUS #146052). We have a problem with CERN CA CRL update for services with fetch-crl + local squid combinaison. A ticket has been opened to CERN's CA team : INC2356750 Are there other sites with the same problem?

Maarten commented that Alice observed the issues (CMS is also affected). CERN CA should check file serving. Vincent commented that it might be related to CFI servers migrated to the new system. Anyway this needs to be checked by CERN CA.

• JINR: NTR
• KISTI: NTR
• KIT: Announced a storage downtime for 1st of April, where we have to update the firmware of our storage systems. All storage will be offline, so all VOs will be affected!
• NDGF: Nothing to report
• NL-T1: NTR
• NRC-KI:
• OSG: nc
• PIC: State of alarm announced in Spain. PIC staff is working from home, all services are supported. HTConder and DCache were upgrade remotely.
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: NTR
• CERN storage services:
  • Brief EOSALICE downtime last Friday: OTG:0055417
  • EOSALICE software upgrade this Thursday: OTG:0055352
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
  • Final availability reports for February sent around
  • Investigating situation with ATLAS SSB metrics not being updated in the database (OTG:0055444)
• MW Officer:
• Networks: NTR
• Security: Security Alert send last Friday, please check your mails

AOB: