Week of 200831

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: Miro (Chair, DB), Borja (Monitoring), Xavier (Kit), Maarten (ALICE), Michal (ATLAS), Dave (FNAL), Christoph (CMS), Andrew (NIKHEF), Julia (WLCG), Andrew (TRIUMF), Xin Zhao (BNL), Christian (NDGF), Vladimir (LHCb), Jose (PIC), Elena (CNAF), Nikolay (JINR)

Experiments round table:

• ATLAS reports ( raw view) -
  • Activities:
    • Ongoing reprocessing
    • RAL MCTAPE deletion: done. 2 million files in 16 days.
  • Issues:
    • 160 lost files at IN2P3 MCTAPE
      • waiting to be declared (after bug in declaration procedure is fixed)
    • CVMFS Stratum 1 at RAL was not snapshotting repositories on Thursday
      • misconfigured httpd
    • Jobs at CA-TRIUMF-DISKLESS fail with “no such file or directory” (GGUS:148376)
      • update from the site: xcache seems working fine, however, event jobs still failing. Need more expert looking.

• CMS reports ( raw view) -
  • Rather good CPU utilization ~270kcores (215k production / 55k analysis), with a peak over 300k during the weekend
  • No major problems

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • MC and WG productions; User jobs.

Sites / Services round table:

• ASGC: NC
• BNL: NTR
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3: NC
• JINR:We are working on the transition from CREAM CE to ARC6
• KISTI: NTR
• KIT: NTR
• NDGF: Short outage last Tuesday was caused by a wrong HAProxy config being rolled out to production servers.
• NL-T1: NTR
• NRC-KI: NC
• OSG: NC
• PIC: NTR
• RAL: NC
• TRIUMF: NTR

• CERN computing services: NTR
• CERN storage services: NC
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
  • OTG:0058724: Network outage did not have direct impact on the infrastructure, but several side services (as ES or DBOD) were affected badly.
  • Final reports for July will be sent around during today
  • dashb-earth.cern.ch has been migrated into MONIT (should be transparent for the users)
  • Retirement of SAM3 dashboards will be postponed for a couple of weeks since we are still working with some users feedback, new temptative target date is 14th of September
  • New endpoint for ALICE vofeed wlcg_sitemon replacing the SAM3 one.
• MW Officer: NC
• Networks: NTR
• Security: NC

AOB: