Week of 180702

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Alberto (Monit), Borja (Chair, Monit), Enricho (Storage), Ivan (ATLAS), Maarten (ALICE), Miroslav P (DB), Olga (Computing), Renato (LHCB), Vincent (Security)
• remote: Alexander (NLT1), Christian (NGDF), David B (IN2P3), David M (FNAL), Marcelo (CNAF), Xavier M (KIT), Xin Zhao (BNL)

Experiments round table:

• ATLAS reports ( raw view) -
  • Software and Computing Week in Hamburg (25-29.06)
  • Production:
    • 295k / 308k job slots (GRID / All)
    • Problems with production at IN2P3-CC and TRIUMF due to removed default Panda Queues. Solved.
  • Transfers
    • DDM proxy permission problems. Solved.

• CMS reports ( raw view) -

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • Data reconstruction for 2018 data, MC simulation, user jobs

Sites / Services round table:

• ASGC: nc
• BNL: NTR
• CNAF: 2TB diskspace added for LHCB, tape downtime during the day.
• EGI: nc
• FNAL: Downtime July 20-21
• IN2P3: NTR
• JINR: NTR
• KISTI: nc
• KIT:
  • Updated HTCondor to recent version.
  • Introduced a queue for CMS single-core sam test jobs, by-passing the scheduler for regular jobs. This is not a high-priority queue and should be used in moderation just for the monitoring jobs.
  • wlcg-sam.cern.ch is unavailable for the last three weeks - any comments on that?

Maarten asked if the usage for the queue was already discussed with the CMS experts, Xavier confirmed it was.

• NDGF: UiO will have power maintenance on Wednesday and Thursday. CE and storage affected. Cluster is draining now, so reduced performance can be expected. Some ATLAS data unavailable.
• NL-T1:
  • Tuesday last week, we upgraded dCache from 2.6 to 3.2.
  • During the same downtime, we upgraded the Centos7 kernels, but we ran into intermittent communication errors on bonding interfaces. After downgrading the kernel on Friday, the problem seems to have disappeared. Unfortunately, some pools were still left disabled leaving some files temporarily unavailable, which we fixed this morning.
  • This afternoon we lost one of our dCache pool nodes due to an hardware problem. We expect this problem to be solved tomorrow.
• NRC-KI: nc
• OSG: nc
• PIC: nc
• RAL: NTR. Advance warning of power work at RAL in the week beginning 23rd July.
• TRIUMF: nc

• CERN computing services: NTR
• CERN storage services: NTR
• CERN databases: NTR
• GGUS: NTR
• Monitoring: Problems with the machine hosting WLCG sam reports, checking with the Cloud team.
• MW Officer: nc
• Networks: NTR
• Security: NTR

AOB:

• NOTE: Next week the CHEP 2018 Conference will be taking place in Sofia.
• The operations meeting next Mon will be virtual .
• You may provide relevant incidents, announcements etc. for the operations record.