Week of 190401

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Alberto (Monitoring), Borja (Chair, Monitoring), Christoph (CMS), Ivan (ATLAS), Maarten (ALICE), Olga (Computing), Remi (Storage), Vincent (Security)
• remote: Andrew (Nikhef), Di (TRIUMF), Dave (FNAL), Dmytro (NDGF), Konrad (LHCb), Marcelo (CNAF), Sang (KISTI), Xin (BNL)

Experiments round table:

• ATLAS reports ( raw view) -
  • NTR
  • Computing - avg - 303k / 329k slots
  • Transfers - avg - 18 GB/s

• CMS reports ( raw view) -
  • Very good resource utilization ~250k cores
    • Usual mix 25% for Analysis, 75% for Production
  • CMS DBS with some instabilities over the last week
  • CMS EOS instance at CERN had some problems after maintenance

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity
    • User jobs and MC productions
  • Issues
    • CERN: several tickets open:
      • Corrupted file(s) on EOS - GGUS:140476
      • Pilots hold at ce514.cern.ch - GGUS:140503
    • PIC: All pilots failed. There was an error in the JobRouting definition in HTCondor-CE - solved ( GGUS:140482)

Sites / Services round table:

• ASGC: NC
• BNL: NTR
• CNAF: NTR
• EGI: NC
• FNAL: NTR
• IN2P3: LHCONE has been upgraded on March 15th. Previous transfer slowdown should be resolved now.

Ivan did ask what was updated, They have increased the bandwidth of the network from 2x10GB connections to 2x20GB

• JINR: NTR
• KISTI: NTR
• KIT: ATLAS has received the 2019 storage milestone last week. CMS and LHCb are in progress. Alice was given the resources much earlier (on request).
• NDGF: NTR
• NL-T1: A routing issue temporary prevented access to Nikhef storage nodes on March 26th, from about 5am to 11am. This was a knock on effect from a routing update at SURFnet. It's now understood and fixed.
• NRC-KI: NC
• OSG: NC
• PIC: NC
• RAL: NC
• TRIUMF: NTR

• CERN computing services: Batch to switch to default submit target of CC7, tomorrow 2nd of April (OTG:0047300).
• CERN storage services:
  • Reminder: CERN AFS external disconnection test for one week on 3 April 2019 from 9:00. More info here: OTG:0048585
  • EOSCMS migration to new QuarkDB Backend (OTG:0048009): took place this morning, went fine but the restart of the instance caused some problems that have finally been fixed by 13h30 (CEST)
• CERN databases:
• GGUS: NTR
• Monitoring: NTR
• MW Officer: NC
• Networks: NC
• Security: NTR

AOB: