Week of 190812

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-scod@cernSPAMNOTNOSPAMPLEASE.ch to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Claire (ATLAS), Gavin (Compute), Miro (DB, Chair), Vincent (Security), Borja (Monitoring), Julia (WLCG)
• remote: Renato (LHCb), Mike (ASGC), Jens (NDGF), Xavier (KIT), Xin (BNL), Marcelo (CNAF), Dave (FNAL), Di (TRIUMF)

Experiments round table:

• ATLAS reports ( raw view) -
  • pilot2/singularity migration converging, remaining sites/queues handled one by one
  • ATLAS started its reprocessing at T1's. Monitoring closely fts load and possible impact on user subscription. Production is OK, ~ 400k job slots running.
  • Since CNAF was in downtime, we took the opportunity to retrieve the datasets that were located at CNAF from CERN using a Preproduction CTA instance instead of Castor. Thanks to Julien from the CTA team to have been very reactive in providing us access/support to this instance.

• CMS reports ( raw view) -
  • NTR

• ALICE -
  • NTR

• LHCb reports ( raw view) -
  • Activity:
    • MC, user jobs and staging.
    • In the coming weeks, we will perform a lot of tape recall.
  • Issues:
    • RAL:
      • GGUS:142350; still issues accessing files on ECHO. Under investigation.
    • CNAF:
      • Outage since mid last week. DT extended until Wed. approx.
    • PIC:
      • Problem with ONLINE.git access (ticket opened GGUS:142673 ). Response: heavy load, response "not optimal"
    • IN2P3
      • FTS failures IN2P3-RDST -> BUFFER (ticket opened GGUS:142670 ). Thousands of "[SE][StatusOfBringOnlineRequest][SRM_FAILURE] ". No response on the ticket.
    • NIKHEF
      • No jobs running ( GGUS:142680 ), coming back now (this morning). Ticket can be closed.

Sites / Services round table:

• ASGC:
  • GGUS:142548
    • The network ipv6 issues to FZK(Germany) and IN2P3(France) were solved
    • There are some files corrupted, and our storage managers declared the lost data already
  • GGUS:142645
    • Some of analysis job try to use aipanda078.cern.ch to retrieve the source code. However, the aipanda078 was decommissioned recently. New submitted job afterward will no longer have this issue.

• BNL: NTR
• CNAF:
  • Downtime extended till Wednesday 14th August:
    • Technicians working hard on bringing the electric system up and running
    • Found the breaking point on the safety system, should be replaced in the next days
    • We should be able to come back operational by tomorrow or Wednesday
    • Downtime at GOCDB Electric system outage
• EGI: NC
• FNAL: NTR
• IN2P3: NC
• JINR: NTR
• KISTI: NC
• KIT:
  • Reminder: Nearline storage outage from Sep 16th till Sep 27th for all VOs!
    • Outage for lhcbsrm-tape-kit.gridka.de
    • Warning for atlassrm-kit.gridka.de and cmssrm-kit.gridka.de
    • Alice has no SE services hosted by KIT in GOCDB
  • Reminder: Online storage outage on Sep 17th for all VOs (GOCDB:27575). We will try and keep the batch farm operative, so you still may send us jobs and work with remote storage at other sites.
• NDGF: NTR
• NL-T1: NC
• NRC-KI: NC
• OSG: NC
• PIC: NC. Apologies. This is the PIC report: understanding LHCb problems, solved by now (13/08). PIC will be in downtime on Tuesday September 3rd, from 09:00 to 14:00 (PIC and CERN local times). We will proceed to upgrade dCache, Enstore and HTCondor in our storage and computing services. Access to computing resources at PIC will be closed shortly before the beginning of the SD.
• RAL: NC
• TRIUMF: NTR

• CERN computing services: NTR
• CERN storage services: NTR
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
  • Draft availability report for July 2019 sent to WLCG office (not yet forwarded to the rest of people?)
• MW Officer: NTR
• Networks: NTR
• Security: NTR

AOB: