Week of 180319

WLCG Operations Call details

• At CERN the meeting room is 513-R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Portal
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local: Kate (DB, chair), Julia (WLCG), Maarten (WLCG, ALICE), Vincent (sec), Gavin (comp), Ivan (ATLAS), Marian (net), Andrea M (FTS), Paul (storage), Alberto (mon), Borja (mon)
• remote: Onno (NL-T1), Sang Un (KISTI), Darren (RAL), Xavier (KIT), Zoltan (LHCb), Christoph (CMS), Di (TRIUMF), Christian (NDGF), Marcelo (CNAF), Xin (BNL)

Experiments round table:

• ATLAS reports ( raw view) -
  • Production - 330k / 470k (peak at 790k)
    • Madgraph generates high IO - under investigation
    • HPC file metadata bug found & fixed
  • Transfers & Storage - Almost full disks everywhere, but we expect to free up space by obsoletion campaign.

• CMS reports ( raw view) -
  • Again good resource usage (for CMS scales)
    • ~160k cores for Production
    • ~45k cores for Analysis
  • Made a bunch of tickets for sites ask to move to a proper version of Squid
  • Submitted tickets for a few remaining sites to conclude the commissioning of multi-core resources
  • Filled up our "unmerged" EOS area
    • Let to failure of SAM tests GGUS:134109
    • Fixed by increasing quota, while cause is still being investigated
  • Due to transfer issues to/from CNAF full re-enabling of the sites got postponed to this week

• ALICE -
  • Lowish activity level on average
    • Last days looking much better

• LHCb reports ( raw view) -
  • Activity
    • HLT farm fully running
    • 2017 data re-stripping ongoing
    • Stripping 29 reprocessing is ongoing
  • Site Issues
    • CNAF: IPv6 (GGUS:134088) and STORM issues.
    • GRIDKA: Pilot can not connect to the CS ( GGUS:133879)

Sites / Services round table:

• ASGC: nc
• BNL: NTR
• CNAF:
  • CNAF will enable IPv6 on the all storage by the end of this month
  • First wet tapes re-read are back and they are ok, the recover of all wet tapes is planned by the end of April.
  • This week is planned a network intervention to add two new cables at CINECA farm. No downtime is needed: the duration will be of a minute, there should be no disruption on the service.
  • There are 2 tickets open for LHCb:
    • (GGUS:134088) IPv6 issue for Storm - the ports were opened during the weekend, the problem remain, is under investigation.
    • (GGUS:133974) Cannot connect to storm - Problematic tape, double proxy issue, may be related to IPV6 issue. Ongoing
  • For CMS there is one tkt:
    • (GGUS:133961) xrootd config at T1_IT_CNAF - Under investigation

• EGI: nc
• FNAL: nc
• IN2P3: NTR
• JINR: NTR
• KISTI: NTR
• KIT:
  • Uploaded SIR about 4300 lost files on tape back in Dec 2017.
  • As a side effect to network maintainance today, the storage systems for ATLAS and CMS lost their internal connectivity and the GPFS clusters fell apart. All dCache pools that were actively writing data experienced I/O errors and disabled themselves. After the clusters were reestablished, we restarted the pools. For ATLAS this procedure unfortunately takes several hours, due to the very large number of files per pool, and they're unable to write into their disk-only storage space until the pools have finished initialising. An unscheduled downtime was added to GOCDB while the problem lasted.
• NDGF: Pool updates at UiO and NSC. Short (<5min) interruptions during reboots.
• NL-T1:
  • Reminder: dCache downtime tomorrow: https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=25004
• NRC-KI: nc
• OSG: nc
• PIC: nc
• RAL: NTR
• TRIUMF: There were issues with the tape system at new data centre after vendor’s unsuccessful firmware update last Thursday. About 6PB tape capacity were not available about 3 days.

• CERN site: Network intervention to upgrade distribution routers OTG:0042813 - Thursday 22 March 06.00 - 08.00 CET. There will be 8 minutes losses of connectivity to multiple CERN services throughout this time window.
• CERN computing services:NTR
• CERN storage services:
  • FTS:
    • new Production instance dedicated to Atlas deployed at CERN. Migration is ongoing.
      • Christoph asked if load is the only reason for a separate FTS instance for ATLAS. Andrea confirmed it - ATLAS is responsible for 70% of the load.
    • Network connectivity loss (OTG:0042885). Thursday 22 March 06.00 - 08.00 CET for 8 minutes.
  • CASTOR:
    • A new kernel for CentOS 7.4 is now available, which includes a patch to fix a RAID5/6 filesystem issue (as well as the Meltdown/Spectre patches). A reboot campaign of all CASTOR disk servers is planned tomorrow (OTG:0043003).
• CERN databases: NTR
• GGUS: NTR
• Monitoring:
  • Final reports for the February availability sent around
  • Still there is a problem with LUCILLE availability (we are checking it)
• MW Officer: nc
• Networks: NTR
• Security: NTR

AOB:

• NOTE: Mon-Thu next week the Joint WLCG & HSF Workshop will be held in Naples.
• The operations meeting next Mon will be virtual .
• You may provide relevant incidents, announcements etc. for the operations record.
• ATTENTION: next meeting on Tuesday April 3 !
• Have a good Easter break !