Week of 141215

WLCG Operations Call details

• At CERN the meeting room is 513 R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Web

Monday

Attendance:

• local: Alessandro (ATLAS), Lorena & Maciej (CERN DB), Maarten (ALICE), Maria D. (SCOD), Stefan (LHCb), Tsung-Hsun (ASGC)
• remote: Christoph (CMS), Michael (BNL), Lisa (FNAL), Rolf (IN2P3), Sang Un? (KISTI), Dimitri (KIT), Ulf (NDGF), Onno (NL_T1), Rob (OSG), Pepe (PIC), Gareth (RAL).

Experiments round table:

• ATLAS reports (raw view) -
  • CentralService & Tier0/Tier1s & general overview
    • FZK storage issue tracked and discussed in GGUS:110693
    • BNL FTS server in trouble, experts notified.
    • several ATLAS DDM and ProdSys2 issues are affecting the overall production, more details below

• CMS reports (raw view) -
  • Rather quiet atm (many CMS colleagues are in Miami for the Winter collaboration week)
  • CMS plans some production during Xmas / New Year

• ALICE - ntr

• LHCb reports (raw view) -
  • "Legacy Run1 Stripping" campaign running full steam and progressing well, prestaging of input data was restarted + MC and user jobs.
  • T0:
  • T1:
    • CNAF: hiccup with SRM on Friday, fixed promptly
    • RAL: WARNING DT on the week-end went bad, ask sites to declare OUTAGE in case a the site is affected -> automatic handling by Dirac

Sites / Services round table:

• ASGC: ntr
• BNL: The transition from USLHCNet to ESnet is complete for the BNL and FNAL T1s and was Very smooth. They now enjoy 40Gbps of dedicated bandwidth out of 340Gbps of the total available. The plan is to also move the LHCONE-ESnet instance to use the ESnet infrastructure for the transatlantic link in January 2015 (120Gbps capacity). Dante and Geant traffic exchange with ESnet will also be able to use this infrastructure then. The South American sites are not part of this change.
• CNAF: not connected
• FNAL: Downtime tomorrow 8am-4pm for the dCache upgrade (including bug fixes and performance improvement).
• GridPP: not connected
• IN2P3: ntr
• JINR: not connected
• KISTI: ntr
• KIT: ntr
• NDGF: A transparent 30' downtime will take place on Thu 18/12 at 12hrs noon UTC.
• NL-T1: ntr
• OSG: ntr
• PIC: Now in scheduled downtime for dCache upgrade until 5:30pm CET.
• RAL: The DNS problem of last Saturday took all day to resolve. Gareth will relay LHCb's observations about changing the Warning status in GOCDB to a Downtime one when an issue takes longer than expected.
• RRC-KI: not connected
• TRIUMF: not connected

• CERN batch and grid services: not present
• CERN storage services: not present
• Databases: ntr. ATLAS has a question on hadoop. It should be sent to the CERN/IT/DSS group.
• GGUS: not present
• Grid Monitoring: not present
• MW Officer: not present

AOB:

• Please note the most recent Did You Know? in case you face GGUS availability issues during the Year End period.
• Please fill http://cern.ch/wlcg-survey and spread the word to T2s, T3s you may work with. The deadline is THIS Friday 2014/12/19.

Thursday

Attendance:

• local: Alessandro (ATLAS), Lorena & Maciej (CERN DB), Maarten (ALICE), Maria D. (SCOD), Stefan (LHCb), Tsung-Hsun (ASGC), Jerome (CERN Grid services), Andrea M. (MW Officer).
• remote: Christoph (CMS), Michael (BNL), Lisa (FNAL), Rolf (IN2P3), Dea-Han (KISTI), Ulf (NDGF), Dennis (NL_T1), Rob (OSG), John Kelly (RAL), Salvatore (CNAF).

Experiments round table:

• ATLAS reports (raw view) -

Alessandro asked whether experiments should take any action to address the vulnerability announced by EGI. Maarten answered that "all WNs should upgraded for sure but NO action is needed by the experiments."

• • CentralService & Tier0/Tier1s & general overview
    • ntr. More info at WLCG Ops Coord today

• CMS reports (raw view)
  • Quite some work assigned to the Grid sites "Xmas production"

• ALICE -
  • expectations for the end-of-year break:
    • mostly steady MC production
    • some raw data reprocessing
      • data being staged, no tape activity foreseen during the break
    • low analysis activity

• LHCb reports (raw view) -
  • "Legacy Run1 Stripping" campaign running full steam and progressing well, prestaging of input data was restarted + MC and user jobs. Note, the stripping campaign will continue over the xmas break.
  • T0:
  • T1:
    • CNAF: some debugging ongoing on mostly SRM protocol access for different kinds of activities to the storage elements, no conclusion/solution yet. Maria suggested to open a GGUS ticket in case the issue takes several days to be resolved.

Sites / Services round table:

• ASGC: ntr
• BNL: busy fixing the vulnerability issue
• CNAF: Investigating the LHCb issue.
• FNAL: ntr
• GridPP: not connected
• IN2P3: busy fixing the vulnerability issue. Wish to see the documentation specific to InfiniBand.
• JINR: not connected
• KISTI: ntr
• KIT: not connected
• NDGF: busy fixing the vulnerability issue. Now updating WNs. Also the dCache pools. Downtime is published for Mon 6am UTC for the headnodes update and 10amUTC for the rest.
• NL-T1: busy fixing the vulnerability issue. Noticed the cvmfs mount lost and emphasises the need to drain suffiiciently before applying the update.
• OSG: busy fixing the vulnerability issue. Wish to see the documentation specific to InfiniBand.
• PIC: We are applying a fix in PIC and the farm is affected, as mentioned below, for all of the LHC VOs supported: this is ATLAS, CMS and LHCb. (sent by email)
• RAL: usy fixing the vulnerability issue. Downtime is published for Tue am.
• RRC-KI: not connected
• TRIUMF: not connected

• CERN batch and grid services: Draining the nodes now for a coming (transparent) CREAM CE update.
• CERN storage services: not present
• Databases: ntr
• GGUS: not present
• Grid Monitoring: not present
• MW Officer: As already broadcasted to sites, vulnerabilities have been discovered in FTS3 and gfal2. The latest available FTS version 3.2.30 and gfal2 2.7.8 are not vulnerable, so they have been set as baselines and sites are suggested to upgrade. Both version will be also available in UMD by the middle of January ( for FTS3 is the first time in UMD)

AOB: Many thanks from the SCODs' team for your presence and wishes for a happy, healthy, creative 2015! Meeting on Jan 5th!