Week of 140630

WLCG Operations Call details

• At CERN the meeting room is 513 R-068.

• For remote participation we use the Alcatel system. At 15.00 CE(S)T on Monday and Thursday (by default) do one of the following:
  1. Dial +41227676000 (just 76000 from a CERN office) and enter access code 0119168, or
  2. To have the system call you, click here

• In case of problems with Alcatel, we will use Vidyo as backup. Instructions can be found here. The SCOD will email the WLCG operations list in case the Vidyo backup should be used.

General Information

• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Web

Monday

Attendance:

• local: Maria Dimou (chair, minutes), Zbigniew Baranowski (IT-DB), Felix Lee (ASGC), Maarten Litmaath (ALICE), Jan Iven (IT-DSS)
• remote: Michael Ernst (BNL), Oliver Gutsche (CMS), Tiju Idiculla (RAL), Lisa Giacchetti (FNAL), Pawel (KIT), Rolf Rumler (IN2P3-CC), Roger (NDGF), Onno Zweers (NL-T1), Jeremy Coles (GridPP), Rob Quick (OSG), Vladimir Romanovski (LHCb).

Experiments round table:

• ATLAS reports (raw view) -
  • NTR

• CMS reports (raw view) -
  • No major issues, processing and production is continuing at high scales

• ALICE -
  • NTR

• LHCb reports (raw view) -
  • Main activity: MC and User jobs
  • T0:
    • Brazilian proxy problem on EOS (GGUS:106434)
  • T1:
    • GridKa : CVMFS problem on WN (GGUS:106405) ongoing. Please close the ticket, if the issue is solved.
    • RAL : GGUS:105571 - inconsistent storage information publishing actually a sensor problem (mixup of tape and disk info)?

Sites / Services round table:

• ASGC: ntr
• BNL: ntr
• CNAF: not connected
• FNAL: ntr
• GridPP: ntr
• IN2P3: Reminder of the batch system outage tomorrow morning.
• JINR: not connected
• KISTI: not connected
• KIT: ntr
• NDGF: A network outage will start Wed at 22hrs UTC and will continue till 4am the next day. This shall affect access to ALICE data. There is also a short outage today of only 10'.
• NL-T1: ntr
• OSG: ntr
• PIC: not connected
• RAL: ntr
• RRC-KI: not connected
• TRIUMF: not connected.

• CERN batch and grid services:
• CERN storage services: A CASTOR CMS node was found mis-configured this past weekend and was removed from production.
• Databases: the patch installation schedule announced last week started today. Normally transparent to the users...
• GGUS:
• Grid Monitoring:
• MW Officer:

AOB: Reminder MW Readiness WG meeting this Wed July 2nd at 4pm CEST at CERN room 513-R-068 with audioconf. Agenda in http://indico.cern.ch/e/MW-Readiness_5

Thursday

Attendance:

• local: Maria Dimou (chair, minutes), Zbigniew Baranowski (IT-DB), Felix Lee (ASGC), Maarten Litmaath (ALICE), Andrea Manzi (MW Officer)
• remote: Michael Ernst (BNL), David Mason (CMS), Gareth Smith (RAL), Lisa Giacchetti (FNAL), Thomas Hartmann (KIT), Rolf Rumler (IN2P3-CC), Roger Oscarsson (NDGF), Dennis van Dok (NL-T1), Sang-Un Ahn (KISTI), Rob Quick (OSG)

Experiments round table:

• ATLAS reports (raw view) -
  • Central Services
  • T0/T1s
    • RRC-KI-T1 failures GGUS:106553 - now solved

• CMS reports (raw view) -
  • No major problems -- keeping resources mostly full with MC production and MC reprocessing
  • Midweek global run until end of week -- some of the first steps towards 2015 datataking
  • Beginning to send test workflows to Wigner resources
  • GGUS:106573 -- req. for site readiness correction by CNAF (actually was a CERN FTS problem) CMS is following up.

• ALICE -
  • NTR

• LHCb reports (raw view) -
  • No report.

Sites / Services round table:

• ASGC: ntr
• BNL: ntr
• CNAF: not connected
• FNAL: ntr
• GridPP: not connected
• IN2P3: There was a batch system outage last Tuesday. An unexpected error caused all queued jobs to be lost before the official downtime started. Now all is back to normal.
• JINR: not connected
• KISTI: The tape system xrootd was upgraded last week. An outage of the CERN-KISTI connection was observed temporarily and went away but the reasons are still being investigated.
• KIT: ntr
• NDGF: Today's network intervention was smooth. dCache pools are being upgraded now and seem to progress well.
• NL-T1: ntr
• OSG: ntr
• PIC: not connected
• RAL: There will be a CASTOR downtime next Tuesday morning till lunch time, affecting ATLAS and declared in GOCDB.
• RRC-KI: not connected
• TRIUMF: not connected

• CERN batch and grid services:
• CERN storage services: no report
• Databases: This week's planned patches are almost finished. There will be a big intervention next week Tuesday 9am-5pm CEST on the shared storage system in the BARN affecting 40 databases, including CMS, ATLAS, WLCG, LHCb, CASTOR that should be transparent but better consider at risk. Details HERE!!
• GGUS: no report
• Grid Monitoring: no report
• MW Officer: ntr

AOB:

• ATTENTION:
  • WLCG Collaboration Workshop in Barcelona July 7-9
  • next ops meeting on Thursday July 10 !

-- SimoneCampana - 20 Feb 2014