Week of 151116

WLCG Operations Call details

• At CERN the meeting room is 513 R-068.

• For remote participation we use the Vidyo system. Instructions can be found here.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• General information about the WLCG Service can be accessed from the Operations Web
• Whenever a particular topic needs to be discussed at the daily meeting requiring information from site or experiments, it is highly recommended to announce it by email to wlcg-operations@cernSPAMNOTNOSPAMPLEASE.ch to make sure that the relevant parties have the time to collect the required information or invite the right people at the meeting.

Links to Tier-1 downtimes

ALICE	ATLAS	CMS	LHCB
	BNL	FNAL

Monday

Attendance:

• local: Luca (SCOD+Storage), Maarten (ALICE), Fernando (Batch), Andrei (DB)
• remote: Dario (ATLAS), Giuseppe (CMS), Raja (LHCb), Michael (BNL), Francesco (CNAF), Rolf (IN2P3), Sang Un (KISTI), Pavel (KIT), Onno (NL-T1), Chris (OSG), Jose (PIC), John (RAL), Di Qing (TRIUMF)

Experiments round table:

• ATLAS reports (raw view) -
  • Grid production activities ongoing

• CMS reports (raw view) -
  • No major issues to report
  • Preparing for Heavy Ion run

• ALICE -
  • Our thanks go to a number of big sites for setting up high-memory queues for heavy ion data reconstruction!
  • KISTI, KIT and NLT1 are ready
  • CERN and CNAF are in progress

• LHCb reports (raw view) -
  • Data Processing:
    • Data processing of pp data at T0/T1/T2 sites.
    • Monte Carlo mostly at T0/T1/T2/T2D, user analysis at T0/1/2D sites
  • T0
    • NTR
  • T1
    • NTR

Sites / Services round table:

• ASGC:
• BNL: NTR
• CNAF:
• FNAL:
• GridPP:
• IN2P3: Full day outage planned for December 8 (to be confirmed 1 week before)
• JINR:
• KISTI: NTR
• KIT: NTR
• NDGF:
• NL-T1: NTR
• NRC-KI:
• OSG: Found 3 problem for the WLCG accounting, next report will be fixed. Still 1 problem to be solved for University of Florida.
• PIC: Full day downtime on November 24 to perform dCache upgrade
• RAL: NTR
• TRIUMF: NTR

• CERN batch and grid services:
  • myproxy.cern.ch's intervention has been rescheduled (Thu 19), please read the announcement placed on the ITSSB
  • LSF was down on Saturday due to a configuration problem after the recent migration from LSF 7 to LSF 9
• CERN storage services:
• Databases:
• GGUS:
• Grid Monitoring:
• MW Officer:

AOB:

Thursday

Attendance:

• local: Luca M. (SCOD+Storage), Maarten (ALICE), Fernando (Batch), Luca C. (DB)
• remote: Christoph (CMS), Raja (LHCb), Michael (BNL), Rolf (IN2P3), Sang Un (KISTI), Thomas (KIT), Andrew (NL-T1), John (RAL), Kyle (OSG)

Experiments round table:

• ATLAS reports (raw view) -

• CMS reports (raw view) -
  • No major issues to report
  • Preparing for Heavy Ion run

• ALICE -
  • also CERN and CNAF are ready for heavy ion data reconstruction
  • our thanks go to all experts involved!

• LHCb reports (raw view) -
  • Data Processing:
    • Data processing of pp data at T0/T1/T2 sites.
    • Monte Carlo mostly at T0/T1/T2/T2D, user analysis at T0/1/2D sites
  • T0
    • BLAH error submitting to ce407 (GGUS:117694). Solved pretty fast - many thanks.
  • T1
    • CNAF : CVMFS error on various worker nodes (GGUS:117700)

Sites / Services round table:

• ASGC:
• BNL: NTR
• CNAF:
• FNAL:
• GridPP:
• IN2P3: NTR
• JINR:
• KISTI: NTR
• KIT: NTR
• NDGF:
• NL-T1: NTR
• NRC-KI:
• OSG: NTR
• PIC:
• RAL: NTR
• TRIUMF:

• CERN batch and grid services:
  • On Monday 23/11/2015 the remaining ARC CE (ce501.cern.ch) will disappear from the site bdii.
    • The resource BDII direct from the CE will still be running.
    • The remaining VO who does not yet submit to HTCondorCE is encouraged to change their config to pull direct.
  • A CREAM issue on ce407 last night required a restart of tomcat.
• CERN storage services: NTR
• Databases: NTR
• GGUS:
• Grid Monitoring:
  • Final availability reports for October sent, and available at the SAM3 UI
• MW Officer:

AOB:

• Vidyo connection problem for Kyle (OSG)