WLCG Tier1 Service Coordination Minutes - 10 February 2011

Attendance

LHC run in 2011 / 2012

Action list review

Release update

Data Management & Other Tier1 Service Issues

Site	Status	Recent changes	Planned changes
CERN	CASTOR 2.1.10 (all) SRM 2.9-4 (all) xrootd 2.1.9-7
ASGC	CASTOR 2.1.7-19 (stager, nameserver) CASTOR 2.1.8-14 (tapeserver) SRM 2.8-2	28/1: 30' of unscheduled downtime for CASTOR due to blade firmware upgrade, core servers had to be rebooted	Upgrade tests ongoing
BNL	dCache 1.9.5-23 (PNFS, Postgres 9)	none	in the process of adding disk space
CNAF	StoRM 1.5.6-3 (ATLAS, CMS, LHCb,ALICE)		certification terminated on Feb 09, upgrade OS to SL5 is delayed a little bit
FNAL	dCache 1.9.5-23 (PNFS) Scalla xrootd 2.9.1/1.4.2-4 Oracle Lustre 1.8.3	none	Moving unmerged pools from dCache to Lustre Deploying scalable SRM servers with DNS load balancing
IN2P3	dCache 1.9.5-24 (Chimera)	Upgraded to version 1.9.5-24 on 2011-02-08	4-day intervention starting Feb 20
KIT	dCache (admin nodes): 1.9.5-15 (Chimera), 1.9.5-24 (PNFS) dCache (pool nodes): 1.9.5-9 through 1.9.5-24	Updated part of the dCache setup to 1.9.5-24 during GridKa downtime on 26.01.2011
NDGF	dCache 1.9.11	None	None
NL-T1	dCache 1.9.5-23 (Chimera) (SARA), DPM 1.7.3 (NIKHEF)		On march 8th a core router will be replaced at NIKHEF. Services at NIKHEF will not be available and be at risk two days afterwards.
PIC	dCache 1.9.5-23 (PNFS)
RAL	CASTOR 2.1.9-6 (stagers) 2.1.9-1 (tape servers) SRM 2.8-6	CMS disk server upgraded to SL5 64 bit on 31/1/11	ALICE disk server upgrades to SL5 64bit on 15/2/11 Plans for CASTOR upgrade to 2.1.10 in March
TRIUMF	dCache 1.9.5-21 with Chimera namespace	None	None

CASTOR news

CERN operations

Development.

xrootd news

dCache news

• Jon: apparently the new Golden Release will be 1.9.12, but FNAL are happy with 1.9.5 and have serious concerns about such a significant upgrade during the 2011-2012 LHC run!
• Maarten: we will discuss this offline with the dCache developers and report in the next meeting

StoRM news

FTS news

• FTS 2.2.5 in certification: FTS 2.2.5 release notes
• FTS 2.2.6 in development: FTS 2.2.6 patch (see list of bugs at the end)

• FTS compatibility with Oracle 11g servers will be tested at CERN in a few weeks from now
  • problems may show up in the DB usage optimization
  • Matt Hodges: this was tested at RAL and the performance was bad indeed
    • e-mail after the meeting: Richard Sinclair is the DBA at RAL-LCG2 who was investigating the problem, and it was a deprecated parameter (commit_write) that was suspected of causing the performance problems that we were seeing.

DPM news

• DPM 1.8.0-1 for gLite 3.2 has been released to Production on Feb 9.
• For gLite 3.1 it remains in the Staged Rollout and the memory leak appears not to be fixed.

LFC news

• LFC 1.8.0-1 for gLite 3.2 has been released to Production on Feb 9.
• For gLite 3.1 it remains in the Staged Rollout and the memory leak appears not to be fixed.

LFC deployment

Experiment issues

WLCG Baseline Versions

• Release report: deployment status wiki page
• WLCG Baseline versions: table

Status of open GGUS tickets

GGUS - Service Now interface: update

Review of recent / open SIRs and other open service issues

Conditions data access and related services

• ATLAS weekly FroNTier meetings

Database services

---++ Database services

• 10.2.0.5 patching status - ALL databases are now running 10.2.0.5 - no major issues found, some minor issues included:
  • OEM agents not reading the 10.2.0.5 DB alert logs properly - bug 10170020
  • Oracle Bug 9184754 found with very specific to ATLAS PANDA workload

• Experiment reports:
  • ALICE:
    • Nothing to report
  • ATLAS:
    • Applied patch 9184754 on ADCR production DB - bug was only affecting PANDA application and was causing single instance crashes every few days
    • On Friday (4th of Feb) morning ATLAS PVSS replication aborted due to foreign key violation on target database (ATLAS offline) by five transactions. Harming transactions were applied on source database (ATLAS online) before without problems - violating the constraint – which should never happen. The problem of constraint inconsistency in PVSS schema is being investigated but the root cause is not known yet. In order to start replication and maintain the replica consistent problematic transactions have been applied without constraint validation.
  • CMS:
    • Nothing to report
  • LHCb:
    • Nothing to report

• Site reports:

AOB

-- JamieShiers - 03-Feb-2011