Week of 080324

Open Actions from last week:

Daily CCRC'08 Call details

To join the call, at 15.00 CET Tuesday to Friday inclusive (usually in CERN bat 28-R-006), do one of the following:

1. Dial +41227676000 (Main) and enter access code 0119168, or
2. To have the system call you, click here

Monday:

[ No meeting this week - Easter Monday! ]

Tuesday:

elog review: Only item a change of CMS FTS manager at FNAL.

Experiment report(s):

ALICE activities this week Mon 24/03 - Fri 28/03:

1. 1. Pass 1 reconstruction of RAW data taken during February/March commissioning exercise (CERN T0)
   2. Replication of specific RAW datasets to named T2s for fast detector-specific analysis (calibration/software tuning)
   3. MC production ongoing all centres

CMS want to verify they can access ggus tickets during long holidays such as Easter.

LHCb are testing their stripping workflow.

Core services (CERN) report:

DB services (CERN) report: Testing Oracle 10.2.04 and have found some issues with OEM agent monitoring which they have taken up with Oracle.

Monitoring / dashboard report:

Release update:

Questions/comments from sites/experiments:

AOB:

Wednesday

elog review: Nothing new.

Experiment report(s):

ATLAS (SC): This week is for throughput testing using junk data. The plan is to build up T0 to T1 transfers at 150% of what is required starting with today and adding 50% each day then running at 150% over the weekend. CNAF has a downtime so their share is going to BNL. NDGF have requested a reduced rate due to insufficient resources. SRMv2 is being used at CERN.

CMS (DB): Will continue their T0 to T1 exports to overlap with ATLAS but there is a downtime coming up at FZK.

Core services (CERN) report: The fixes deployed at CERN to the LCG CE that reduce their load by an order of magnitude are being packaged for external distribution and should be ready next week. Patches to the 1.6.7 version of lcgutils should also be ready for distribution next week.

DB services (CERN) report:

Monitoring / dashboard report: Some basic elog analysis tools (breakdowns per experiment, site, urgency etc.) have been prepared. The idea is to run them weekly and distribute the results as a regular report.

Release update:

Questions/comments from sites/experiments:

AOB:

Thursday

elog review: Nothing new.

Experiment report(s):

ATLAS: were running their throughput tests at 50% yesterday. There were some Castor fixes at the end of the day and today will try to reach 100% of the rate today then 200% on Friday then throttle back to 100% for the weekend. A current problem is they no longer see BNL on the ATLAS dashboard.

CMS: asking how to get actions from IT on non-urgent issues during long holiday periods such as Easter. The use case there was to allow some dataops team members to access the DAQ worker nodes to look at LSF issues. These are non-urgent requests but which cannot wait for weeks. It was agreed to discuss this at the next F2F meeting.

LHCb: are evaluating the requirements for local worker node disk space in case they switch to copying production input data to local WN disk rather than reading via remote I/O. Current estimate is 5 GB. They will be running some production to exercise the manpower of their stripping workflow.

Core services (CERN) report: Some Apache patches are in the pipeline for FTS servers.

DB services (CERN) report:

Monitoring / dashboard report:

Release update:

Questions/comments from sites/experiments: ATLAS are planning a bulk CERN site change to about 60K LFC entries. The consensus was that this is a relative simple operation that should only take a few tens of minutes of real time.

AOB:

Friday

elog review: nothing new.

Experiment report(s):

ATLAS (SC): 1) Yesterday the export rate was supposed to be increased to 100% of nominal. Unfortunately there has been a setup problem in the ATLAS T0 machinery and not enought LSF jobs were submitted. Therefore the rate has been moderate for the all day of yesterday. The problem has been fixed this morning and currently the machinery is ramping up to 200%. At the moment the thoroughput is about 1.1GB/s and increasing.

2) There is a problem of data display with the dashboard. Some entries which should end up in the production dashboard, actually end up in the T0 dashboard and vice versa. The problem has been identified (http virtual servers) and the solution is to deploy a separate frontend. This has been done, but some action is needed in terms of port opening and scanning from the CERN security team (request made yesterday night by Benjamin Gaidoz). GridView shows correct information as far as I can tell.

3) SARA is not getting data on disk since disk is full (being cleaned up now). NDGF does not get data on disk since disk is full (can not be cleaned up centrally since NDGF does not use LFC and this is the only supported catalog for the central deletion tools). NDGF people should do the cleaning (they have been alerted). CNAF is in scheduled downtime. No monitoring events are being collected from BNL (in dashboard and in GridView). Need to follow up with BNL.

Core services (CERN) report: A bug has been found in the CERN SRMv2 databases. This is in fact fixed in a rollout due on Monday so the rollout has been brought forwards.

DB services (CERN) report: Two major activities. 1) Preparation of new RAC 5 and 6 hardware (both now on critical power) to migrate experiment services next week. 2) Upgrade of integration RACs to Oracle 10.2.0.4. This version will probably not be used for CCRC'08,

Monitoring / dashboard report:

Release update:

Questions/comments from sites/experiments:

AOB: Next Tue/Wed are F2F and GDB days so the next regular daily meeting will be on Thursday 3 April.