JRA1.1 - EMI-1 Development and Test Plans

EMI-1 Component Disk Pool Manager (DPM)

The Disk Pool Manager (DPM) is a lightweight solution for disk storage management (ease of installation, configuration, low effort of maintenance), while providing all required functionality for a grid storage solution (support for multiple disk server nodes, different space types, multiple file replicas in disk pools, etc.)

• EMI File Access Solution
• Managed by data area
• Part of Product Team CERN Data Management [ Details ]
• Component in ETICS : TBD (Oliver URI)
• Supported Platforms : SL5 - 64 bit
• EMI 1 Release
  • DPM Version 1.8.1
  • Tracker Item : https://savannah.cern.ch/task/?18582
  • Most known bug fixes provided in EMI 1
    • Savannah #46961: correct handling of offset/count in srmLs
    • Savannah #61037: Improved performance in dpm-drain (parallel file replication) (see optional feature below)
    • Savannah #75064: retrial for root requests after db reconnection
• SA2 Testbed
  • 2011-02-22 : Production Version deployed [ More Details ]
  • New features not already deployed, still a lot of packaging and configuration work to be done before it
• Verification and Validation Plan
  • Available (EGEE-III times) : [ More Details ]

Overall Component Status

Mandatory Features for EMI 1 - Addressed Technical Objectives

• EMI technical objective Data Area 1 : All storage elements publishing initial GLUE 2.0 storage information and one storage client (library) is capable consuming that. (M12)
  • Addressed by developers : [In Testing and Certification]
• EMI technical objective Data Area 3 : All storage elements offering support for the http(s) protocol (M12)
  • Addressed by developers: [In Testing and Certification]
• EMI technical objective Data Area 4 : All storage elements offering at least a prototype-level support for the "file://" access protocol. (M12)
  • Addressed by developers : [In Packaing and Documentation]
  • Warning: Might not be part of EMI 1 release due to ongoing packaging and configuration work

Mandatory Features for EMI 1 (ready)

The following features are implemented and considered to be part of the EMI 1 release 'Kebnekaise'.

GLUE 2.0 : The Information Provider in DPM will publish data using the GLUE 2.0 and GLUE 1.3 schema in parallel. However, initially the content will be the same. Over time the information published with GLUE 2.0 will become richer.

• Status: Developed; now in testing and certification phase
  • 2011-02-18 : Official notification that implementation is finished
• Verification : Testing Strategy 1 - GLUE2 Information [ More Details ]
  • 2011-02-01 : Oliver: the unit, deployment and functionality tests are run every night during/after the build (currently only in the gLite environment but this will change). I will capture this in the "Verification and Validation Plan".
• Milestone 1: 12/2010 - Development in beta quality present [Achieved]
• Milestone 2: 01/2011 - Tests in place for continous internal JRA1 testings during 02/2011 (JRA1.7, JRA1.8) [Achieved]
  • 2011-02-18: Tests come from the infrastructure area
• Feature Deadline : 02/2011 [Achieved]
• Addressing EMI technical objective Data Area 1 : All storage elements publishing initial GLUE 2.0 storage information and one storage client (library) is capable consuming that. (M12)
• Development Task Tracker : [ Details ]

HTTPS : DPM provides http(s) access

• Status : Developed; now in testing and certification phase
  • 2011-02-22 : Official notification that implementation is finished
• Description : [ More Details ]
• Verification : Testing Strategy 2 - SEs Access Protocols [ More Details ]
• Milestone 1: 12/2010 - Development in beta quality present [Achieved]
• Milestone 2: 01/2011 - Tests in place for continous internal JRA1 testings during 02/2011 (JRA1.7, JRA1.8)
• Feature Deadline : 02/2011 [Achieved]
• Addressing EMI technical objective : Data Area 3 : All storage elements offering support for the http(s) protocol (M12)
• Development Task Tracker : [ Details ]

Mandatory Features for EMI 1 (packaging and documentation)

NFS 4.1 : DPM will provide a pre-production version for the NFS 4.1. (pNFS) server.

• Status: Experimental version developed
  • 2011-02-18 : Official notification that implementation is finished; plan to provide the prototype at the same time as EMI 1, although not necessarily as part of the release
• Verification : Testing Strategy 2 - SEs Access Protocols [ More Details ]
  • 2011-02-01 : Oliver: the unit, deployment and functionality tests are run every night during/after the build (currently only in the gLite environment but this will change). I will capture this in the "Verification and Validation Plan".
• Milestone 1: 12/2010 - Development in beta quality present
• Milestone 2: 01/2011 - Tests in place for continous internal JRA1 testings during 02/2011 (JRA1.7, JRA1.8)
• Feature Deadline : 02/2011 [Achieved]
• Addressing EMI technical objective Data Area 4 : All storage elements offering at least a prototype-level support for the "file://" access protocol. (M12)
• Development Task Tracker : [ Details ]

Optional Features for EMI 1 (ready)

Improved drain performance : DPM will speed up the draining of data pools in case of hardware maintenance or decommissioning.

• Status: Developed; now in testing and certification phase
  • 2011-02-18 : Official notification that implementation is finished
• Description: There is the use of a 3rd party drain mechanism (3rd party rfcp) which was not there before, and the introduction of multi-threaded drain to improve throughput further.
• More information: [ Details ]
• Verification:
  • 2011-02-22 : Oliver: We have tests, yes, although they are not run regularly (we only do performance tests 'on demand').
• Addressing EMI component improvements (no particular technical objective)

Feature Summary for EMI 1

P = Production Quality
experimental = Pre-Production Quality

Optional Features towards EMI 2 (planned and/or in development)

Support for Roles : With DPM priorities can be given to particular Roles.

• Status: Planned
  • 2011-02-18 : Planned for EMI 1 - but not in EMI 1, because of underestimation of the integration work and a reconsideration of the priority of this work item.
• Addressing EMI component improvements

Instrumentation of disk servers Monitoring : In preparation of the EMI common task Monitoring the DPM storage element will improve the internal monitoring and will provide more information to track down problems.

• Status: Planned
  • 2011-02-18 : Instrumentation has not happened before EMI 1 thus not relevant for it
• Addressing EMI component improvements

Instrumentation of disk servers Management of streams to pools : DPM will provide mechanisms to steer the number of streams from individual pools to better manage resources.

• Status: Planned
  • 2011-02-18 : Instrumentation has not happened before EMI 1 thus not relevant for it
• Addressing EMI component improvements

Tracking