TWiki> EMI Web>EmiProjectStructure>JRA1>EmiJra1>EmiJra1T7Integration>DJRA1_6_2-OBJECTIVES (2011-05-25, FloridoPaganelli) EditAttachPDF

DJRA1.6.2-OBJECTIVES

Main goals of DJRA162

The purpose of this document is twofold:
  • a) capture and present the current integration status of EMI components, as of "EMI-1". Describe the "state of art of integration of EMI components", set a reference point so that further EMI releases could be compared against the initial status. (Andre)
  • b) serve as a workplan, contain a "list of tasks, assignments" for EMI product teams and for the integration task personnel itself, so that in the project everybody would cleary know which integration tasks must be carried out by which dates in order to reach the EMI-1 release.

This leads to the following questions regarding the status report for DJRA1.6.2:

  • 1) Did you achieved the objective successfully, so it can be considered as completed?
  • 2) Have you performed any integration tests to verify the integration? (Yes/No)
  • 3a) If no, are you still going to perform any integration tests? (Short description)
  • 3b) If yes, what testing methodology have been applied? (Short description).
  • 4) Do you have any general comments regarding the integration aspects of this objective?

This leads to the following questions regarding the work plan for DJRA1.6.2:

  • 1) Who is responsible for the integration (Product team) ?
  • 2) What are the corresponding ETICS components (if already available)?
  • 3) Are there any plans for integration testing? How could the integration task be tested (specific integration test suite, standard compliance tests) ?
  • 4) Do you plan to perform any integration tests on the EMI SA2-testbed?
  • 5) Do you have any general comments regarding the integration aspects of this objective?

DNA1.3.2 objectives relevant for DJRA1.6.2's status report

Area Title Status Comment Wiki Link
Compute Extend job definition language, resource information (GLUE model) and job management service capabilities so that EMI compute clients are able to request access to virtualized resource managers and appliances. obsoleted: replaced by another virtualization/cloud objective Replaced by C10. Clarify why and how the cloud objective replaced the GLUE objective  
Compute Successful computational usage of emerging computing models i.e. clouds with EMI components (scaling out to clouds). obsoleted: replaced by another virtualization/cloud objective Not mentioned in DJRA1.6.1. because no integration aspect. Won't be mentioned in status report.  
Data (D1) All storage elements publishing initial GLUE 2.0 storage information. Delivered in EMI-1 Persons responsible (Patrick?) have to be contacted to get more details. Affected components: Storm, dCache, DPM
Data GLUE2.0
Data (D2) Using https instead of httpg for the SRM protocol as a prototype implementation in one storage element and client (library). Delivered in EMI-1 Persons responsible (Patrick?) have to be contacted to get more details.

Florido: ARC Data people answered that gLite client seems to be elected. Howhever ARC clients already have this functionality under testing. 		SRM HTTPS
Data (D3) All storage elements offering support for the http(s) protocol. Partly delivered in EMI-1 (except of STORM) Persons responsible (Patrick?) have to be contacted to get more details.  
Data (D4) All storage elements offering at least a prototype-level support for the "file://" access protocol. Delivered in EMI-1 Persons responsible (Patrick?) have to be contacted to get more details. NFS file protocol
Data (D8) Storage elements offering support for the WebDav protocol. Still to be done for STORM, DPM. Partially delivered in EMI-1 DJRA1.6.1 ref: 4.2.1 Todo: Find out for which components finished. Persons responsible (Patrick?) have to be contacted to get more details.  
Data (D10) Overall consolidation of data area by adopting a consistent interpretation of SRM Delivered in EMI-1 Florido: ARC will follow discussions and implement whatever if needed. SRM consolidation
Data (D15) Agreement over a common storage acounting record including the refinement, definition and adoption (if/when applicable) of relevant standards. Achieved, see xxx Florido: ARC people involved, ongoing  
Data (D20) SRM-capable clients and services should add “file://” to the already supported access protocols. Delivered with EMI-1 for STORM To clarify: Relevant for other clients?

Florido: ARC client supports it already. 		 
Security (S1) Agreement on a minimal common set of security attributes to be used in policies. Achieved, see xxx    
Security Consolidation and reduction in the number of security CLIs so that the users don’t have to face the very different clients and utilities. Merged into 6 Obsoleted: merged with another objective. Todo: Clarify which objective    
Infrastructure (I1) Provide early internal guidelines for integrating messaging into potential EMI target components Achieved No integration aspect  
Infrastructure (I2) Design a common EMI service registry that is required in order to discover all the service endpoints of the different middleware components Achieved, draft design    
Infrastrucure (I5) Investigate possible use cases for a common standard messaging system for the information services and L&B. achieved Probably not an integration objective  
Cross (X9) The legacy Globus security infrastructure (GSI) will be replaced with a common security solution based on TLS/SSL and EMI delegation method. ALL Partially delivered in EMI-1 execept of delegation part Also listed in integration objectives table  

Moved items:

Data (D3) All storage elements offering support for the http(s) protocol. To be done for STORM M?? ongoing ? HTTPS integration JJ: I would remove it but it was in DJRA1.6.1, move to status
Data (D5) File Catalogue Access from UNICORE data M18 ongoing ? ? JJ: it was in DJRA1.6.1, move to status
Data (D6) A storage client is capable consuming GLUE 2.0 information published by storage elements. M16 ongoing ? ? Was in DJRA1.6.1 at least for ARC and UCC client
Data (D9) Using https instead of httpg for the SRM protocol as a production implementation in all the storage elements and clients utilizing the EMI delegation. M25 ongoing ? ? mentioned DJRA1.6.2 Status Report, problem with delegation

Identified integration objectives from DNA1.3.2 relevant for DJRA1.6.2's work plan

Area Title Due Status Integration aspect Responsible Comment
Compute (C1) Glue 2.0 support in job management services (LDAP and/or XML rendering). M16 Partially delivered in EMI-1 (todo: CREAM LDAP and XML) Glue 2.0 integration Florido? Florido: ARC supports GLUE2 both server and client side.
However XML GLUE2 schema has been recently published so XML side has to be reviewed.
Compute (C2) Glue 2.0 support in matchmaking modules and client tools. M22 ongoing Glue 2.0 integration ?  
Compute (C3) Implementation of the agreed common job submission and management methods (EMI-ES interface) in all the Ces. M19 ongoing EMI-ES integration ?  
Compute (C4) Implementation of the agreed common job submission and management methods (EMI-ES interface) in compute elements. M20 starting EMI-ES integration ? same as C3?
Compute (C8) EMI Computing Services should provide fully integrated solutions to interface with identified set of batch systems.. M20 ongoing Batch system interface integration ?  
Compute (C10) Implement the EMI cloud strategy within compute area. M32 starts later ? ? To be clarified if cloud strategy implies implementations/integration work
JJ: nothing to do with integration
Compute (C12) Implementation of the common parallel execution framework across the EMI computing services. M32 ongoing ? ? Implementation = Integration work ?
Compute (C13) Extend the parallel computing capabilities to better address multi-core jobs on all emerging architectures resources, multi-node execution on interconnected clusters; and special scenarios like advanced topologies, FPGAs, GPGPUs M32 ongoing ? ? Implementation = Integration work ?
JJ: I would remove it
Data (D7) All storage elements publishing full set of GLUE 2.0 storage information and the EMI data client is capable consuming that. M22 ongoing ? ? TODO: check when EMI data client is released it has something to do with client harmonisation
Data (D8) Storage elements offering support for the WebDav protocol. Still to be done for STORM, DPM M25 ongoing
? ? Todo: Check if already mentioned in DNA1.3.1.If yes add to DJRA1.6.2 Status Report
Data (D11) Providing a common set of data access libraries at least between gLite and ARC M22 ongoing

Florido: Answer from the Task Leader: the current plan is to deliver a plan in PM18 (project month 18), implement whatever needed with PM22 and do testing until PM36. For ARC, one thing we'll need to do is to implement a gLite data plugin in ARC. 		? Florido?  
Data (D13) Integration of SRM-based access into UNICORE storage management M22 Starts later ? Andre  
Data (D16) Add support for storage space usage accounting to SEs/FTS based on the agreed record. M32 Starts later ? ?  
Data (D18) Integration of AMGA-based access to UNICORE storage management. M28 Starts later ? Andre Concerns AMGA, UAS-D
JJ: this objective is still in discussion as far as I know, I vote for removal
Data (D19) Design and implement the next generation FTS, a distributed next generation file transfer service that amog others utilizes the common messaging system M30 Preparatory investgation

Florido: No FTS in ARC. Most likely there will be need for adopting changes in the ARC data clients. 		? ? Ideally an glite expert
JJ: don't see integration aspect here
Security (S2) Simplified management of security credentials by reducing the complexity of handling certificates and integrating different security mechanisms like Shibboleth and Kerberos across the EMI stack that allows users to use their own authentication system to access a ``Grid''. M22 ongoing ? Andre  
Security (S3) Provide common authentication libraries supporting X.509 and optionally SAML M22 ongoing - API definition ready ? Andre  
Security (S4) Agreement and full support for a common single X.509 and SAML based Attribute Authority Service integrated with all EMI components. M25 ongoing ? Andre  
Security (S6) Implementation of the security area consolidation plan. M32 Starts later ? Andre To be clarified what implementation means here
Security (S7) Provide a transparent solution for encrypted storage utilizing ordinary EMI SEs M32 Starts later ? Andre To be clarified if integration work will arise
JJ; I would remove it
Infrastrucure (I6) Implement the common EMI Registry M22 Preparatory investigation Integration testing ? JJ: Shiraz is implementing that, I will contact him
Infrastrucure (I7) Fully utilize and support the GLUE2 information model in information components including development of validation tools M20 ongoing GLUE2 integration testing ? Florido: ARC has some functionality tests to check that the info published is compliant to GLUE2 schema. Realized using Lawrence's validator. Can provide links if needed
Infrastrucure (I10) Implement or adapt the accounting record publishers of compute and data area services to use the common messaging system M28 starts later ? ? Todo: Identify integration aspect
Cross (X6) Implementation of the EMI SAML profile all over the middleware stack. M28 starts later SAML integration testing Andre  
Cross (X7) Integration of the compute area services with the ARGUS authorization framework M18 ongoing ? Andre  
Cross (X8) Initial integration of the storage elements with the ARGUS authorization framework M22 ongoing ? Andre  
Cross (X9) The legacy Globus security infrastructure (GSI) will be replaced with a common security solution based on TLS/SSL and EMI delegation method. ALL M30 Partially delivered in EMI-1 execept of delegation part ? Andre  
Cross (X10) Adapt or implement monitoring interfaces, sensors, providers for compute, data, security and infrastructure services to allow the use of standard monitoring tools preferably based on the common EMI messaging system I M30 Starts later ?   JJ: don't see integration here (it is integration with "outside" elements, out of scope)
Cross (X11) Complete the rewrite of components utilising the new AuthN libraries. All M32 Starts later ?   Todo: Identify integration aspect
Cross (X14) Adhere to operating system standards for service operation and control regarding configuration, log and temporary file location and service start/status/stop. All M18 ongoing ?   Todo: Identify integration aspect
JJ: out of scope
Cross (X15) Port, release and support EMI components on identified platforms (full distribution on SL6 and Debian 6, UI on SL5/32 and latest Ubuntu). All M22 ongoing ?   Todo: Identify integration aspect
JJ: integration with "outside", out of scope
Cross (X16) Provide optimized semi-automated configuration of service backends (e.g. databases) for standard deployment. All M30 Starts later ?   Todo: Identify integration aspect
JJ: integration with "outside", out of scope
Cross (X17) Introduce minimal DOS protection for EMI services via configurable resource limits. All M28 Starts later ?   Todo: Identify integration aspect
JJ: out of scope
Cross (X18) Provide and support monitoring probes for EMI services (e.g. Nagios). All M18 ongoing ?   Todo: Identify integration aspect
JJ: out of scope

Identified DNA1.3.2 objectives with no integration aspect regarding DJRA1.6.2

Area Title Due Status Comment
Compute (C5) Investigate solutions to improve interactive access capability of at least one EMI Computing Element. M18 preparatory investigation  
Compute (C6) Agreement over a compute accounting record (UR). M16 ongoing  
Compute (C7) Support for the agreed compute accounting record (UR) by compute area services. M22 starts later  
Compute (C9) Consolidation and harmonization of compute area clients/APIs M25 ongoing  
Compute (C11) Proposal for a common parallel execution framework, a “backend” across the different computing services to allow users to execute parallel applications in a uniform way. M32 ongoing  
Data (D12) Solve the synchronization problem of the storage elements and the file catalogue M25 ongoing  
Data (D21) Investigate the possibility to support http/webdav for LFC in order to provide a standard user-friendly access method to catalogues. M16 Preparatory investigation There seems to be no integration work at the moment
Data (D22) Investigate solutions to work with EMI data services in the context of persistent data Ids. M24 Starts later There seems to be no integration work at the moment
Security (S5) Investigate solutions to work with EMI data services in the context of persistent data Ids. M18 Preparatory investigation There seems to be no integration work at the moment
Infrastrucure (I3) Investigate possible use cases for a common standard messaging system in the accounting area. M14 ongoing  
Infrastrucure (I4) Investigate possible use cases for a common standard messaging system for the service monitoring and management. M14 ongoing  
Infrastrucure (I8) Provide guidelines for 3rd parties to integrate messaging into their service/application based on the EMI experience M24 starts later non-technical objective?
Infrastrucure (I9) Deliver the EMI cloud architecture and strategy M18 ongoing non-technical objective?
Infrastrucure (I11) Devise a plan for substantial simplification and reduction in the number of infrastructure area CLIs, libraries, internal components and services M18 ongoing non-technical objective?
Infrastrucure (I12) Investigate service remote management interface for compute, data, security and infrastructure services, including remote configuration change and service management, utilizing the messaging system M24 starts later non-technical objective?
Cross (X1) Define the Information Flow architecture describing messaging and non-messaging based information exchange of the EMI components (e.g. service registry, information system, accounting, monitoring, and instrumentation). A common information exchange between the EMI components is preferable. M9 Achieved non-technical objective?, so nothing for status report
Cross (X2) Investigate possible use cases for a common standard messaging system in the computing area M12 Achieved non-technical objective?, so nothing for status report
Cross (X3) Investigate possible use cases for a common standard messaging system in the data area M12 Achieved non-technical objective?, so nothing for status report
Cross (X4) Agreement on common EMI delegation method. S M18 ongoing non-technical objective?
Cross (X5) Definition of a common SAML profile all over the middleware stacks. S,D,C M16 partially achieved non-technical objective?
Cross (X12) Publish coherent GLUE2-based version information as part of service description in order to facilitate service discovery and monitoring. All M16 Preparatory investigation non-technical objective?
Cross (X13) Improve usability of client tools based on customer feedback by ensuring a ) better more informative, less contradictory error messages b) coherency of commands line parameters. All M22 Preparatory investigation non-technical objective?
Cross (X19) Evolve EMI components to meet specific user requests. All M32 ongoing  
Cross (X20) Increase performance of EMI services. All M32 ongoing  

-- AndreGiesler - 06-May-2011

Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r12 - 2011-05-25 - FloridoPaganelli
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback