Per-section analysis

OVERALL EMI STANDARDIZATION STRATEGY

C&P relevant parts from "EMI Standardization Strategy". Document could not be found on wiki so I put it here.

STATE-OF-THE-ART

To be taken from previous version.

OVERVIEW OF REQUIRED WORK

EMI 1 Timelines

Text by Morris: "This section describes the overall EMI 1 timelines within the context of the EMI Development Roadmap." Probaby it should be "context of EMI Standardization Roadmap".

EMI STANDARDIZATION Objectives

Seems to be similar or superset of previous section. At least I see no difference. Probably previous one can be dropped.

"In addition, this section covers the general requirements of integration with standard operating systems." There is only one standard OS I know about - POSIX. And I think we have no product which would run on pure POSIX. And it is not impelmented by anyone AFAIK.

Overview of Product Changes

Text ffrom Morris: "This section gives a brief overview of the EMI Products affected by the defined Technical Objectives ...". Seems contradicting name of section. Probably both are meant.

Software STANDARDIZATION Strategies

Suggest way of representing information would probably text full of: Area name - nothing to be done. Probably needs reconsidering.

Standardization Pre-Studies

Participation in the Open Grid Forum

Participation in THE Standardization Development Organization XYZ

AFAIK no PT from EMI is participating in any SDO except OGF. Needs to be clarified. TODO: make questionary.

CONTRIBUTIONs To the SIENA Standardization ROADMAP

RELATIONSHIP WITH OTHER DCI STANDARDIZATION WORK

Only DCI which is know to me is EGI. EGI has released standards roadmap which we can mention here - https://documents.egi.eu/document/206 .

PLANS FOR PERFORMING SOFTWARE STANDARDIZATION ACTIVITIES

Morris suggests this section to have references to "EMI 1 Development and Test Plan" which he is currently writing. Same document is also known under name Software Development and test Plan (SDTP) and not to be confused with "Technical Development Plan" DNA1.3.1 .

PROJECT PLANNING AND OVERSIGHT

Lists Products standardization plans. DoW does not list products. But it gives names of components

CREAM, WMS, UNICORE TSI, UNICORE XNJS, UNICORE UAS (Job Management), A-REX, libarcclient, MPI, AMGA, DPM, FTS, Hydra, LFC, StoRM, UNICORE UAS (Data Management), libarcdata, dCache, Arcproxy, Argus, Delegation, gLExec, GridSite, Proxy Renewal, SLCS, Trustmanager, UNICORE Gateway, UNICORE XUUDB, VOMS, APEL, BDII, DGAS, L&B, Service Discovery, UNICORE Service Registry, HED, Grid Monitor, libarcclient service discovery

and those related to standardization - CREAM (OGSA-BES), StoRM, dCache, DPM (SRM), VOMS, Argus (XACML, SAML), APEL, DGAS, JURA (UR), common APIs (SAGA), UNICORE OGSA-BES, ARC OGSA-BES

But recently released "Technical Development Plan" defines PTs (table on page 12) which can be used as contact points for obtaining information about "products". That document also gives diferent list of components than DoW. For simplicity since here I assume that "product" = "component". Below is list of PTs with probably relevant (for standardization) components and comments next to them.

ARC Data Libraries - adoption of SRM and changes in SRM.

ARC Compute Clients - adoption of EMI ES (not a standard), BES, JSDL.

ARC Compute Element - adoption of EMI ES (not a standard), BES, JSDL.

ARC Classic SE - not relevant

ARC Information System - adoption of GLUE2

ARC Security Utils - adoption of VOMS SAML, XACML, ...

ARC Container - not relevant

gLite MPI - can MPI be considered as standard? are they affected by EMI ES?

gLite Job Management - adoption of EMI ES, BES, JSDL

CERN Data - adoption of SRM and changes in SRM

dCache - adoption of SRM and changes in SRM, HTTP(S) for data transfer, WebDAV (at both points)

StoRM - adoption of SRM and changes in SRM, HTTP(S) for data transfer, WebDAV (at both points)

AMGA - SQL interface? Probably not relevant

APEL Client - APEL is accounting service. Adoption of UR and RUS. Participation in OGF?

DGAS Client - DGAS is accounting service. Adoption of UR and RUS. Participation in OGF?

gLite Information System - Glue model, gLite service info providers, gLite site info providers. - Adoption of GLUE2.

SAGA-SD-RAL - SAGA as OGF standard. SD = Service Discovery. Adoption of GLUE2. Adoption of EMI ES information interface.

VOMS - Adoption of SAML, SAML profile of EMI security group.

gLite Security - Trustmanager/Util-java/LCAS/LCMAPS/glexec/SCAS/LCMAPS-plugins-c-pep/Hydra/STS/Delegation

Java/SLCS/Pseudonymity - seems like only Trustmanager is relevant. - Adoption of XACML profile.

CESNET Security - org.glite.security.proxyrenewal, org.glite.security.gss, org.glite.security.gsoap-plugin, org.gridsite - probably org.gridsite is relevant for EMI ES and SRMs and subject for adoption of enhanced delegation.

Argus - adoption of XACML profile.

L&B - can UR be relevant? EMI ES is probably related but in opposite direction.

Unicore Target System - do we have any relevant batch interface standrds?

Unicore WS Interfaces - UAS, OGSA-BES, Service Registry - adoption of EMI ES (alternative to UAS and BES)? Check also against EMI registry.

Unicore Client and APIs - probably adoption of EMI ES interface.

Unicore Container (aka UNICORE Service Hosting) - no relevant

Unicore Security - adoption of SAML and XACML profiles.

Registry - in development - strongly push them to standards.

Messaging - no standrds mentioned in plans. Are there any messaging standards?

Standardization Plan

According to Morris "... planning must be consistent with system-level planning and shall include all applicable items in the SDTP ...". So far there is no SDTP so it is not known which items should be here.

"List the affected Products and links to their development plans". So far it is difficult to hvae links to product plans because we have PTs and not products. But if we substitute PT for product then it probably should be list of wiki URLs. Or maybe also DJRA1.1.1, DJRA1.2.1 and DJRA1.3.1 can be used. So it is not clear yet what to refer to. TODO - check if there are any words about standards. And if not request those deliverable to be rewritten

"Examples are the EMI-1 work performed w.r.t. to the GLUE2 adoption." Statement is unclear - there is no product or PT called "EMI-1". Strange. I guess it just means all PTs involved in Glue2.

--++++ Standard Compliance Plan

"In particular it must describe the standard compliance test cases and test plans or reference those listed in the DJRA1.6 Integration deliverable." I doubt it is possible to do before implementation is in place unless third party tools are used. Maybe good idea - require all PTs to use only third-party tools for compliance testing. And if there are no such tools then create dedicated PT for developing them Of course such PT must be made of independent people.

--+++ STANDARDIZATION and Standard Compliance Tasks

"The section is organized in subsections, one subsection for each Product. Each sub-section contains a description of the standardization and standard compliance test cases and their implementation." That looks like 99% overlap with previous section. Is it really needed? Of course having more pages of same informaition can bore any reviewer to death, hence increasing probability of positive review

--++++ Expected outcome of the related development and testing activities

"Expected outcome of the related development and testing activities" - probably "passed"

--+++ Other Software Development Activities

--++++ Risk Management

Too late?

--++ COMMON EMI STANDARDIZATION ROADMAP

List of standards and year of adoption.

Reference to EGI stand. roadmap.

--++ JRA1 Key Performance Indicator CONTRIBUTIONS

KJRA1.1 and KJRA1.2 . Not clear for which date those values must be given.

--++ SCHEDULES AND ACTIVITY NETWORK

These I know nothing about and have no mentioned tools.

Information to be collected

1. PT name

2. Products relevant for standardization

3. Implemented standards (for soa)

4. Standards group is planning to adopt.

5. Standards which need to be adopted

6. Does group participate/participated in development of any of those standards

7. For covered standards is it really useful for EMI to participate?

8. For not covered standards is it worth and is it possible to participate? Why not particiating?

9. Is all functionality of products covered by standards?

10. For covered - are those satisfactory?

11. For not covered - why not? Which are potentially applicable standards?

Questionary

This is a base for letter to be sent to all PTs/middlewares. It is rough text and probaly needs some little adaptation per PT or maybe even per product/component.

Dear leader/participant,

We are seeking for information necessary to develop a plan for EMI standardization related activities. Please read following questions carefully and provide excessive but related information.

If whole or portions of requested information is available online you may simply refer to them instead of quoting. But in that case please either provide exact reference (web page X, section Y explains our commitment to activity Z) or some hints on how to interpret referred information.

If your PT covers multiple components please state which components is relevant for referred standardization activities.

Among questions You may find information which was already collected about your components from various sources. Please check if all collected information is correct.

It is not necessary to provide this information per each PT. So if You feel it would be more efficient to combine answer with another PT(s) or provide for whole middleware, please do that.

This questionary is more fine grained continuation of previous one that took place last summer. So if You are sure that You already provided all the information You are free to refer to your previous answer.

When asked about about time it is preferable to use relative EMI timing like EMI year 1, EMI month 24, by EMI 1 release.

When asked about standards there is no need to mention commonly used ones like TCP, HTTP or SOAP. We are speaking about non-trivial ones.

So questions are:

1. Do your components implement any standards? Are those standards satisfactory for your needs (please provide arguments if not)?

2. Do You plan to adopt more standards including not released yet? If yes, do You have any time estimations?

3. Do You plan to drop any of those in use now? If yes, do You have any time estimations?

4. Among standards mentioned in previous answers have your project team participated, participating or planning to participate in their development?

5. Do You think some of solutions your components are using deserve standardization?

6. Are there any solutions/fields in your components which could benefit from adopting some existing standard? Or some wish-to-have standard?

Available information

ARC Data Libraries

• 2. Data libraries
• 3. SRM, GridFTP
• 4. SRM modifications, WebDAV (EMI year 3).
• 5. GLUE2 if servers implement GLUE2 (see gLite data management)
• 6. Following activity of EMI data groups.
• 7. -
• 8. -
• 9. File catalogs (LFC, RLS) seems to be not covered by any standard
• 10. Yes.
• 11. Is it possible to adopt something for file catalogs (WebDAV, RNS of OGF)? Can LFC be standardized?

Answer processed

ARC Compute Clients

• 2. Client utilities and underlying libraries and plugins (mostly plugins because they are implementing interfaces).
• 3. OGSA-BES with exrensions, JSDL with extensions, WSRF with simplifications
• 4. EMI ES (not a standard but to become a feedback for OGF PGI)
• 5. Need some standard for service discovery (RNS )
• 6. Feedback to OGSA-BES (pre EMI), feedback to JSDL (pre EMI), EMI ES and hence OGF PGI.
• 7. Probably OGF PGI will not produce anything useful by end of EMI. And if it does, EMI will not have time to implement it. But if planning for future it is worth trying to make sure PGI standard is not too academic.
• 8. -
• 9. Currently almost nothing is standard.
• 10. OGSA BES/JSDL needed many extensions. EMI ES will replace it?
• 11. Needs to be identified. Refer to other PTs/components.

Note: See also "ARC Compute Element"

ARC Compute Element

• 2. ARC Compute Element
• 3. OGSA-BES, JSDL - with extensions. WSRF simplified.
• 4. EMI ES (partially standard), whatever is identified for resource discovery.
• 5. OGF PGI?
• 6. EMI ES and OGF PGI.
• 7. See reference in note.
• 8. Resource discovery - to be identified. WSRF - already established.
• 9. Yes, with extensions.
• 10. No, extesnions were needed.
• 11. Not identified yet.

Note: See also "ARC Compute Clients".

ARC Classic SE

• 2. ARC Classic SE
• 3. GridFTP
• 4. None
• 5. None
• 6. No
• 7. -
• 8. Standard is already established
• 9. Yes
• 10. Yes.
• 11. -

Note: component is being phased out.

ARC Information System

• 2. Classic Infoserver (localLDAP), Classic Infoindex (EGIIS), Grid Monitor, infoproviders (ARC Compute Element)
• 3. None (not counting LDAP)
• 4. None
• 5. Must by sunchronized with outcome of EMI Registry
• 6. -
• 7. -
• 8. -
• 9. Probably not
• 10. -
• 11. to be identified

Note: depending on EMI Registry those components may be out or may be in scope of standardization.

ARC Security Utils

• 2. update-crls, nordugrid map, arcproxy
• 3. X.509 proxy, X.509 AC (for VOMS), MyProxy protocol (OGF)
• 4. -
• 5. SAML (VOMS SAML?)
• 6. No
• 7. No
• 8. -
• 9. VOMS is not standard
• 10. Yes
• 11. SAML EMI profile?

ARC Container

• 2. None
• 3. -
• 4. -
• 5. -
• 6. -
• 7. -
• 8. -
• 9. -
• 10. -
• 11. -

Note: this component is involved in standardization through other components using it.

gLite MPI

• 2. To be identified
• 3.
• 4.
• 5.
• 6.
• 7.
• 8.
• 9.
• 10.
• 11.

gLite Job Management

• 2. CREAM, WMS, CEMon

BLAH (Is BLAH local invention or is it a standard?), CEMon (related to EMI Registry?), CREAM (computing element), WMS (intermediate computing service)

• 3. CREAM implements JSDL and BES (not supported anymore) and Glue 1.3; WMS implements JSDL and Glue 1.3; all implement WS-I (SOAP over HTTP +).
• 4. Upgrade from Glue 1.3 to Glue2 - no time estimate; EMI ES for CREAM by PM18; OGF PGI (if anything produced); Something for resource discovery if defined in EMI.
• 5. Should WMS talk to CREAM using some standard? Should CREAM implement some standard for querying information/managing service? Should WMS be accessible through standard interface? Should CEMon talk to CREAM through some standard interace?
• 6. CREAM participates in EMI ES and in OGF PGI; EMI harmonization of compute area APIs and clients (https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T2Compute_Client):
• 7. EMI is devoting some effort for harmonization which may be achivied through standardization.
• 8. -
• 9. No. See above, also delegation is no standardized. Usable standardizations wrt the interface with batch systems should be useful.
• 10. -
• 11. -

Answer processed.

gLite Data Management

• 2. FTS, GFAL, lcg_utils
• 3. SRM and GridFTP (DPM, clients - FTS, GFAL, lcdg_utils)
• 4. HTTPS as data transfer protocol. SRM is under review for simplification.
• 5. Anything for LFC? Can FTS interface become standard? * DPM - NFS4.1 (control or data), currently a prototype, expected for EMI-2 * DPM - HTTP/WebDAV(control or data) - full support expected after NFS4.1 is completed, * DPM - HTTPS for SRM - EGI-2 * DPM/LFC/FTS - Glue2.0 (for EMI-1), SSL (replacing GSI, EMI-2) ** gfla/lcg_util - Glue2.0 (PM15)
• 6. Participated in SRM2.2
• 7. No related ongoing standards development
• 8. No
• 9. Seems like all is covered.
• 10. As SRM is under review then probably not everything is satisfactory
• 11. No such functionality

Answer processed

dCache

• 2. dCache storage and clients
• 3. SRM, GridFTP
• 4. WebDAV. NFSv4.1. Is it for data access (replacement for GridFTP) or in parallel to SRM?
• 5. Probably nothing.
• 6. Ask
• 7. -
• 8. -
• 9. It is SRM specific storage management so probably yes.
• 10. Ask
• 11. -

StoRM

• 2. StoRM
• 3. SRM2.2, GridFTP
• 4. HTTP(S): EMI-1, WebDAV(for control and for data): EMI-x, x>2.
• 5. No
• 6. Participated in SRM.
• 7. Yes, SRM.
• 8. -
• 9. Probably yes
• 10. No complaints
• 11. -

Answer processed

AMGA

• 2. It would be nice to have standard interface for AMGA
• 3. Ask
• 4. Not known
• 5. How about OGF RNS?
• 6.
• 7.
• 8.
• 9.
• 10.
• 11.

Note: "AMGA is the ARDA Metadata Grid Application" - ARDA?

APEL Client

• 2. APEL?
• 3. ?
• 4.
• 5. UR, RUS?
• 6.
• 7.
• 8.
• 9.
• 10.
• 11.

Note: This only covers client. So its functionality is fully defined by capabilities of service. Hence probably one should not expect too much standardization efforts from this PT. But need to ask anyway. These components seem to be similar to DGAS.

DGAS Client

• 2. DGAS?
• 3. ?
• 4.
• 5. UR, RUS ?
• 6.
• 7.
• 8.
• 9.
• 10.
• 11.

Note: This only covers client. So its functionality is fully defined by capabilities of service. Hence probably one should not expect too much standardization efforts from this PT. But need to ask anyway. These components seem to be similar to APEL.

gLite Information System

• 2. BDII, Glue model, Service info provider, site info provider, Infosys test suit, two cli (lcg-info and lcg-infosites)
• 3. GLUE 2.0 LDAP rendering, LDAP.
• 4. None
• 5. Let's see what comes from Resource discovery
• 6. GLUE 2.0
• 7. Interoperability
• 8.
• 9. Yes
• 10.
• 11.

Answer processed

SAGA-SD-RAL

• 2. SAGA Service Discovery APIs:
• 3. SAGA is OGF standard
• 4.
• 5.
• 6.
• 7.
• 8.
• 9.
• 10.
• 11.

Note: Related to EMI Registry. Otherwise looks like obvious case.

VOMS

• 2. VOMS, VOMS-Admin
• 3. SAML-tokens
• 4. REST? Is there any interface standardized in SAML? Ask
• 5. EMI SAML profile (not a standard but close)
• 6. EMI SAML
• 7. Interoperability
• 8.
• 9.
• 10.
• 11.

gLite Security

• 2. Trustmanager/Util-java, LCAS/LCMAPS, glexec, SCAS, LCMAPS-plugins-c-pep, Hydra, STS, Delegation Java, SLCS, Pseudonymity
• 3. SLCS is related to Shibboleth, STS implements OASIS WS-Trust. Still more information is needed.
• 4.
• 5.
• 6.
• 7.
• 8.
• 9. Important for so many components.
• 10.
• 11.

Java/SLCS/Pseudonymity

Note

on wiki it is same page as "gLite Security". Hence no separate info.

CESNET Security

• 2. glite-security-gss, glite-security-gsoap-plugin, glite-security-proxyrenewal and org-gridsite
• 3.
• 4.
• 5.
• 6.
• 7.
• 8.
• 9.
• 10.
• 11.

Note: refrence is quite messy. Questionary is a must.

Argus

• 2. Argus Authorization Service
• 3. XACML, Hessian
• 4. EMI XACML Profile (not a standard but close)
• 5.
• 6. EMI XACML
• 7. Interoperaility
• 8.
• 9. Mostly
• 10.
• 11.

Plan - https://twiki.cern.ch/twiki/bin/view/EMI/ArgusPTWorkplan

L&B

• 2. L&B server and clients
• 3. RSS, JDL (but seems to be not a standard), ULM(?), Common Logging Format (?)
• 4. EMI-ES, messaging (see messaging) or WS-Notification, WS-DAI(R/X) for data access.
• 5.
• 6. None. Only participating in choice of standards for messaging.
• 7.
• 8.
• 9. No
• 10.
• 11. "L&B works with messages carrying information on grid components processing their tasks, especially when the responsibility for a task is being handed over from one component to another. Schema and meaning of these messages are worth standardizing, similarly to the usage record format standardized by OGF." Standard for service discovery is needed.

Answer processed

Unicore Target System

• 2. TSI
• 3. OGF JSDL 1.0, OGF SPMD extension to JSDL 1.0, OGF HPC Base Profile 1.0
• 4. Successor to JSDL 1.0 once it is developed.
• 5. As TSI is interacting with cluster batch system may we should check if there is any standardized batch system interface.
• 6. Planning to participate in the development of the next JSDL version (2.0).
• 7. Let's wait for JSDL 2.0 starting
• 8. -
• 9. Yes
• 10. Yes
• 11. -

Answer processed

Unicore WS Interfaces - UAS, OGSA-BES, Service Registry

• 2. OGSA-BES, Service Registry
• 3. OGSA BES, JSDL
• 4.
• 5. EMI ES (not a standard but close),
• 6. EMI ES and hence OGF PGI
• 7. Interoperability
• 8.
• 9. In parallel to OGSA BES also uses own interface.
• 10. Probably not (like every computing service uses own extensions)
• 11.

Unicore Client and APIs

• 2. UCC, UNICORE internal client libs, HILA
• 3.
• 4.
• 5.
• 6.
• 7.
• 8.
• 9.
• 10.
• 11.

Note: information is merged with "Unicore WS Interfaces". UNICORE clients follow whatever is done in the WS interfaces.

Unicore Container (aka UNICORE Service Hosting)

• 2. WSRFLite, X
• 3. WSRF, WS-Addressing, OGSA WSRF Basic Profile
• 4. REST services using the JAX-RS specification (http://jcp.org/en/jsr/detail?id=311) (by EMI-1).
• 5. -
• 6. Participated in development and interoperability testing of the "WSRF" and "OGSA WSRF Basic Profile" standards.
• 7. -
• 8.
• 9. The SAML based authentication and trust delegation used in UNICORE deserve standardisation. See also Unicore security.
• 10. -
• 11. -

Answer processed.

Unicore Security

• 2. UNICORE Gateway, UVOS is phased out, XACML Entity, uas-authz, UNICORE security libs
• 3. * Gateway: X.509 proxy (with additional module, to be redone), passes SAML through, has WS-Addressing, * Security libs implement SAML assertions and protocol parsing, signing with WS-Security, * XACML entity: XACML 1.x and XACML 2.0, SAML Profile for XACML, * uas-authz: SAML 2.0: attribute assertion query protocol, pushed attribute assertion parsing, SAML XACML attribute profile.
• 4. probably nothing new except EMI XACML and SAML profiles (not stadards but close). New support for X.509 proxy with new Authn libraries of EMI.
• 5. -
• 6. No
• 7. -
• 8. -
• 9. * Explicit trust delegation would be nice to have as a standard. * We plan to implement a security session concept on top of web services. The aim is to minimize the amount of SAML assertions which are currently passed with each request, what is a significant overhead. I'm not aware about a concrete standard that implement the following

functionality but it is possible that one exists. If so then yes. Otherwise we can do this using WS-Addressing.

• 10. Yes
• 11.

Plan - https://twiki.cern.ch/twiki/bin/view/EMI/UNICORESecurityWorkPlan Answer processed.

Registry

• 2. Not defined yet
• 3. Not defined yet. Solutions from different middlewares use different standard and non-standard solutions. Ask.
• 4.
• 5.
• 6.
• 7. If any standard under development is chosen it would be probably beneficial to participate. Pro: good opportunity for common solution.
• 8.
• 9.
• 10.
• 11.

Messaging

• 2. No defined products yet
• 3. None. There are no products and even no finalized plans for products.
• 4. STOMP (de-facto standard) is strongest candidate. AMQP and JMS are also possible. But no time estimation is available due to easly design stage.
• 5. None identified
• 6. Currently participating in STOMP 1.1
• 7. No
• 8. No
• 9. No. Still in design phase after all.
• 10. Yes. Although requirements are not collected yet, so probably yes is not 100% strong.
• 11. Too early.

Answer processed

Identified standards and standardization activities

SRM

GridFTP

Adopted. How about v2?

WebDAV

HTTPS (as data transfer protocol)

NFS4.1

SSL as replacement for GSI

GLUE 1.3 and 2

OGS-BES (+extensions)

JSDL

EMI ES as feedback to OGF PGI

WSRF

WS-Addressing

OGSA WSRF Basic Profile

LDAP

Too low level. Schemas are hopefully GLUE.

X509 - proxy, AC

MyProxy (OGF)

SAML

Mostly for VOMS attributes. Unicore has own attributes.

SAML EMI profile

WS-I (a bit more than SOAP over HTTP)

SAGA (OGF)

WS-Trust (SLCS)

XACML

EMI XACML profile

WS-Notification (or messaging)

RSS

REST (JAX-RS used by Unicore)

WS-Security

STOMP (de-facto, messaging)

Areas not covered by standards

File catalogs and metadata services

X.509 delegation

SAML based trust delegation

Registry (push hard)

-- AleksandrKonstantinov - 14-Dec-2010

• EMI-DJRA1.5.1-CDSREF-Standardization_Work_Plan_and_Status_Report-v6_0.odt: DJRA 1.5.1 rewrited document - work in progress