UNICORE AMGA Integration

This document is to record facts, ideas and opinions for the AMGA integration into UNICORE.

Amga/Unicore Use Cases

1. Amga as a catalog
   • user search for the data relevant for its grid job by submitting a query to the Amga server
   • Amga returns an "agnostic" URI which is then used as input in the grid job description (JSDL file)
   • The job is submitted to Unicore, the resources from URIs staged-in in the job preparation phase
   • Important limitation: Access rights management. Unicore server must be authorized to access the resources from the "agnostic" URIs
2. Amga as a storage
   • since Amga can also store the data, the 1) scenario can be "specialized": the "agnostic" URI will point to the AMGA server
   • the rest of the process remains unchanged
3. Amga as a directly accessible catalog (and storage)
   • in 1) and 2) user accesses the Amga service by using Amga client independently from Unicore
   • Unicore client could be extended to access the Amga server directly (for search and access metadata in the process of job preparation)
   • Important limitation: EMI incorporates only the command line Unicore client whereas Amga offers a graphical client. So the value-added (comparing to the current state) is questionable. Unicore graphical client cannot be included in the EMI.
4. Amga as a Unicore storage
   • Amga could be accessed as a storage via Unicore Storage Service
   • An adapter (implementation of the Unicore.StorageManagement interface) must be implemented
   • User will then access Amga via Unicore facade (by using Unicore client)
   • Important limitation: the current storage manager interface does not support metadata operations (like search). Changing that is impossible, as it affect other storage implementations and destroys the clear defined abstraction of the storage elements in the Unicore.

Integration

The use cases yield three levels of integration of the Amga in the Unicore stack:

1. enabling access to the resources indexed by the Amga server (this is partially achieved now if access rights issue is managed)
2. incorporating Amga client in the Unicore command line client
3. incorporating Amga server in the Unicore service stack

Integration Timeline

• M16: evaluation
• M25: implementation
• M28: deployment

Discussion

(22.06.) Comment from Jedrzej:

I have discussed the issues here in Juelich with other developers. From our perspective the 3rd level is a no-go as it brings a lot of work, involves non-trivial security questions and provides no value-added for the user.

2nd level integration is doable. It boils down to adding one or two additional commands to the command-line client (to search and access the metadata). I have heard in Lund that there is a Java API for accessing the Amga server. (see Links) However, here strong involvement of the Amga developers is needed as we don't have the expertise of working with the API and configuring the Amga client. On our part we provide the knowledge and support needed for extending the Unicore command line-client.

(06.06.) Questions answered by Bernd:

Neither of us had an exact idea on which point until UNICORE job treatment the job gets its remote data (like from LFC/SRM), since we agree that it doesn't happen during the actual job execution. Maybe you could point this out a bit for us?

A job description (in JSDL format) contains data stage in sections, which are processed by the UNICORE/X server before submitting the job to the resource manager (e.g. Torque). Each stage in is composed of a Source URL and a target file name. As you pointed out this works differently from glite, where data staging is done after the job is sent to the batch system. The data staging support in UNICORE is extensible, i.e. stage-in handlers for new URL schemes (ftp://,http://,etc) can be added. For sure, from within a job one could invoke some external tools, but for data staging I would say this is a bad idea, and it does not conform to the "usual" UNICORE way of life.

Can we consider whether an implementation similar to LFC/SRM access is useful?

Since AMGA is primarily a metadata store, adding some search tools to UCC might be more to the point. On the other hand, since client tools already exist, even this is not necessary. Provided the data is accessible to UNICORE, i.e. on some storage which is accessible to the UNICORE/X server, no extra work is required...

Links

AMGA Java API

-- ChristianLoeschen - 23-Jun-2011

• lund_amga_meeting_notes: Notes from the AMGA/UNICORE meeting in Lund