Survey of existing clients for compute and related interfaces

This survey is supposed to set the ground for the compute client consolidation and harmonization task force. It is intended to list and categorize existing clients in order to guide the harmonization or consolidation activities. The effective date considered for this survey is the EMI-2 release due at the end of April 2012.

Participants

• Björn Hagemeier (UNICORE)
• Andre Merzky (SAGA)
• Antony Wilson (SAGA)
• Alvise Dorigo (CREAM)
• Zsombor Nagy (ARC)
• Zdeněk Šustr (L&B)
• Balazs Konya

Existing Clients

The clients to be evaluated are

• EMI-ES Clients
• WMS Clients
• CREAM Clients
• UNICORE/X
  • UAS
  • BES
• ARC Client
• L&B

The following sections will deal with each of the clients separately, listing their characteristics in the following categories:

• Available platforms
• Features and capabilities
  • Job execution
  • workflows
  • Brokering/match-making
  • ...
• Users and user communities
  • Number
  • Field of interest
• Programming languages (mostly relevant for client side APIs)
• Documentation status
• List of products and higher level frameworks which rely on the client/library
• Developer team and effort, how much is provided through EMI
• "What will happen when EMI ends?" (post-emi plans)

EMI-ES Clients

ARC

See the general ARC Client section.

CREAM

• Supported platforms: SL5/6 EPEL;
• C++ APIs and CLI (separate components); APIs have a quite high level of abstraction, in the sense that a third developer has not to deal with SOAP and low level communication details.
• Deps: gSOAP, GridSite, gSOAP-Plugin/GSS, VOMS, Globus;
• External deps: boost, xml2
• Supported operations: activity creation and management (status, cancel, resume, restart, wipe, pause, info), activity list; delegation initialization/renewal and info; support for both (voms-)proxies and "pure" certificate/key; support for ISB/OSB movement is planned as possibile EMI2-update
• ES Client is not yet released; it will be available in EMI2 release in combination with ES service implementation in CREAM; hence no user community yet.
• ES UI user's guide here (almost finished). No API nor developer's guide; only end user CLI guide so far; developer's guide will be written later.

UNICORE

UNICORE developers have implemented a native client library for EMI-ES during the course of the EMI-ES implementation. As usual in the UNICORE environment, they are based on the use-core Web Services stack that adds additional functionality into the plain XFire stack. The programming language is Java, such that the library is available on all platforms for which a Java implementation exists.

Based on the native EMI-ES library, a backend for the HiLA Grid Programming API has been implemented, which automatically exposes EMI-ES functionality in any code based on HiLA. This is particularly true for the HiLA Shell, which is

Summary

See TestPlan26 for further details.

List of products and higher level frameworks which rely on the client/library

Developer team and effort, how much is provided through EMI

Post EMI Plans

WMS Clients

Platforms

• SL5/6

Features

• Job Submission and management (cancel, status, extented info aka "logging-info"); ISB shipment (jobsubmission) and retrieval (dedicated command line)
• Job list match: search for best matching resources (CEs) for the current JDL's requirements (by mean of query to WM)
• Delegation
• Documentation WMS UI here

The WMS Client sends jobs to the WMProxy service that in turn sends them to the match maker (MM). The MM choses the best "fitting" resource (CE) to submit the jobs to.

Users

• HEP community

Programming Languages

• C++

List of products and higher level frameworks which rely on the client/library

Developer team and effort, how much is provided through EMI

Post EMI Plans

CREAM Clients

Platforms

• SL5/6

Features

• Job submission and management (cancel, status, info, resume, pause, purge) with ISB shipment
• Job's OSB retrieval
• Delegation
• End user's UI guide: emi1 and emi2.
• C++ Developer's guide (emi1 so far) here
• Python Developer's guide here

The CREAM Clients are used for direct interaction with the CE (direct submission/cancellation, status query, etc. ). No brokering capabilites so far.

Users

• HEP community and other smaller VOs

Programming Languages

• C++
• Deps: gSOAP, boost, gSOAP-Plugin, GSS, VOMS, GridSite, Globus

List of products and higher level frameworks which rely on the client/library

Developer team and effort, how much is provided through EMI

Post EMI Plans

UNICORE/X Clients

• BES Clients
• UAS Clients
• URC
• HiLA

Platforms

• Java
  • Windows
  • Linux
  • Mac

Features

• Job Execution and management
• File Management and access
• Workflows
  • URC

Users

• URC: all UNICORE users, ~1000
• UCC: ~400
• HiLA: ~10
• UAS Clients: being a programming API, all other clients depend on them
• BES clients: UCC and possibly others
• Fusion, Chemistry, Biology, Bio-Informatics, Life-Sciences

Programming Languages

• Java

List of products and higher level frameworks which rely on the client/library

Developer team and effort, how much is provided through EMI

• 1 FTE

Post EMI Plans

• Continue as before EMI
• Distribution via SF.net
• Keep the RPM and Debian packaging developed in EMI

ARC CE

Platforms

Packages is provided for the following platforms:

• RedHat EL 4, 5, 6
• Scientific Linux 5, 6
• Fedora 3-16
• Debian 4.0-6.0
• Ubuntu 8.04-11.10
• Mac OS X 10.6+
• Microsoft Windows XP, Vista, 7

All both 32- and 64-bit architectures.

Features

• Proxy creation and credential handling capabilities (including VOMS and MyProxy support)
• Job execution (including resubmission) to general BES, ARC XBES, ARC GRIDFTPJOB, EMI-ES, CREAM
• Brokering/match-making
• Job management
• Resource discovery from EMI Registry, Top-BDII and Site-BDII, EGIIS, ARIS
• Data movement (staging in and out, list, copy, remove, make directory)
• Job list synchronization
• Test job utility
• Job description translation capability (converting between different job description languages)

Users

• Several hundreds of users (CLI)
• ATLAS jobs being sent to ARC resources (library)
• SWING (library)
• Nordic NGIs (CLI and library)
• Baltic NGIs (CLI)
• Ukrainian communities
• other communities from all sciences (natural sciences, humanities, medical sciences etc.)

Programming Languages

The CLI and the library is written in C++ as a modular framework. The API of the library is supported in the following programming languages:

• C++
• Python
• Java

Documentation status

Exist: user manual (somewhat outdated), job description manual, library manual (rather outdated). Missing: developer manual.

List of products and higher level frameworks which rely on the client/library

Various portals (Lunarc, P-Grade, SCMS etc), graphical user interfaces and workflow tools (ARC GUI, GANGA, ARC Control Tower etc)

Developer team and effort, how much is provided through EMI

1.5 FTE developers + testers + packagers (almost everything funded by EMI)

"What will happen when EMI ends?" (post-emi plans)

• NorduGrid will continue taking care of it.
• Additional international funding will be sought.

L&B

Platforms

• SL5/6
• Debian

Features

• workflows
• monitoring
• job status/event queries
• messaging/notifications (subscribe/receive)

Users

• All gLite users (HEP, computational chemistry...)
• Dashboards (VO, infrastructre-wide)
• Czech NGI users

Programming Languages

• C
• C++ and Java bindings
• limited querying capability supported through a WS interface

Documentation status

• Complete
• http://egee.cesnet.cz/en/JRA1/LB/documentation.php

List of products and higher level frameworks which rely on the client/library

• WMS
• RTM

Developer team and effort, how much is provided through EMI

• Team located at CESNET: approx. 1.8 FTE (whole L&B product), all funded by EMI
• occasional community contributions

"What will happen when EMI ends?" (post-emi plans)

• Integrated with the Czech NGI, support will be available on national level, primarily answering to the needs of the national grid.
• International funding will be required (and sought) for international-level support, following a no-guarantee grace support period of approx. six months.

SAGA

SAGA is not a client targeted at a specific service implementation but rather an abstraction for which there are multiple implementations. This section is intended to document SAGA in the same manner as the othehr clients anyhow.

Platforms

• any Unix with boost > 1.33.1, python and g++
• binary packages for SL5/6, Ubuntu
• additional dependencies depend on supported backends

Features

• multiple backends
• relatively high level of abstraction
• Implementation of SAGA API spec of OGF
• Job Execution and Management
• File Management and access
• Some other packages which are not widely used

Limitations

• No native workflow support
• API too rich for some, not rich enough for others...
• boost dependency
• adaptors are in very different levels of maturity

Target Services

• Globus
• SSH
• Torque / PBS / SGE
• Condor
• OGSA-BES (HPC-BP): Arc, UNICORE, Genesis, QosCos, BES++
• Others
• gLite/CREAM not well tested

Users

• ~20
• Genome Sequencing, Molecular Dynamics, Medical imaging, CS research, education

Programming Languages

• C++ API, Python API, CLI
• Python is more widely used than C++

Post EMI Plans

Not an EMI product, will go on as it used to.

Summary

Features

* L&B is a monitoring tool: The Ys must be interpreted in the sense that L&B monitors that aspect, not that it performs it.

Platforms

Development

-- BjoernHagemeier - 04-Apr-2012