Improved usability of EMI products

Based on input from the PTs, this page lists some of the most important improvements on usability achieved for the individual products. For more details, please refer to the release notes of the individual products.

AMGA

 AMGA GUI client (AMGA Manager) ; general-purpose GUI toolkit for AMGA service.

AMGA Manager has many easy-to-use and intuitive features for improving usability and convenience: automatic query composition,   importing and exporting of metadata into a spread sheet, manipulating metadata schema (entries, ACL, index, attributes and others), managing user/group and filtered metadata searching with hiding complexities for accessing AMGA service.. AMGA Manager works on various heterogeneous platforms: Linux(32bit/64bit), Windows(32bit/64bit), and Mac.

Improved documentation includes detailed and up-to-date information on configuration variables (mandatory settings, default values, upgrade installation, scripts for bdii/emir and so on).

Federation can mount a remote directory to a local directory and redirects user's query to other multiple AMGA nodes and integrates the results.

New AMGA commands to improve usability

• Added version command which prints a version of the connected AMGA server
• Added "s" option with "ls" : "ls -s" command prints only sub-directories. * Added some AMGA commands to set mapping from VOMS Role/group to an AMGA user/group dynamically without modifying AMGA configuration file statically.

APEL

NOTE - As the APEL Client is not used by many users, in that the end users see the results of a site running the APEL Client in the EGI Accounting portal, we have based the following on the improved usability for the APEL system administrators.

• The latest version of the EMI APEL client now supports sites publishing Summaries rather than JobRecords which will speed up the publication process; an improvement for sites with a large volume of accounting records.

• It is now possible to summarise the data by CE as well which will provide better remote diagnostics for Sites.

• Comprehensive documentation is now available for the APEL client packages.

ARC

ARC core:

ARC Container (HED) code did not have any development tasks assigned directly. But because it provides base layer for other ARC code it evolved significantly according to their needs. ARC HED API was extended significantly to provide new features and support new use cases. Also for HED along with other ARC products (client and data libraries) extensive documentation was developed thus making it usable development environment for projects outside ARC.

ARC CE:

ARC CE system administration documentation was re-written from scratch to provide a comprehensive overview as well as a step-by-step guide to setting up the CE. The A-REX configuration reference was greatly improved, and the A-REX startup script now offers an option to validate the configuration. Integration of A-REX with ARGUS also provides simpler user authorisation management for ARC CE administrators. In addition the nordugridmap tool was completely re-written and is much more reliable and easier to use. The new CE-staging framework developed during EMI now provides much improved error handling and more intelligence in scheduling data transfer. The Jura accounting tool is now integrated with APEL, further integrating ARC into the EGI ecosystem. Several small improvements were made to the LRMS modules, notably Condor, making it easier for administrators to integrate ARC into their existing local setups. The addition of EMI-ES support to A-REX introduces a standards-compliant job management interface, opening up usability to a wider array of job management tools.

ARC clients:

The ARC Client components was significantly improved wrt. error handling and error reporting, and improved usage logic. Usability was also improved in terms of added features which among others includes use of the EMI CANL library, support for credentials stored using PKCS #11 and PKCS #12, integration with EMI data library, commands for creating directories and renaming files, improved GLUE2 support, support for resource discovery against EMIR, support for submitting jobs to EMI ES computing services, and the possibility to submit jobs directly to a computing service.

ARGUS

No particular usability improvements reported.

gLite infosys

No particular usability improvements reported.

CREAM

During the EMI project the CREAM service was evolved by enhancing the overall performance, scalability, fault tolerance and especially the usability which involves directly both users and the administrators. From the user's point of view the main benefit provided by the improved usability is due by a complete and simple to use Command Line Interface (CLI) which allows her to submit and control easily her jobs. Moreover CREAM can now provide to the user detailed information about her own jobs by paying particular attention on the reasons about the job failures. Another important achievement was the development of the EMI-ES specification which guarantees an high interoperability and integration level among the different middleware stacks supported by EMI. The administrator instead confides on an set of simplified and enhanced procedures concerning the service installation and configuration. The logging level was enriched by new technical messages very useful for monitoring the service's status and for debugging any failures may occur during its life cycle. Moreover the integration with ARGUS uniforms and makes easier the administration of the user authorization system on local resources.

Finally the CREAM documentation was improved and now provides detailed product information by covering in deep aspects concerning the service capabilities as well as the installation and the manual/automatic (i.e. YAIM) configuration. In particular the service's settings, the list of configuration parameters, their means including the default values were fully described. In addition new specific topics were included for completeness as the migration of a CREAM service to another physical host, the known issues and the troubleshooting hints.

gLite MPI

The major usability improvement of mpi-start is the new command line arguments that allow using the product directly as main executable in a job submission and its integration with the EMI-ES ParallelEnvironment. These command line arguments removed the need of declaring environment variables to control the behavior of the software.  Other usability improvements include: a new man page and general update of documentation; an enhanced configuration module  that allows both user- and system-level configuration; better logging for easier tracing of problems; and improvements in packaging (FHS compliance and Debian support).  

gLite Security

Hydra

Product Hydra did not do anything with regards to improved usability as no user requests were received for such changes.

Pseudonymity

Product Pseudonymity did not do anything with regards to improved usability as no user requests were received for such changes.

STS

Product STS did not do anything with regards to improved usability as STS is initial version so no changes practical.

WNoDeS

The WNoDeS build system were re-structured to make binary RPMs from source RPMS. The whole WNoDeS documentations were improved: in particualr the system guide describes the meaning of the configuration variables, each WNoDeS service, and how to manage the deployment of the Cloud part on top of the WNoDeS core. The Cloud server was migrated to use the EMI Common authentication library (CANL).

dCache

No particular usability improvements reported.

CERN Data

No particular usability improvements reported.

StoRM

No particular usability improvements reported.

Cesnet Security

The site access control components (LCMAPS, gLExec) were restructured to comply with the security logging policy and add differentiated logging levels in accordance with these guidelines. This allows system administrators easier integration with centralized logging infrastructure and ensures easier traceability for access control events. Packaging was improved to fully comply with standard OS distribution guidelines to ease installation. The addition of supporting tools for gLExec improves end-user adoption of credential switching methods by automating common user-level tasks used in sandboxing applications.

GridSite

Being mainly a library with little direct user or administrator interaction, GridSite did not provide many opportunities for improving usability. Error reporting has been improved by adopting caNl, the command-line suite was simplified by removing rarely used commands, and several small improvements (pkg-config, C++ compatible headers) were implemented to make the product friendlier to developers.

ProxyRenewal

Being a service that is normally neither contacted directly by users, nor configured by administrators, coupled with the fact that the product was only maintained and not actively developed during EMI, ProxyRenewal offered little opportunity for improving usability. Its related component, the myproxy-yaim configuration package, has been extended with several features requested by administrators to ease selected configuration tasks.

gsoap/gss

Being a library with no direct user or administrator interaction, this product provided little opportunities for improving usability. Extensions were implemented to allow sites to set-up authentication with Kerberos rather than x509 certificates if required by their respective user communities.

L&B

In gLite's Logging and Bookkeeping, there was major progress towards simplifying access to information. Many types of information originally accessible only through the L&B API, requiring the use of the L&B client, were made available through popular and widely used protocols. These include: the full power of the Job querying API can now be used with Query Strings included in HTML requests, so that even complex queries with many diverse conditions can be run from an unmodified Web Browser; L&B notifications can be delivered through any STOMP/OpenWire-based messaging broker, avoiding the need to use L&B's native notification client; L&B notification registrations can be set-up as permanent by site administrators, so that the L&B notification client is no longer needed to make and refresh the registrations; L&B's existing RSS interface has been extended to allow users to set-up custom RSS feeds with powerful querying capabilities; identity mapping feature based on Globus gridmap files was implemented to allow for aggregation of data for users working with multiple grid identities; custom HTML headers can be inserted into L&B's HTML output, allowing site administrators to pre-format queries typical for their users' domain and make them easily available as a part of L&B's HTML interface; access control configuration was improved to simplify assignment of roles; L&B server configuration was made accessible to site administrators remotely through the HTML interface; a configuration script compatible with present-time SiteInfo.def files used by YAIM has been prepared to allow for smooth transition to the post-EMI era where YAIM won't be available.

VOMS

No particular usability improvements reported.

SAGA-SD-RAL

No particular usability improvements reported.

UNICORE

UNICORE UCC

The UNICORE commandline client (UCC) was migrated to use the EMI Common authentication library (CANL). This improves usability by adding support for additional types of credentials (such as OpenSSL style PEM files) and Globus-style truststores. Certificate related error reporting was greatly extended so users can obtain a very detailed and human friendly description of the problems found. To simplify access to the new features offered by CANL, the online help includes information on all CANL options. The output of the help function of UCC was re-structured to make it easier to read.

Many changes were made that directly increase the usability of UCC:

• some commands can now operate on multiple arguments, such as the job submission and management fuctions. For example, this can be used to submit multiple jobs with a single UCC run

• data movement commands have been adapted to be more UNIX-like and thus familiar to most users

• failed data transfers are re-tried using the default protocol

• data movement operations show progress in a fashion similar to the familiar UNIX 'scp' command

• the informational commands such as 'list-sites' or 'system-info' include more useful information

• the 'shell' mode was improved by a better commandline completion and the possibility to set and unset variables

• the end-user documentation was improved

• a new feature allows for full file permissions and ACL handling

UNICORE HiLA

HiLA is now available as RPM/DEB package, making installation and operation easier. HiLA was migrated to use CANL, offering more keystore/truststore formats and improvements related to CRL handling (such as configuration of CRLs using wildcard expressions) and proxy certificate validation. Certificate related error reporting was greatly extended so users can obtain a very detailed and human friendly description of the problems found. HiLA has been extended to support an abstract job model with pluggable mappings. So far, mappings for JSDL and EMI-ADL have been defined, allowing for JSDL job descriptions to be used as client side input for the EMI-ES submission interface. Conversely, EMI-ADL descriptions can be used to submit jobs to JSDL based submission interfaces by means of the mapping implementations. The abstract job submission model has been equipped with a convenient programming interface to facilitate the easy creation of job submissions in code.

UNICORE UNICORE/X

The UNICORE/X server is now available as RPM/DEB package, making installation and operation easier. UNICORE/X was migrated to use CANL, offering more keystore/truststore formats and improvements related to CRL handling (such as configuration of CRLs using wildcard expressions) and proxy certificate validation. Certificate related error reporting was greatly extended so users can obtain a very detailed and human friendly description of the problems found. The documentation was improved and includes detailed and up-to-date information on configuration variables (mandatory settings, default values). There is more information logged, the log levels are unified across all products and the information being logged is more user (administrator) oriented. A new feature allows for full file permissions and ACL handling

UNICORE TSI

The TSI is now provided as RPM/DEB package, making installation and operation easier.

UNICORE Registry

The Registry server is now provided as RPM/DEB package, making installation and operation easier. The Registry was migrated to use CANL, offering more keystore/truststore formats and improvements related to CRL handling (such as configuration of CRLs using wildcard expressions) and proxy certificate validation. Certificate related error reporting was greatly extended so users can obtain a very detailed and human friendly description of the problems found. The documentation was improved and includes detailed and up-to-date information on configuration variables (mandatory settings, default values). There is more information logged, the log levels are unified across all products and the information being logged is more user (administrator) oriented.

UNICORE Gateway

The Gateway server is now available as RPM/DEB package, making installation and operation easier. The Gateway was migrated to use CANL, offering more keystore/truststore formats and improvements related to CRL handling (such as configuration of CRLs using wildcard expressions) and proxy certificate validation. Certificate related error reporting was greatly extended so users can obtain a very detailed and human friendly description of the problems found. The documentation was improved and includes detailed and up-to-date information on configuration variables (mandatory settings, default values). There is more information logged, the log levels are unified across all products and the information being logged is more user (administrator) oriented.

UNICORE XUUDB

The XUUDB server is now available as RPM/DEB package, making installation and operation easier. The XUUDB was migrated to use CANL, offering more keystore/truststore formats and improvements related to CRL handling (such as configuration of CRLs using wildcard expressions) and proxy certificate validation. Certificate related error reporting was greatly extended so users can obtain a very detailed and human friendly description of the problems found. The documentation was improved and includes detailed and up-to-date information on configuration variables (mandatory settings, default values). There is more information logged, the log levels are unified across all products and the information being logged is more user (administrator) oriented.

UNICORE UVOS

The UVOS components are now available as RPM/DEB packages, making installation and operation easier. UVOS was migrated to use CANL, offering more keystore/truststore formats and improvements related to CRL handling (such as configuration of CRLs using wildcard expressions) and proxy certificate validation. Certificate related error reporting was greatly extended so users can obtain a very detailed and human friendly description of the problems found. The documentation was improved and includes detailed and up-to-date information on configuration variables (mandatory settings, default values). There is more information logged, the log levels are unified across all products and the information being logged is more user (administrator) oriented. Configuration options related to HTTP and security of UVOS server and client are the same as the options used in other UNICORE components.

EMIR

In order to fulfill the goal of having a common service registry to index all of the EMI products, the EMIR - a federated service registry, has been realised. It mainly consists of two parts: server and client. The registry server is based on a state of the art REST Web services with simplified (JSON) and standardized GLUE 2.0 information model with scalable NoSQL database in the backend. To ease the integration of EMIR with all of the EMI services, a general purpose client-side publisher (EMIR-SERP) utility and a Java API has been implemented. These clients played a vital role in rolling-out EMIR within the EMI project, hence the major Grid technologies, such as, gLite, UNICORE, ARC, and dCache have used EMIR to publish their services.

Several improvements have been made during the integration and testing phases, that were either due to the feedback from other product teams or internal testing. It covers following important aspects of EMIR,

• Improved error reporting (and logging) on server and client side,
• Cleaner and fine grained (service discovery) REST API with support for rich querying, registration, status, and immediate ancestor/successor registry node information
• On the client publisher's (EMIR-SERP) side, the implementation was extended to support publishing from BDII
• Adherence to the OGF’s GLUE 2.0 model by accepting registrations in XML as well as in JSON format
• Integration with CANL includes support for more keystore/truststore formats and improvements related to CRL handling (such as configuration of CRLs using wildcard expressions) and proxy certificate validation.
• Both client and server components of EMIR have strictly complied with the EMI packaging policies while enabling the platform specific distributions for Scientific Linux and Debian

Documentation: During its inception a technical document explaining the architecture and design was initially provided. Thereafter a guide with detailed description regarding the installation, configuration, and data model for every subsequent release has been made available.

EMI Messaging

No particular usability improvements reported.

EMI Common

No particular usability improvements reported.

EMI CaNL

As a product that did not exist before EMI, improvement in usability is difficult to judge. However, caNl helped to improve the usability of other products that rely on it by introducing common procedures and common, mutually compatible error reporting between different products implemented in different programming languages.

WMS

• totally restructured the build system, reduced number of binaries, easier for admins to deal with.
• improved doc, properly describing the meaning of each variable, mandatory variables and so on.
• migration to argus-based authZ, easier for site deployment.
• FHS compliance: easier to retrieve logs, configuration, start services etc.