Access monitoring

Overview

The infrastructure monitoring team proposed two different ways in which storage providers could implement support for access monitoring. The proposal is accessible at http://wdtmon.web.cern.ch/wdtmon/dash/http.html

DPM (and xrootd of course) support the xrootd f-stream solution. dCache and StoRM will consider their options.

1) Formatted UDP stream

A properly formatted UDP stream, as implemented in the xrootd f-stream - http://xrootd.org/doc/dev4/xrd_monitoring.htm#_Toc346032371

2) Messaging + json

Access summaries conforming to a documented json schema delivered over the messaging system

A proposed schema is available at http://wdtmon.web.cern.ch/wdtmon/dash/http.html

The TF proposed the following amendments after consideration of the schema;

• A “final resolution” field which can indicate success or error code should be added.
• Metadata operations do not need to be logged, only I/O
• Transmission of application meta-data should be via the interface recently agreed between the FTS team and dCache, namely the “ClientInfo” HTTP header with an arbitrary string as the value. (https://its.cern.ch/jira/browse/FTS-245 for some more information).
• It would be beneficial if 3rd party copies could be explicitly identified. This could be done via the Application Specific Metadata to indicate who initiated the transfer. Note that this is unstructured data and any general consumption of this info would benefit from a format convention. This metadata is transmitted via a "ClientInfo" header.
• VO can be made mandatory but allowed to be NULL.
• Drop *_domain from the schema as the relation between [server|client]_domain and [server|client]_host is difficult to define usefully in a distributed system.
• VO, FQAN, role can all be multiple, but in the absence of a decision on how this should be handled the “first in the list" logic should be implemented for now.
• Some attributes may be subject to constraints on transmission outside a particular jurisdiction (e.g. sending the DN over the Atlantic). Such attributes are all optional, that's enough as far as the schema goes. Storage providers can make their publication configurable if they wish.

-- OliverKeeble - 2016-04-29