GridView message Formats and implementation

Introduction

This page describes how to use the Messaging System for the Grid in order to publish records into GridView DB. At this moment, we will focus on the GridFTP use case.

Topic definition

Topics distinguish the general type of information we should receive if we subscribe. Topics are defined as follows: <locality>.<informationType>[.<relatedTo>]

• Locality allows for a first big filter on the scope for which the information is relevant. Possible values are: {local; site; regional; grid}
  • local - messages are meaningful within a local scope
  • site - messages are meaningful within the site scope
  • regional - messages are meaningful at the ROC level
  • grid - messages are intended to be consumed by anyone within the grid.
• InformationType refers to the type of data coming from the source. Each specify their own language: messages sent to this topic should share common fields that relate to the type. For the moment the following are defined: {probe; usage; summary}
  • probe - messages are defined in GridMonitoringProbeSpecification. They are originated on probes that send metrics about particular services.
  • usage - messages refer to information about the usage of a particular service. [ex: detailed information on transfers by gridFTP]
  • summary - messages which send summarized information. [ex: Dashboard]
• RelatedTo depends on the specified informationType. If it's a probe information, we will be listening/publishing to either metricOutput or metricDescription. If it is usage, we might define subsets such as {dataTransfer, jobDetails, service}

For GridFTP in the GridView context, topic is defined as /topic/grid.usage.transfer

Message Formats

Message formats should be according to the described at GridMonitoringProbeSpecification#Message_Formats. That is, key: value pairs, where keys depend on the Message Class definition.

In addition, each message class defines the number of keys that may exist, as well as which of the keys are mandatory.

org.wlcg.usage.transfer Message Class v0.1

For any data transfer the following fields are considered mandatory in order to be understandable by a consumer: {VO, fileName; startTime; endTime; srcHost; destHost; transferProtocol; numberBytes;} This is part of org.wlcg.usage.transfer definition. depending on the transfer protocol, additional fields may be defined. For instance we might additionally have: {srcService; destService; tcpBufferSize; gridftpStreams;}

• Mandatory fields definition:
  • VOName - name of the VO where the transfer occured.
  • fileName - transferred filename
  • startTime - timestamp of the transfer start
  • endTime - timestamp of the transfer end
  • srcHost - the source host, where the transfer originated
  • destHost - the destination host for the transfered file
  • transferProtocol - the transfer protocol in use ( eg. GridFTP )
  • numberBytes - the number of bytes of the transfered file
• GridFTP specific fields:
  • publishingHost - the name of the host who published the particular log
  • gridftpStreams - number of streams used in the transfer.
• Other fields:
  • srcService - the service URI where the transfer originated
  • destService - the service URI of the transfer destination
  • tcpBufferSize

Each message may contain multiple records, but referring to a single transferProtocol. Besides being included in each record, transferProtocol should be placed in the message header to allow for implementation of selectors by the consumer. There must be no blank line between EOT and the next message.

Here's an example of records that may be sent on a single message:

transferProtocol: GridFTP
voName: biomed
fileName: /storage/biomed/ligands/amylase/7318512_1.pdbq
startTime: 13-02-2008T09:47:41.001Z
endTime: 13-02-2008T09:47:41.234Z
srcHost: clrauvergridse01.in2p3.fr
destHost: 212.191.227.131
numberBytes: 2925
gridftpStreams: 1 
EOT

Using MSG publisher

First step, a configuration for the probe should be in place. The RPM is called msg-publish-simple and is available in the GridMonitoringYumRepository. An example template is provided in the package. This template should be put in place with no other changes but, eventually, the endpoints to be used.

   [publisher]
   endpoints=http://gridmsg001.cern.ch/message stomp://gridmsg001.cern.ch:6163
   #leave untouched. 
   messageClasses=org.wlcg.usageTransfer

   org.wlcg.usageTransfer_reqattrs=VO fileName startTime endTime srcHost destHost numberBytes gridftpStreams
   org.wlcg.usageTransfer_headerattrs=transferProtocol
   org.wlcg.usageTransfer_destination=/topic/grid.usage.transfer

Second step is sending the messages. For such purpose, you may use the command line tool msg-publish, under /opt/lcg/bin:

  Usage: /opt/lcg/bin/msg-publish org.wlcg.usageTransfer [FILE1] [FILE2...]
  Where [File1][File2]...[Filex] contain the messages in the defined format.

Each file should contain records, EOT separated, according to the format defined above. TransferProtocol should be the same for each different message, and besides stated in the records, sent as an header, in order to enable selectors if necessary. There are some options that mimic the configuration file, except for -b(bulk publishing), which we will overlook for the moment.

MSG consumer, Database Views and Triggers

The consumer instance is responsible for simply calling the adequate view on the Oracle Database. It is provided in an rpm msg-consume2oracle from the GridMonitoringYumRepository. Configuration file is called /opt/lcg/etc/msg/msg-consume2oracle.conf folder, and should contain

org.wlcg.usageTransfer_dbattrs=serviceType metricName metricStatus performanceData summaryData detailsData voName serviceURI timestamp
org.wlcg.usageTransfer_dbtable=v_usageTransfer
org.wlcg.usageTransfer_destination=/topic/grid.usage.transfer

Please see the template configuration file contained in the rpm package, under /opt/lcg/etc/msg/msg-consume2oracle.conf.example.

Inside the database, a view "V_USAGETRANSFER" and additional triggers on insert will need to be in place adequately. The MSG consumer will call the view according to what is defined in the configuration, and will be checking if all the required message fields are in place.

An example schema, in which transferprotocols names/ids are stored in a separated table, could be :

------------------------------
-- Message Class corresponding view
------------------------------
CREATE OR REPLACE VIEW V_USAGETRANSFER ("NUMBERBYTES", "SRCHOST", "DESTHOST", "TRANSFERPROTOCOL", "FILENAME", "VONAME", "STARTTIME", "GRIDFTPSTREAMS", "ENDTIME") AS 
  SELECT  NUMBERBYTES, SRCHOST, DESTHOST, TRANSFERPROTOCOL, FILENAME, VONAME, STARTTIME, GRIDFTPSTREAMS, ENDTIME FROM USAGETRANSFER;

------------------------------
-- TransferProtocols table definition.
------------------------------
DROP TABLE TRANSFERPROTOCOLS;
CREATE TABLE TRANSFERPROTOCOLS (
       protocolID INTEGER NOT NULL,
       protocolName VARCHAR2(100) NOT NULL,
       isDeleted char(1) DEFAULT 'N' NOT NULL 
); 

ALTER TABLE TRANSFERPROTOCOLS ADD CONSTRAINT TrsPrt_pk PRIMARY KEY (protocolID);
ALTER TABLE TRANSFERPROTOCOLS ADD CONSTRAINT TrsPrt_protocolName_ix UNIQUE (protocolName);
ALTER TABLE TRANSFERPROTOCOLS ADD CONSTRAINT TrsPrt_isDeleted CHECK (isDeleted IN ('Y','N'));

create sequence trsPrt_seq start with 1 nomaxvalue nocycle cache 256;

------------------------------
-- USAGEDATATRANSFER table definition
------------------------------
DROP TABLE USAGEDATATRANSFER;
CREATE TABLE USAGEDATATRANSFER( 
   NUMBERBYTES NUMBER NOT NULL ENABLE, 
   SRCHOST VARCHAR2(1000 BYTE) NOT NULL ENABLE, 
   DESTHOST VARCHAR2(1000 BYTE) NOT NULL ENABLE, 
   FILENAME VARCHAR2(1000 BYTE) NOT NULL ENABLE, 
   VONAME VARCHAR2(100 BYTE) NOT NULL ENABLE, 
   STARTTIME VARCHAR2(100 BYTE) NOT NULL ENABLE, 
   ENDTIME VARCHAR2(100 BYTE) NOT NULL ENABLE, 
   GRIDFTPSTREAMS NUMBER NOT NULL ENABLE,

   transferProtocolID NUMBER NOT NULL ENABLE, 
        CONSTRAINT USAGEDATATRSF_TRSFPROTOCOLS_FK FOREIGN KEY ( transferProtocolID)
          REFERENCES TRANSFERPROTOCOLS( protocolID ) ENABLE
);
         
------------------------------
-- Trigger definition, inserting protocolName if it doesn't exist ( likely we should prefer an exception to be raised, but this is just an example )
------------------------------

create or replace TRIGGER V_USGTRF_INSERT_TRG instead of insert on V_USAGETRANSFER
declare
  protocol varchar(100);
  protocolid number;
begin
  protocol := lower( :new.transferprotocol );
  begin 
    select protocolID into protocolid
      from TRANSFERPROTOCOLS
      where protocolName=protocol;
  exception
    when no_data_found then
      INSERT INTO TRANSFERPROTOCOLS (protocolId, protocolName) values ( 
          trsPrt_seq.nextval, protocol ) ;
      SELECT protocolID into protocolid 
        from TRANSFERPROTOCOLS
        where protocolName = protocol;
  end;     
  
  INSERT INTO USAGEDATATRANSFER (NUMBERBYTES, SRCHOST, DESTHOST, transferProtocolID, FILENAME,
 VONAME, STARTTIME, ENDTIME, GRIDFTPSTREAMS)
 values (:new.numberbytes, :new.srchost, :new.desthost, protocolid, :new.filename, :new.voname, :new.starttime, :new.endtime, :new.gridftpstreams);
end;

org.wlcg.usage.jobStatus Message Class v0.1

For any data transfer the following fields are considered mandatory in order to be understandable by a consumer: {jobId, ownerDN, voname, bkHost, networkHost, lastUpdateTime, stateName, stateStartTime, exitCode, doneCode} This is part of org.wlcg.usage.jobStatus definition. depending on the transfer protocol, additional fields may be defined. For instance we might additionally have: {condorId;}

• Mandatory fields definition:
  • jobId - Jobid of the job as assigned by the LB server
  • ownerDN - Owner of the job
  • voname - VO to which the job belongs
  • bkHost - L&B server name with port
  • networkHost - Resource Broker of the job
  • lastUpdateTime - Last update time of this job for the may change for a state and stateStartTime
  • stateName - State of the job
  • stateStartTime -
  • exitCode - Exit code of the Job
  • doneCode - Done Code of the job. Meaningful for the statename = "Done" [Showing successful completion of failed completion]
    
• Other fields:
  • condorId: CondorID of the job
  • destSite: Destination Site URL (see the format) of the site [where the job is scheduled. not meaningful in case of statename "submitted","Ready", " Waiting"]
  • statusreason: Reason for the state changes. and description.

Each message may contain multiple records. There must be no blank line between EOT and the next message.

Here's an example of records that may be sent on a single message:

JOBID:https://rb115.cern.ch:9000/yx-VWCYTaGrJBscftHT8hw
OWNERDN:/DC=ch/DC=cern/OU=OrganicUnits/OU=Users/CN=asciaba/CN=430796/CN=Andrea Sciaba
BKHOST:rb115.cern.ch:9000
NETWORKHOST:rb115.cern.ch:7772
VONAME:cms
LASTUPDATETIME:1202439701
STATENAME:Waiting
STATEENTERTIME:1202439701
CONDORID:asdg
DESTINATION:http://lxb6118.cern.ch:80/JobDestinations1
EXITCODE:0
DONECODE:0
STATUSREASON:0
EOT