Gridview XML Interface

Accessing GridFTP transfer data

Gridview uses data transfer logs of GridFTP servers to calculate the transfer rates between the sites. These values are calculated by dividing the total data transfered in a particular time span by time This calculated data is used by Gridview front-end for visualisation and other reporting tools for monthly reports generation. Developers and users of other tools and interfaces had requested for a programmatic interface to fetch and display this data in their applications, reports etc. In response to these requests a programmatic interface is developed which provide this data in XML.

Required data can be queried from the interface by setting various options in the URL specified. These options are passed to the server using HTTP GET method. Based on these options the result set is provided in XML.

At present this interface supports

• Avg Throughput from/to site(s)
• Avg Throughput from/to site(s) per VO distribution
• Avg Throughput from/to sites(s) per site Distribution
• Selection of source and destsite on tier and Region.

Programmatic interface accepts URL as an input. Here is the list of options that can be provided in the URL.

Option Name	Description
VO_name, VO_id	Can be specified as an array of comma separated list of Name/ID of VOs interested. When specified only transfers corresponding to these VOs are considered. If more than one VO is specified then VO tag will appear in the o/p
Src_Site_name, Src_Site_id	Can be specified as an array of comma separated list of Name/ID of Source Sites interested.
Dest_Site_name, Dest_Site_id	Can be specified as an array of comma separated list of Name/ID of Destination Sites interested.
start_time	Start time of the period interested. If not specified it is taken as current time of previous day. The timestamp should be provided in the format 2007-05-06T21:00:00Z'
end_time	End time of the period interested. If not specified it is taken as current time. The timestamp should be provided in the format same as start_time
summary_period	Periodicity of summary interested. It can be 'hourly', 'daily', 'weekly', 'monthly'.
value_fields	Comma separated list of fields required. It depends on the module being queried. Currently it only accepts throughput
heirarchial_order	Order of Hierarchy in the XML output. It is comma separated list of fields in required order.
Src_Region_name	Name of the source region requested. When this option is specified then all source sites of the region is provided in o/p.
Dest_Region_name	Name of the destination region requested. When this option is specified then all destination sites of the region is provided in o/p.
Src_Tier/Dest_Tier	Its value can be 1, 2 or 3. This is to query all sites of a tier.

Remember xxX_Tier and xxx_Region_name option cannot be specified together. When Region_name or Tier option are specified then the Site_name/Site_id fields are overridden by Region_name and Tier.

Here is the URL

http://gridview.cern.ch/GRIDVIEW/pi/xml/gridftp-xml.php

Example URL

Following URL provides data for weekly throughput from CERN to INFN-ROMA1 under the VO Atlas and CMS Tier 1 sites from 2008-09-01T01:00:00Z to 2008-10-10T00:00:00

http://gridview.cern.ch/GRIDVIEW/pi/xml/gridftp-xml.php?Src_Site_name=CERN-PROD&Dest_Site_name=INFN-ROMA1&VO_name=atlas,cms&summary_period=weekly&start_time=2008-09-01T00:00:00Z&end_time=2008-10-10T00:00:00Z

Accessing SAM data

Gridview uses Test results collected by SAM to calculate availability and reliability of Sites. These values are calculated as per the approved algorithm. This calculated data is used by Gridview front-end for visualisation and other reporting tools for monthly reports generation. Developers and users of other tools and interfaces had requested for a programmatic interface to fetch and display this data in their applications, reports etc. In response to these requests a programmatic interface is developed which provide this data in XML.

Required data can be queried from the interface by setting various options in the URL specified. These options are passed to the server using HTTP GET method. Based on these options the result set is provided in XML.

At present this interface supports

• Overall site Availability/Reliability
• Service Availability/Reliability of Sites.
• Service Availability/Reliability of Sites in a Region.
• Service Availability/Reliability of Sites in a Tier
• Hourly and Daily Availability/Reliability of sites.

Programmatic interface accepts URL as an input. Here is the list of options that can be provided in the URL. Field names that are common to SAM XML Interface are kept same.

Option Name	Description
VO_name, VO_id	Can be specified as an array of comma separated list of Name/ID of VOs interested. This specifies the defining VO used for calculation of availability. If multiple VOs are specified then values for all the VOs are provided in the o/p. If more than one VO is specified then VO tag will appear in the o/p
Site_name, Site_id	Can be specified as an array of comma separated list of Name/ID of Sites interested.
start_time	Start time of the period interested. If not specified it is taken as current time of previous day. The timestamp should be provided in the format 2007-05-06T21:00:00Z'
end_time	End time of the period interested. If not specified it is taken as current time. The timestamp should be provided in the format same as start_time
summary_period	Periodicity of summary interested. It can be 'hourly', 'daily', 'weekly', 'monthly'.
value_fields	Comma separated list of fields required. It depends on the module being queried. For SAM it can be availability, reliability, maintenance, unknown, down, aup, adown, aunknown, amaintenance
heirarchial_order	Order of Hierarchy in the XML output. It is comma separated list of fields in required order.
Service_name	Name of the service requested. It can be 'All' to request all services. If nothing is given then overall Site figures are provided.
Region_name	Name of the region requested. When this option is specified then all sites of the region is provided in o/p.
Tier	Its value can be 1, 2 or 3. This is to query all sites of a tier.

Remember Tier and Region_name option cannot be specified together. When Region_name or Tier option are specified then the Site_name/Site_id fields are overridden by Region_name and Tier.

Here is the URL

http://gridview.cern.ch/GRIDVIEW/pi/xml/sam-xml.php

Example URL

Following URL provides data for daily availability, reliability, maintenance, unknown and down for service CE for all Tier 1 sites from 2008-09-01T01:00:00Z to 2008-09-05T02:00:00 using CMS as the defining VO.

http://gridview.cern.ch/GRIDVIEW/pi/xml/sam-xml.php?VO_name[]=CMS&summary_period=daily&heirarchial_order=timestamp,site&value_fields=availability,reliability,unknown,maintenance,down&Site_id[]=1000046&Service_name=ce&start_time=2008-09-01T01:00:00Z&end_time=2008-09-05T02:00:00Z&Tier=1

A Sample output is

<?xml version="1.0" encoding="utf-8" ?>
<GridviewDocument>
  <GridviewReportHeader>
    Gridview daily Report for the Period 2008-09-01T01:00:00Z - 2008-09-05T02:00:00Z
  </GridviewReportHeader>
  <GridviewOutput>
    <Timestamp epoch="1220313600" UTC="2008-09-02T00:00:00Z" count="0">
      <Site id="1" name="RAL-LCG2" tier="1" region="UKI" count="0">
        <service id="1" name="CE" count="0">
          <availability> 1 </availability>
      <reliability> 1 </reliability>
          <unknown> 0 </unknown>
          <maintenance> 0 </maintenance>
        </service>
      </Site>
      <Site id="13" name="INFN-T1" tier="1" region="Italy" count="1">
        <service id="1" name="CE" count="0">
          <availability> 1 </availability>
          <reliability> 1 </reliability>
          <unknown> 0 </unknown>
          <maintenance> 0 </maintenance>
        </service>
      </Site>
      <Site id="17" name="FZK-LCG2" tier="1" region="GermanySwitzerland" count="2">
        <service id="1" name="CE" count="0">
          <availability> 1 </availability>
          <reliability> 1 </reliability>
          <unknown> 0 </unknown>
          <maintenance> 0 </maintenance>
        </service>
      </Site>
   .
   .
   .

    </Timestamp>
  </GridviewOutput>
</GridviewDocument>

Salient features of XML output of the Gridview data are:

• Site tag is inside timestamp tag. This is what is meant by hierarchical order. You can change this order by setting the ‘heirarchial_order’ input parameter.
• All tag names are in small case
• Timestamp is provided in epoch as well as UTC
• Name and ID of VO is specified in VO tag as attribute.
• Only the ‘value_field’ elements are end elements.
• Count tag is symbolic it is just a counter to elements within a tag.
• Root tag of the document is GridviewDocument.
• Inside the root tag possible tags are: output ( GridviewOutput ), header ( GridviewReportHeader), Error ( GridviewError )
• This gives separate tags for different part of the document and makes it more comprehensive.

-- KislayBhatt - 24 Feb 2009