Quattor installation of Gridview

Gridview Related Links:

1. Gridview Twiki Page
2. Gridview Service Dash Board Notes
3. Gridview Deployment at CERN
4. Gridview Admin Guide
5. Quattor Installation of Gridview
6. Gridview Software Release Status
7. Installation and Configuration of Gridview Publisher
8. Installation and Configuration of Gridview Web Service Clients
9. Using Gridview XML Interface
10. URL for Excel Report generation
11. Downloads
12. New Gridview Interface
13. Gridview Monthly Availabilities and Reports

Contents

Gridview can be installed and configured on production systems using Quattor. For this the following are available:

1. Set of quattor templates for gridview software
2. Gridview NCM component for configuring gridview
3. Sindes script for retrieving and delivering sensitive information to gridview machines

Quattor templates

The list of quattor templates for gridview are:

• pro_declaration_component_gridview.tpl : Declarations for Gridview specific CDB data structures
• pro_software_component_gridview.tpl : Template containing configurations, common to all gridview instances. This template alone should be included in a machine's template after activating the required gridview modules. This template will in turn include other required templates.
• pro_software_component_gridview_frontend.tpl : Template configuring Gridview frontend instances.
• pro_software_component_gridview_summarizer.tpl : Template configuring Gridview summarizer instances.
• pro_software_component_gridview_synchronizer.tpl : Template configuring Gridview synchronizer instances.
• pro_software_component_gridview_archiver.tpl : Template to be configuring Gridview archiver instances.
• pro_component_httpd_defaults_gridview.tpl : Apache specific settings for gridview frontend
• pro_component_tomcat_defaults_gridview.tpl : Tomcat specific settings for gridview archivers.
• pro_component_php_defaults_gridview.tpl: PHP settings for gridview.

Individual templates and their usage

pro_software_component_gridview

This template is the master template to be included in the gridview machine's template. It contains

• Addition of common software packages such as Oracle instantclient and logrotate.
• Configuration of logrotate service for logs in /var/log/gridview directory
• Configuration of Sindes service for retrieving secure information
• Conditional inclusion of other gridview templates based on which module is to be installed in the system

pro_software_component_gridview_frontend

This template should be included in those machines which are to be configured as Gridview frontends. It contains

• Addition of necessary packages (httpd and php) and their configurations
• Configuration of logrotate service for httpd logs
• Gridview frontend specific configurations

Paths to be set in the machine template to activate gridview frontend:

"/software/components/gridview/frontend/active" = true;

Pre dependencies

• spma
• sindes
• php
• httpd

Post dependencies

• altlogrotate

Files modified/created:

• /etc/httpd/conf/httpd.conf
• /etc/httpd/conf.d/ncm-ssl.conf
• /etc/httpd/conf.d/ncm-gridview.conf
• /etc/php.ini
• /etc/logrotate.d/httpd.ncm-altlogrotate
• /opt/gridview/etc/gridview-common.conf

pro_software_component_gridview_summarizer

This template should be included in the template of the machines which are to be configured as a gridview summarizer. The template contains

• Inclusion of necessary packages (php and cron)
• Configuration of PHP
• Addition of specific CRON jobs depending on which of the different summarizers are to be activated on this machine.

Paths to be set in the machine template to activate individual gridview summarizers:

"/software/components/gridview/summarizer/sum_dt/active" = true;         # For activating data transfer summarizer
"/software/components/gridview/summarizer/sum_js/active" = true;         # For activating job summarizer
"/software/components/gridview/summarizer/sum_sa/active" = true;        # For activating service availability summarizer

One or more of the three Gridview summarizers can be activated on a summarizer machine. It should be noted that the activation statements should be entered before including the template.

Pre-dependencies

• spma
• sindes
• php

Post-dependencies

• cron
• altlogrotate

Files modified/created:

• /etc/php.ini
• /etc/cron.d/ncm-gridview-summarizer*
• /etc/logrotate.d/gridview.ncm-altlogrotate

pro_software_component_gridview_synchronizer

This template should be included in the template of the machine which are to be configured as gridview synchronizers. The template contains

• Inclusion of necessary packages (php and cron)
• Configuration of PHP
• Addition of specific CRON jobs depending on which of the different summarizers are to be activated on this machine.
• Synchonizer specific configurations

Paths to be set in the machine template to activate gridview synchronizers:

"/software/components/gridview/synchronizer/gocdb/active" = true;         # For activating GOCDB synchronizer
"/software/components/gridview/synchronizer/cicdb/active" = true;         # For activating CICDB synchronizer

One or more of the two Gridview synchronizers can be activated on a synchronizers machine. It should be noted that the activation statements should be entered before including the template.

Pre dependencies

• spma
• sindes
• php

Post dependencies

• cron
• altlogrotate

Files modified/created:

• /etc/php.ini
• /etc/cron.d/ncm-gridview-synchronizer*
• /etc/logrotate.d/gridview.ncm-altlogrotate
• /opt/gridview/etc/gridview-common.conf

pro_software_component_gridview_archiver

This template should be included in the template of those machines on which Gridview archivers are to be configured. The template contains

• inclusion of necessary packages (java, tomcat, rgmaclient) and their configurations
• Configuration of grid_service_certificates component to copy certificates to /opt/gridview/certificates
• Archiver specific configurations
• Enabling rgma-archiver services through init.d subsystem if rgma archiver is enabled
• Enabling tomcat service if web service archiver is enabled

Paths to be set in the machine template to activate gridview archivers:

"/software/components/gridview/archiver/rgma/arch_dt/active" = true;    # For activating rgma data transfer archiver
"/software/components/gridview/archiver/rgma/arch_js/active" = true;    # For activating rgma job archiver
"/software/components/gridview/archiver/wsarch/arch_dt/active" = true; # For activating web service data transfer archiver
"/software/components/gridview/archiver/wsarch/arch_js/active" = true; # For activating web service job archiver

One or more of the different Gridview archivers can be activated on a archiver machine. It should be noted that the activation statements should be entered before including the template.

Pre-dependencies

• spma
• sindes
• rgmaclient # For R-GMA archivers
• grid_service_certificates # For R-GMA archivers
• tomcat # For web service archivers

Post dependencies

• altlogrotate
• chkconfig

Files modified/created:

• /etc/tomcat5/tomcat5.conf
• /etc/tomcat5/server.xml
• /etc/tomcat5/tomcat-users.xml
• /opt/gridview/gridview-common.conf
• /opt/gridview/rgma-authentication.conf
• /opt/gridview/rgmaarch.conf
• /opt/gridview/wsarch.conf
• /etc/logrotate.d/gridview.ncm-altlogrotate

Secure Information Delivery through Sindes and the Sindes Gridview script

The gridview application needs to be configured with secure information which cannot be embedded inside rpms, CVS or quattor templates. This information includes:

• Gridview Oracle DB usernames and passwords
• GOCDB Oracle DB usernames and passwords
• CIC Oracle DB usernames and passwords

This information is kept in the CERN Sindes service and delivered to gridview machines. The secure information file in the 'gridview' item in SINDES is copied into /opt/gridview/etc/sconfig file by Sindes and the SINDES-script-gridview script. This file is further used by Gridview NCM Component to generate the application's configuration files.

Contents of the SINDES gridview component

File

A file that specifies DB connection parameters for the Gridview DB and the external GOCDB and CIC DBs. The syntax of this file is as below:

@COMMON_DB_READER_HOSTSPEC@=
@COMMON_DB_READER_USERNAME@=
@COMMON_DB_READER_PASSWORD@=
@COMMON_DB_READER_DATABASE@=
@COMMON_DB_READER_OPTIONS@=
@COMMON_DB_READER_PORT@=
@COMMON_DB_READER_PROTOCOL@=

@COMMON_DB_WRITER_HOSTSPEC@=
@COMMON_DB_WRITER_USERNAME@=
@COMMON_DB_WRITER_PASSWORD@=
@COMMON_DB_WRITER_DATABASE@=
@COMMON_DB_WRITER_OPTIONS@=
@COMMON_DB_WRITER_PORT@=
@COMMON_DB_WRITER_PROTOCOL@=

@SYNCHRONIZER_GOCDB_DB_HOSTSPEC@=
@SYNCHRONIZER_GOCDB_DB_USERNAME@=
@SYNCHRONIZER_GOCDB_DB_PASSWORD@=
@SYNCHRONIZER_GOCDB_DB_DATABASE@=
@SYNCHRONIZER_GOCDB_DB_OPTIONS@=
@SYCHRONIZER_GOCDB_DB_PORT@=
@SYNCHRONIZER_GOCDB_DB_PROTOCOL@=

@SYNCHRONIZER_CICDB_DB_HOSTSPEC@=
@SYNCHRONIZER_CICDB_DB_USERNAME@=
@SYNCHRONIZER_CICDB_DB_PASSWORD@=
@SYNCHRONIZER_CICDB_DB_DATABASE@=
@SYNCHRONIZER_CICDB_DB_OPTIONS@=
@SYCHRONIZER_CICDB_DB_PORT@=
@SYNCHRONIZER_CICDB_DB_PROTOCOL@=

Script

The SINDES-script-gridview script copies the above file to /opt/gridview/etc/sconfig

Gridview NCM Component

The Gridview NCM component goes about the task of configuring the Gridview application on the machine. Depending on which gridview module is active on the machine, the component performs different tasks.

The component has pre-dependencies on

   * spma
   * sindes
   * httpd (on frontend system)
   * php  (on frontend, summarizer and synchronizer)
   * tomcat (on archiver)
   * rgma (on archiver)
   * grid_service_certificates (on archiver)

The component has post dependencies on

   * cron (on summarizer and synchronizer)
   * altlogrotate
   * chkconfig (on archiver)

The component configures different modules of gridview as follows:

Common configuration

The gridview-common module will be installed on all gridview machines. The Gridview NCM component creates the following file.

/opt/gridview/etc/gridview-common.conf

Archiver configuration

The NCM Component will create following configuration files for the archiver module

• /opt/gridview/etc/rgma-authentication.conf
• /opt/gridview/etc/rgmaarch.conf
• /opt/gridview/etc/wsarch.conf

The Gridview NCM Component should be run anytime a configuration path related to Gridview changes.

Quattorized Deployment of Gridview:

The gridview application is deployed through quattor on four machines that are part of the gridview cluster. The machines and the modules running on them are:

Machine name	Gridview Module installed
gridview101	Frontend
gridview102	Frontend
gridview103	Summarizer, Synchronizer
gridview104	Archiver

Lemon Monitoring and Alarms

The following existing standard alarms should be enabled on Gridview machines

Machine	Gridview module	Sensor	Metric	Metric_ID	Exception	Exception_ID
gridview101, gridview102	gridview-frontend	Linux	httpd	4019	httpd_wrong	30040
gridview103	gridview-summarizer, gridview-synchronizer	Linux	crond	43	crond_wrong	30021
gridview104	gridview-archiver	Linux	tomcat	40	tomcat_wrong	30055

Here are two New metrics and exceptions for gridview104 only:

Machine name	Sensor	Metric	Class	Metric_ID	Exception	Exception_ID
gridview104	Linux	gridview-archiver-rgma-dt	system.numberOfprocesses	24	gridview-archiver-rgma-dt_wrong	30136
gridview104	Linux	gridview-archiver-rgma-js	system.numberOfProcesses	25	gridview-archiver-rgma-js_wrong	30137