• ServiceChallengeTwoProgressSARALogbook

Preparation

• Scripts - RT
  • Probably modify the '''radiant-load-generator''' script to account for the oplapro nodes that will be in use.
  • Write cleanup script to wipe out disk/filesystem on Rembrandt cluster.
• Logging
  • Parse gridftp logs and put stuff in postgres. - RT

Done. On rembrandt[5-8] there are cron jobs running that parses the gridftp logs and inserts the entries of the past hour in a postgres database that is running on winnetou.matrix.sara.nl. This database can be queried at http://winnetou.matrix.sara.nl/cgi-bin/sc.cgi.

• • Parse gridftp logs and put the last so many entries in http server. - RT
  • Make sure that '''oplapro80.cern.ch:/var/log/radiant/radiant-NL.log''' files are archived. - RT

Done. In /afs/cern.ch/user/t/trompert there is a script called start_daemon.sh that does this job. It starts up the daemon and saves the existing /var/log/radiant/radiant-NL.log to ${HOME}/var/log/radiant/radiant-NL.log_`date +%s`

• • Ganglia - ???
  • MRTG graphs - ???
• Configuration Rembrandt cluster
  • kernel - ???
  • disk - ???
• Configuration Network - ???
• Preliminary tests - ???

Configuration Information - Monitoring

GridFTP logs

The scripts used for the gridftp logs may be found at: [http://winnetou.matrix.sara.nl/SC/SC-03-05/ http://winnetou.matrix.sara.nl/SC/SC-03-05/].

Web Page

On the rembrandt nodes cron jobs ([http://winnetou.matrix.sara.nl/SC/SC-03-05/gridftplogcp.sh gridftplogcp.sh]) are running every two minutes (every even minute) that do a tail -100 of /var/log/gsiftp.log and copy that into /home/rembrandt0/ron/var/www/gsiftp_rembrandt[3-6].log. On rembrandt5 a cron job runs (gridftplog2html.sh)that every odd minute that merges these files and does a sort based on end time on these files. Finally, this cron jobs parses the gridftp log output to a html file and copies it to winnetou.matrix.sara.nl were it can be viewed at [http://winnetou.matrix.sara.nl/SC/gridftplog.html http://winnetou.matrix.sara.nl/SC/gridftplog.html].

Postgres Database

On the rembrandt nodes cron jobs (gridftpsql.sh) run every hour that parse the entries in the gridftp logs of the past hour and inserts them in a postgres database. The postgres database can be queried at: " target="_blank">http://winnetou.matrix.sara.nl/cgi-bin/sc.cgi]]. The SQL table is called gridftp and contains the following data:

CREATE TABLE gridftp (
   host text,
   VO text,
   username text,
   src text,
   dest text,
   block_size bigint,
   nstreams bigint,
   buffer_size bigint,
   nbytes bigint,
   nstripes bigint,
   throughput double precision,
   volume text,
   file text,
   code int,
   start_year int,
   start_month int,
   start_day int,
   start_hour int,
   start_minute int,
   start_second int,
   start_microseconds int,
   end_year int,
   end_month int,
   end_day int,
   end_hour int,
   end_minute int,
   end_second int,
   end_microseconds int,
   start_date date,
   end_date date
);

SET DateStyle TO 'European';

Monitoring

• <Gridftplog> of the last transfers upto two minutes ago on each node.
• <Gridftplog hourly>. Contains the hourly averaged throughput.
• <Gridftplog daily>. Contains the daily averaged data transport.
• <Gridftplog for all sites>.
• <Postgres> database with all transfers upto an hour ago or less.
• <Ganglia> Rembrandt Cluster monitoring.
• <xanax>

* Individual GigE channel in the 10GigE can be seen on:

• channel 1
• channel 2
• channel 3

-- JamesCasey - 17 Jun 2005