Grid Publisher Development

Introduction

The goal of this wiki page is to plan but also track the work on Grid Publisher foreseen by the Monitoring Working Group.

Plan

Phase I

Goal: Set up a simple ActiveMQ based environment and test it using GridFTP log data:

• take historical logs from CERN production services to replay or generate simulated data transfer events
• adapt log parser (Python) to publish to ActiveMQ broker
• set up a single instance of broker under Tomcat application server
• develop a simple message summarizer (Java) that consumes GridFTP log messsages in ActiveMQ, does simple summarisation (for example number of events per minute) and publishes the results back to ActiveMQ
• put an additional consumer that simply reads the messages
• test the capabilities of the system in following dimentions:
  • number of messages per second
  • number of parallel consumers/producers
  • number of topics
  • complexity of JMS selector

Details:

• machines for test installation provided by James (lxb6117, lxb6118)
• an existing Python implementation of STOMP protocol pointed out by James, to be evaluated/extended by Wojtek

Phase II

Goal: Evaluate possible broker network configurations, set up and test the most promising one(s), test durable subscriptions

• enable persistent messaging on broker running at lxb6117 (MySQL database)
• provide high availability and fault tolerance by Master-Slave broker configurations (Shared Filesystem based, JDBC based)
• test networks of brokers and brokers' discovery methods
• test client-side dynamic failover

Phase III

Goal: Develop security mechanisms by using digital signatures and encryption on the message level with Grid certificates (proxies).

Details to come

Progress tracking

2007-07-19 - planning meeting

• Participants: James Casey, Ian Neilson, Wojciech Czech, Piotr Nyczyk
• Goals achieved: Initial planning

2007-07-30 - phase I tests report

• report1.pdf: Report on phase I tests

2007-08-09 - phase II tests report

• report2.pdf: Report on phase II tests

2007-08-09 - evaluation of tests results

• Participants: James Casey, Ian Neilson, Wojciech Czech, Piotr Nyczyk
• Goals achieved: Planning to test prototype in SAME configuration

2007-09-19 - Grid Messaging System Presentation

• presentation.pdf: Wojciech Czech presentation on test results and ActiveMQ overview.

2007-10-25 - ActiveMQ extended Tests Presentation and Report

• Participants: Max Boehn, James Casey, David Collados, Gavin Mccance, PiotrNyczyk, Daniel Rodrigues
• Summary: Presentation of the 1st month of work on the MSG ( Messaging Service For Grids )
• Final Remarks:
  • Double Check inconsistent results on the ActiveMQ broker capabilites
  • Further test with more fine-grained measurements, namely consumers throughput and latencies.
  • Stick to ActiveMQ 5.0, unless unusable
  • Time visualization on impact of aditional producers or subscribers. ( graphs of messages/second per each producer/ consumer )

• report-A-18Oct2007.pdf: First Month Work Report
• Evaluation_of_ActiveMQ_broker_for_the_Grid_Messaging.ppt: First Month Work Report

2008-01-29 - Openlab Presentation ( MSG: An Overview on a Messaging System for Grid )

• Summary: Presentation on test results and further directions for the MSG in Openlab Quarterly Meeting
• Further Steps:
  • Test with different topics, queues, + master slave configurations;
  • Implement SAM messaging (based on Piotr Prototype
• MSGOpenlab_v1.2.pps: MSG Openlab Presentation

2007-01-30 - Test Plan

Further test plans:

clients on different machines, improvedClient	1 Consumer, 1 Producer	3Consumers, 3 Producers	5 Consumers, 5 Producers
Queue Performance	1 Consumer 1 Producer	3 Consumers 3 Producers, 1 queue	3 Consumers, 3 Producers, different queues
Master/Slave	1 Consumer 1 Producer Queue	3 Consumers 3 Producers, 1 topic	100 Producers, 1 Consumer, Queue
Master/Slave,SAN
Master/Slave,JDBC
*Network Of Brokers