LCG Nightly Configurator

Background

The aim of the nightly build system is to help the Application-developers when building cross-platform software, by ensuring that the applications builds and works on the supported platforms. The build system handles the build environment, and tests the software after a successful build.

The LCG Nightly Configurator is a tool made for easing the setup of the build system.

Introduction

The LCG Nightly Configurator [LNC] is build around a web-interface which enables total control over the configuration, while checking the validity and consistency of the input.

Because of the fairly big amount of hidden complexity in the application it is recommended to skim the first section of this document before exploring the interface.

Getting started

The first thing to know is that all information is editable (depending on your access rights), even though it might look like plain text. This is part of the validation scheme; if a data-field is not “closed” meaning that you see an input field, it is impossible to submit the configuration. This principle makes it easy to see wether a configuration is valid or not, since, special data types requires server-side validation before closing the input field.

[fig] A simple configuration, notice the red text in the open input field, this configuration will not submit because of the invalid CVS tag.

The second thing to know before jumping in, is the work-flow.

To build up a configuration you have different data types that needs to be structured in order to describe which projects are build together and on what architectures. Data types

In the configuration you will find three basic data types:

• Projects
• Slots
• Parameters

Projects

Projects are as the name imply the different projects you want to build. They are configured on the dedicated page called projects by giving the project name, address for the repository and repository type. If you LNC in a administrated environment, you will have to add the e-mail addresses for each user that needs access to configuring the given project to the contacts section. In some cases you might want to configure tag names that doesn't exist in the repository or is recognized by the server; in that case you can add a rule to the ignore section.

Slots

Slots contains projects, and information about how and where to execute the given projects.

After creating a slot, you give it a name, and a description. When you press the small triangle next to the name, it will expand and you are able to configure the different aspects of the build. The simplest configuration is done by;

• Add the projects you want build (they have to exist by being defined as described earlier) by clicking on New Project and select a project from the drop-down menu that appears.
• Insert a tag name by clicking on New Tag if you have correctly configured the repository it will validate the tag and turn back to text if valid, or stay in the input field and turn red if it not found in the repository, the third option is that if a ignore rule kicks in, it turns blue and back to text without server-side validation.
• In the Platforms section you add the different platforms you want your project build on, in the same way as you added projects.
• In the Days section you can click on a given day to select when to run the build.
• Press Save configuration, and type in the reason for the change. The server should respond wether the configuration was written to disk or not.

Back-end configuration

Configuring the site is done mainly in the file called config.yaml located in the store directory. Parameters are explained inside the file.

Extending the configurator system

In order to account for changing demands and proprietary configurations, it is possible to extend the build system. The process is not 100% streamlined but logical elements are as far as possible separated into meaningful chunks.

Extending LBC without changing the core is possible in the following areas:

• Repository plug-ins: add specific repository types, and filtering of tags by making a new module
• New data elements: but creating a XSL template and adding event handles to the javascript it is possible to extend the webpage with new information.
• Direct editing in the core code: since the system is based mainly on Python and Javascript, the entire system is directly editable.

Adding a new repository type

• Create a new repository type by copying /cgi-bin/repository/cvs.py to /cgi-bin/repository/[new repository name].py the file name is important as it serves as an identifier in the system.
• In /store/config.yaml look for the parameter called repositoryTypes add the file-name without .py to the list.

also in the config.yaml file add a line to getLog or just hard-code it into your plug-in file later

• In the new plug-in file you can do whatever you need to retrieve and compare, as long as validateTag (repos , config ) returns a dictionary object with the following response:


  { "found" : "true" | "false" }

  

The two objects repos and config contains:

  repos = {
    "project" : project-name,
    "tag" : tag-to-be-found,
    "cvsPath" : repository path defined on the website,
    "cvsType" : repository type (not needed here, since your plug-in is loaded for you)
  }

  config contains a dictionary object with all parameters defined config.yaml
• Done, you can now select the repository type from the drop down menu on the projects page.

Adding new data elements

As with adding new repository types, adding elements is a multi-step process. First of all decide how to represent the information on the website. Adding simple lists is by far the easiest, specific validation methods and special representation in the configuration.xml file is beyond the scope of this guide.

Overview of the files that needs changing:

• /store/config.yaml
• /templates/toHTML.xsl
• /tmplates/toXML.xsl
• XSL-T document for represenation (/templates/xsl/xhtml/...)
• /javascript/configurator.js

As an example, lets add a new parameter section, with just a single editable field, to each slot.

Because it’s the slots section we have to edit /templates/xsl/xhtml/slot/slot.xsl

Add something like:

<div class="newparameter">
<div class="head section">
     <span class="switch switch-closed">  </span> 
     New parameter
</div>
<div class="wrapper">
   <ul class="newparameters mov">
      <xsl:apply-templates select="parameters/parameter"/>
   </ul>
   <span class="new"><img src="images/new_small.png" title="add new path"/><span>New Path</span></span>
</div>
</div>

This adds a new section to the slots, called New parameter with the class div.newparameter as selector and the unordered list ul.newparameters containing parameters. Notice <xsl:apply-templates select="newparameters/parameter"/> this tells the XSLT parser to look for data in the configuration.xml files at newparameters/parameter , relative to the slot tag:

<?xml version="1.0" encoding="UTF-8" ?>

<configuration>
   <general>
      ...
   </general>
   <slot name="New slot" description="New slot description">
      
      <newparameters>
         <parameter name="new parameter" />
         <parameter name="new parameter" />
      </newparameters>
      
      <projects>
         <project name="New Project" tag="New tag">
            <dependence project="New Project dependence" tag="New tag" />
            <change package="New package change" value="New value" />
         </project>
      </projects>
   
      <platforms>
         <platform name="New platform" />
      </platforms>

         ...
            
      <days mon="true" tue="true" wed="true" thu="true" fri="true" sat="true" sun="true" />
            
   </slot>
</configuration>

(simplified configuration.xml, notice the structure around newparameters )

Now we have added a place in the structure where we want our new data elements, next we write a XSL template that contains the list elements:

<?xml version="1.0" encoding="UTF-8" ?>

<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                        xmlns="http://www.w3.org/1999/xhtml">
<xsl:output method="xml" indent="yes" 
                      doctype-system="http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd" 
                      doctype-public="-//W3C//DTD XHTML 1.1//EN"/>


   <xsl:template match="newparameters/parameter">
         <li>
            <div class="newparameter-lielement">
               <div class="header drag">
                  <span class="left">
                     <span class="newparameter-element editable">
                        <xsl:value-of select="@name"/>
                     </span></span><span class="right">
                                <span class="del">
            <img alt="" src="images/del.png"/></span></span></div></div>
         </li>
   </xsl:template>
</xsl:stylesheet>

(Because we want to add new elements later on, after the page is generated, we need to add the list elements in a separate template file, to make it accessable to the core.)

Save the file in the /template/xsl/xhtml/... structure, say: /template/xsl/xhtml/slot/newparameter.xsl in order for the page to find the new XSL document, you have to add a line to /template/toHTML.xsl :

     <xsl:include href="xsl/xhtml/slot/newparameter.xsl" />

So far so good, if you add a new structure to the current /store/configuration.xml file containing a newparameters section with a few parameters, it should be rendered on the website.

In order to make the elements editable, we have to add an event handler to /javascript/configurator.xml look for:

  ////  HANDLES /////
  
   handle = [
      ["span.slotTitle", "text"], //Handler for slot title name
      ["span.addmail", "email"], //Handler for adding email addresses
                     .
                     .
                     .

- and add another like to the array with: ["span.newparameter-element", "text"], this added a clear-text handle. When you click the text area it turns into a input-field.

In order to save the changes, we need to edit /template/toXML.xsl which contains the XSL-Transform back to the configuration.xml structure.

Add something like:

<newparameters>
        <xsl:for-each select=".//div[@class[contains(.,'newparameter-lielement')]]">
      <parameter name="{.//span[@class[contains(.,'newparameter-element')]]}" />
   </xsl:for-each>
</newparameters>

The selection rules are based on XPath: http://www.w3.org/TR/xpath

Syntax reference for XSL-T: http://www.w3.org/TR/xslt.html

Now the page is able to load and save the new element type.

The last thing we need to add, is a single line to the /store/config.yaml file, that tells the page where to look if you want to add a new parameter element to the page.

#XSLT mappings
elements:
    slots : "xsl/xhtml/slot/slot"
    projects : "xsl/xhtml/slot/projects/project/project"
       .
       .
       .

Add newparameter : "xsl/xhtml/slot/newparameter.xsl" and we are done.

This is the most basic thing to add, if you want different event-handles, back-end validation and so on, it requires a deeper understanding of the structure.

Direct editing information

The core is build to support Python 2.3, and requires the following modules:

• libxml2
• libxslt
• yaml (included)

The website uses jQuery for cross-browser javascript and AJAX support.

Links

http://www.jquery.com

-- MortenDamJoergensen - 19 Aug 2008