Build Configuration and Integration Policy

• field contents are inherited from the parent module upon creation but can be changed when needed
• common fields of each module are:

ETICS name

unique name of the module, see module naming conventions

display name

name displayed in the applications, usually kept empty so that the name is displayed

description

short description of the module, unless the property ${package.description} is set, the component description text is introduced in the Description field of the packages by the ETICS packager. For RPM please refer to Fedora Guidelines - Summary and Description [R4], for Debian please refer to Debian Policy Manual - Description [R5]

vendor

must be set to EMI

vcsroot

root of your anonymous CVS or SVN or WGET account. This value will be used in checkout commands with the property ${vcsroot}. Examples are :pserver:anonymous@etics.cvs.cern.ch:/cvs/etics for CVS, for SVN or http://myserver.com/download for WGET

repository

ETICS repository instance to be used. By default http://eticssoft.web.cern.ch/eticssoft/repository. Do not change unless you know what you are doing!

homepage

website of the project or package for more information

created

read only, creation date

modified

read only, last modification date

id

read only, internal identifier

• Additional fields available only on components:

  license

  software license of the software

  package name

  used by the ETICS packager as name of the package. If not specified the component name is used by default.

  download

  location from which packages can be downloaded, used mainly for external components which are taken from other location and added in ETICS

* A configuration is an ETICS object which contains all the information to checkout, build, test and package a specific version of a component

• To better understand, ETICS modules are similar to CVS modules as software containers while ETICS configurations are similar to CVS tags as versions over time of a component
• Configurations can be created inside projects, subsystem and components
• Project and subsystem configurations do not contain any checkout/build/test instructions. They are used instead to group children configurations in subsystem and in turn project releases
• Component configurations are used to define the checkout/build/test instructions
• An example can be the following configuration tree. A new project release (version 2.0.0) is composed of version 2.5.0 of the servers and of version 6.0.0 of the clients. The server subsystem is in turn composed of three versions of server components and the client subsystem is composed of three versions of client components. In this example only the component configurations define instructions for checkout/build/test while the other configurations are used to group configurations in releases:
  • P: my-project_R_2_0_0_1
    • S: servers_R_2_5_0_1
      • C: web_server_R_2_5_1_1
        • component checkout/build/test instructions
      • C: database_server_R_2_0_1_1
        • components checkout/build/test instructions
      • C: database_schemas_R_3_0_0_1
        • components checkout/build/test instructions
    • S: clients_R_6_0_0_1
      • C: java_api_R_5_2_2_1
        • components checkout/build/test instructions
      • C: python_api_R_5_0_0_1
        • components checkout/build/test instructions
      • C: cli_client_R_5_6_1_1
        • components checkout/build/test instructions

• Examples:
  • emi-unicore-ucc_B_TRUNK
  • emi-nordugrid-arc-nox_R_1_0_0_1
  • emi-glite-ce-cream-client-api-c_R_1_7_1_1
  • emi-dcache-nameserver-pnfs_B_HEAD

version

composed of major, minor and revision, it defines the version of the software defined in the configuration

age

also known as release, it defines the changes in packaging

tag

if a VCS is used, this field defines the tag or branch to be used in the checkout command. This is done to avoid changing the checkout commands at every new release and only change this field which is then used in the checkout command as system property ${tag}

profiles

it is used to specify a list of profiles which trigger the ETICS plugins. For instance adding the string 'findbugs' would trigger the findbugs plugin during the build

download from

see section below

• This value is used to generate the upload/download locations of all types of packages: TAR.GZ, DEB, RPM.
  • to address TAR.GZ is used as it is
  • to address RPM, the TAR.GZ file name is removed and the following RPM name is added (preserving the directories): ${packageName}-${version}-${age}.${architecture}.rpm
  • to address DEB, the TAR.GZ file name is removed and the following DEB name is added (preserving the directories): ${packageName}_${version}-${age}_${os}_${architecture}.deb

• To change the standard RPM or DEB package names to be used to replace the TAR.GZ package name it is possible to define the system properties ${package.rpm.name} and ${package.deb.name}.
• This field can also be reset to directly point to the RPM or the DEB file.

• this is the location used for dependencies to:
  • check if the package is available in binary format in the repository.
    • If the RPM is found and the build has root or sudo privileges, the package is installed in the build node.
    • If the RPM or the privileges are missing but the tar.gz is found, it can be downloaded as binary during the checkout.
    • If not, the component is built from source
  • upload the binaries in the ETICS repository after the build from source

• Properties are key=value pairs which can be used in the VCS/build/test commands.
• Properties can be added/edited and removed in each configuration.
• A property reference can be added in commands by typing ${PROPERTY_NAME} and it will be replaced at runtime with its value from the ETICS CLI client.
• Properties are inherited from parent configurations and get overwritten in the following order Project -> Subsystem -> Component (i.e. properties in component configuration overwrite properties with the same name in subsystem configurations)
• Properties can be prefixed with the module name (i.e ${MODULE_NAME.PROPERTY_NAME}) to access properties of other components (if these components are first level dependencies).
• Four types of Properties are available:
  • User properties: user defined properties can be used as variables in the commands. Users define both the property KEY and VALUE and then uses the property as ${KEY} in the commands.
    • This allows editing of command parameters without the need of changing the command itself but instead by just changing the property value.
    • Properties can be explicitly defined in the ETICS CLI client when executing a command (etics-checkout, etics-build or etics-test) using the option -p and, if set, they overwrite all properties available in the configurations. This can be used for customization of single jobs.
    • Since properties are inherited, a property set at project/subsystem level can then be used in hundreds of component configurations. This then allows users to change the behavior of many configurations by editing a single variable at project or subsystem level.
  • System properties: are ETICS-defined properties which can be used to customize the system or to get useful information from the system. A full list of System properties is available with description in the Configuration WA in the "Add System Property" panel. Examples are:
    • Many properties can be used to tune the ETICS packager such as: ${package.prefix} which defines the root of the package, ${package.configFiles} which is a comma-separated list of files to be marked as CONFIG files
    • ETICS Plugins are usually configured using properties such as: ${findbugs.src.location} which defines the source location of the Java code to be analyzed, ${junit.failure} which defines whether the component must be marked as Failed if some unit tests fail
    • Several properties provide the user information about the platform on which the job is running such as: ${libDir} which is defined as "lib", "lib64" or "lib32" according to what OS and architecture the job is running on
  • Filed properties: are properties that match exactly fields of a module or a configuration. In general, every field of the current module or configuration can be accessed as property. ${tag} accesses the current configuration field named "Tag", ${moduleName} accesses the current module name.
  • Dependency resolution properties: are properties used to resolve dynamic or version constrained dependencies.
    • They are usually set in project configurations (to have the same version of a component in the whole project) , dependency resolution properties are of the format: moduleName.DEFAULT=configurationName
    • This properties are then used by the ETICS CLI client to resolve the exact versions of dependencies which have been set as dynamic in the component configurations
    • If more than one definition is found in the build, the latest version found is used as dependency for all the component no matter what was the original definition specified
    • See section below for more information on dependencies.

• Dependencies are used to relate component configurations. Dependencies have a scope which can be "Build-time", "Runtime" or "Build and Runtime" and a type which can be "Static", "Dynamic" or "Version constrained"
• Scope:
  • Build-time dependencies are used when a component needs some files from another component while building (compile/link/document/etc.)
    • Once a Build-time dependency is set, ETICS gives to the component a property named ${[COMPONENT NAME].location} which points to the root of the binaries of the dependency-component in order to address the needed files.
    • ETICS also resolved the build order accordingly making sure components are built AFTER their dependencies
  • Runtime dependencies are instead used to add in the binary package metadata (RPMs or DEBs) the right REQUIRES tags in order to force the installation of the dependency when the package is installed
    • Runtime (only) dependency are usually not checked-out nor built as they are only used to configure package metadata. If YUM repositories are created and users want to have also runtime dependencies added into the repository. An option can be used to force the checkout/download of runtime dependency: --runtimedeps (or "checkout runtime dependency as well" from the web application submission form)
  • Build-time and Runtime dependencies behave as both runtime and build-time dependencies

• Type:
  • Static dependencies are dependencies which do not use the ETICS default version of a component, but hardcode a specific version in their dependency metadata. This is done by setting the specific configuration (therefore version) of the dependency they want to use
  • Dynamic dependencies are dependencies which specify only the module name and let the version be resolved at higher level. The versions get resolved by the ETICS CLI client by analyzing the properties defined as moduleName.DEFAULT=configurationName
  • Version constrained dependencies are dependencies which specify a version range (i.e. > X.Y.Z) as acceptable for a component. As runtime dependencies, the package metadata (i.e. REQUIRES) is set with the version constrained (> X.Y.Z). As build-time, the ETICS CLI client will just verify if the default version (the one specified by ${moduleName.DEFAULT}) satisfies the constraint. If it satisfies, that version is used. If not, an error is triggered.

• VCS commands are usually used to download somehow the source code into the build machine
• The checkout command is a shell line which is executed by ETICS from the workspace directory.
• Useful system properties which can be used in the checkout command are:

  ${moduleName}

  the name of the module (usually is the component name since the commands sit only in component configurations)

  ${tag}

  the TAG field of the configuration

  ${majorVersion}, ${minorVersion}, ${revisionVersion} and ${age}

  the various parts of the version field of the configuration M.m.R-A

  ${version}

  "${majorVersion}.${minorVersion}.${revisionVersion}"

  ${vcsroot}

  the vcsroot field of the configuration

• When checking-out a module (project, subsystem or component), by default the specified module and all its children modules are checked-out from source by executing their checkout commands
• Everything else is:
  • For dependencies resolved as OS.DEFAULT:
    • If the build has root or sudo privileges, it is installed from YUM or APT
    • If not, it is marked as Unresolved, and the package name appears in the list of packages to install (Section Execution Info of the ETICS report)
  • For dependencies resolved to configurations:
    • If already installed in the system, it is just used
    • If not installed, an RPM is found in the registered repository (according to downloadFrom) and the build runs with root or sudo privileges, it is installed
    • If an RPM is not found or the privileges are not correct, the tar.gz is downloaded from binary from the registered repository and placed in the ${stageDir}
    • If the tar.gz is not found either, the checkout command of its configuration is executed
• The behavior of how to checkout components which are not the selected module or its direct children, can be changed using options when launching etics-checkout or when submitting a remote-build or remote-test

* Examples are:

mkdir -p ${moduleName}

if no source code at all is required

cvs -d ${vcsroot} co -r ${tag} myCVSandETICSmoduleName

if the CVS module name does correspond to the ETICS module name

cvs -d ${vcsroot} co -r ${tag} -d ${moduleName} myCVSmoduleName

if the CVS module name does not correspond to the ETICS module name

svn co ${vcsroot}/${moduleName}/tags/${tag} ${moduleName}

typical subversion command

mkdir -p ${moduleName}/src; cd ${moduleName}/src; wget http://my.location.com/myPackage.tar.gz; tar xzf myPackage.tar.gz; rm -rf myPackage.tar.gz;

any shell command can be specified if it creates the right directory and downloads somehow the source code.

• Mistakes are:

  echo "NO CHECKOUT"

  because it does not create the directory

  cvs -d ${vcsroot} co -r ${tag} myCVSmoduleName

  if the CVS module name DOES NOT correspond to the ETICS module name

• Build commands are executed for each built configuration after the checkout phase (by executing the command etics-build). ETICS automatically generates a build order to properly satisfy build-time dependencies.
• Build commands are executed inside the component directory within the workspace (in the ${workspaceDir}/{moduleName} directory unless the property ${srcDir} which overwrites this behavior is defined)
• All targets must include shell commands which will be executed as they are after each property is replaced.
• The build commands are composed of targets which are executed in order: init, configure, compile, doc, checkstyle, install, packaging, test, prepublish, publish, postpublish.
• Two targets have a special behavior:
  • install is used to define which files have to be packaged/staged. This is done by copying all the files in a special directory which can be addressed via the property ${prefix}.
  • packaging is used to create packages (RPMs, TGZs, DEBs). If this target is left empty, the ETICS packager will execute and package the files previously copied in the directory ${prefix}. The package format depends on the platform but usually is TGZ and (DEB or RPM). If this target is instead defined, ETICS will just execute the specified command.
• The "clean" target of each built component is instead only executed (alone) when the option "-t clean" is added to the etics-build command