Interim gLite Release Process for gLite 3.1 and 3.2

A metapackage patch is a patch that affects only one metapackage and that contains:

• A list of new packages affecting the metapackage:
  • produced by the product team owning the metapackage or by other product teams.
  • existing already as a dependency of the metapackage or defined as a new dependency.
• The metapackage package itself.

When a product team wants to release a new version of a metapackage to production, they have to decide which platforms are going to be affected by the change and open a new patch PER PLATFORM in Savannah. The initial state of the patch is With Provider. The following fields are mandatory:

• Status: With provider
• Assigned to: responsible in the product team
• Release Path: Standard
• Summary: string to identify the patch, for example: OS/ARCH METAPCKAGE update
• gLite release: gLite 3.X
• Operating System: SLX
• Architecture: i386 or x86_64
• YUM repo file URL: URL to download the YUM repo file to be able to install the packages released in the patch
• gLite subsystem tag(s)/ETICS configurations: glite configuration names of the subsystems released in the patch
• RPM names: list of URLs of the released packages
• Affected Metapackages: METAPACKAGE_NAME
• Metapackage Changes: None
• Metapackages to be reconfigured: METAPACKAGE_NAME if applicable
• Metapackages to be restarted: METAPACKAGE_NAME if applicable
• Configuration changes: description if any
• Release notes: description
• List of attached bugs fixed in the patch + internal patches

Bear in mind the following rules:

• RPM names: The package names are provided with the URL from where they can be downloaded. Normally a URL in the ETICS repository when the packages are built in ETICS, otherwise, any other valid URL. The URLs can be easily copy and pasted from the ETICS build report -> Package List -> Package type: rpm and Download as text. Note that due to GGUS ticket 56917, the volatile repo appears in the URL instead of the permanent repo. This has to be changed manually until the problem is fixed. Change volatile/default with registered in all the affected URLs.
• RPM names: There's no need to specify a delta anymore. PTs can copy and paste the full list of packages resulted after an ETICS build of the metapackage in the Savannah patch.
• RPM names: The metapackage is produced by the product team and it's always included in the list of RPMs. See the next section to know how to prepare a metapackage in ETICS.
• Release Path: It is used to mark the patch as a metapackage patch. Its value should be Standard.
• gLite subsystem tag(s)/ETICS configurations: it's very important to fill in this field since this is what the Integration Team will use to maintain a project configuration with the versions of Verified packages. A subsystem configuration should always be specified.
• YUM repo file URL: URL to download the YUM repo file to be able to install the packages released in the patch. The YUM repository specified here should be the one created after a registered build is launch. No volatile repositories are allowed for the certification of a patch. Use any of the URLs specified in the ETICS report where [protect=0].
• Affected metapackages: It will now contain ONLY one metapackage. This is because in the new model, each metapackage is released in one patch because the repositories are now per metapackage.
• Metapackages to be reconfigured: It will now contain ONLY the metapackage name released in the patch, if applicable. Otherwise set it to None.
• Metapackages to be restarted: It will now contain ONLY the metapackage name released in the patch, if applicable. Otherwise set it to None.
• Release Notes: The Release Notes are prepared by the Product Team responsible for the release. Check the Developers Guide for the details about the Release Notes guidelines.
• List of attached bugs: A metapackage patch should include a dependency on the corresponding bugs specific to the service that it is supposed to fix. Bugs relevant to other internal patches should not be attached to the metapackage patch. However, the internal patches should be attached as explained below.
• List of attached internal patches: A metapackage patch should include a dependency on all the internal patches also released with the new metapackage. Check the Internal Patch twiki where the list of internal patches is tracked. You can also query which patches contain a certain package by using the RPM query form in the Savannah patch browsing.

The following fields are no longer needed but are still kept to acces the information in old style patches:

• Metapackage Changes: It is no longer needed. Set it to None. The product team is responsible for adding or removing new dependencies in the metapackage in ETICS. After that, it's only necessary to list the packages and the metapackage in the RPM names field. Since this field is mandatory, for a while we'll need to put some value, use None.

Note that certain fields will be filled in once the ETICS builds for the packages to be released are done. This means that at the time of opening the patch you may not be able to fill all the fields completely. PTs should make sure the corresponding task in EGEE JRA1 workplan is up to date and contains dependencies to the corresponding patches.

Product teams will be able to use the same project configuration to build the middleware and to create metapackages. The project configuration to be used is called glite_3_X_cert and it will contain the versions of Verified packages.

Metapackages in ETICS

Metapackages should be defined under the relevant ETICS subsystem for the product team. In some cases like WN, UI or VOBOX, there is no such subsystem and in that case the metapackages should be under org.glite.node in ETICS. The metapackage should only contain first level dependencies. However, the product team should also provide any second level dependencies that are not part of SL or DAG repositories. The second level dependencies will be included in the metapackage patch so that these packages can be added to the gLite repositories.

Some metapackages already have a org.glite/node/metapackage_name component in ETICS. The existing metapackage components have at least one configuration called org.glite.node.METAPACKAGE_NAME_3_X_Y created by the Integration Team. This configuration contains the list of dependencies for the mepackage version 3_X_Y in the production repository for the 3.X release. Editing the configuration in the ETICS workspace and selecting Dependencies will show the list of package dependencies. All of them are defined as Runtime dependencies and are marked as using default configuration.

Use this configuration to re-create a new one under the relevant subsystem. You can do this by running the etics command: etics-configuration prepare -o METAPACKAGE_NAME.ini -c METAPACKAGE_NAME_3_X_Y org.glite.node.METAPACKAGE_NAME

This will give you an ini file you need to modify to change the module name and use it to upload the configuration under a different subsystem.

If your metapackage doesn't exist under org.glite.node, check the section below to create the configuration yourself.

Metapackage configuration

If you want to start defining your own metapackage components, you first need to create a new component under the relevant subsystem ( org.glite.node if there's no associated subsystem). Then add a new configuration for it.

A template of the ini file to create the metapackage component for the first time is shown below where SUBSYSTEM_NAME has to be replaced by the name of the subsystem under which you want to create your metapackage component, and METAPACKAGE_NAME has to be replaced by the name of the metapackage:

;
; INI Template file for the object "Component" called "org.glite.SUBSYSTEM_NAME.METAPACKAGE_NAME"
;

; The section [Component-<component-name>] contains component information

; The section [Parent] contains the parent pair (type, name) information

[Component-org.glite.SUBSYSTEM_NAME.METAPACKAGE_NAME]
vendor = EGEE
description = METAPACKAGE_NAME
repository = http://eticssoft.web.cern.ch/eticssoft/repository/
packageName = METAPACKAGE_NAME
vcsroot = :pserver:anonymous@glite.cvs.cern.ch:/cvs/glite
licenceType = ASL 2.0
download = None
displayName = METAPACKAGE_NAME
homepage = None
name = org.glite.SUBSYSTEM_NAME.METAPACKAGE_NAME

[Parent]
Subsystem = org.glite.SUBSYSTEM_NAME

To create the first configuration, we suggest you use the last available production version of the metapackage. You need to get the list of its dependencies.

• Refer to the gLite web pages to know which is the last version of the metapackage in production:
  • For 3.1: http://glite.web.cern.ch/glite/packages/R3.1/
  • For 3.2: http://glite.web.cern.ch/glite/packages/R3.2/
• Query the metapackage to get its list of dependencies. In the RPM list of each metapackage, you have a link to the metapackage itself. i.e. =rpm -qpR http://glitesoft.cern.ch/EGEE/gLite/R3.2/glite-GENERIC/sl5/x86_64/RPMS.updates/glite-LFC_mysql-3.2.2-0.x86_64.rpm=
• Use the list of dependecies to create the configuration ini file.
• The version syntax of a metapackage is 3.1.X-0 for gLite 3.1 or 3.2.X-0 for gLite 3.2. It's up to the product team to choose the version of a metapackage as long as it's higher than the last production version. Note that in order to follow the same policy used in the past before the product team model, the recommendation is to increment only by one the version of the metapackage.

The template below can be used as an example to create a component configuration. SUBSYSTEM_NAME has to be replaced by the name of the subsystem under which you have created your metapackage component, PLATFORM has to be replaced by either default, sl5_x86_64_gcc412, slc4_x86_64_gcc346 or slc4_ia32_gcc346. METAPACKAGE_NAME has to be replaced by the name of the metapackage and X, Y, Z have to be replaced with the version of your metapackage:

[Configuration-org.glite.SUBSYSTEM_NAME.METAPACKAGE_NAME_R_3_X_Y_Z}]
profile = None
moduleName = org.glite.SUBSYSTEM_NAME.METAPACKAGE_NAME
displayName = METAPACKAGE_NAME_R_3_X_Y_Z
description = METAPACKAGE_NAME for version R_3_X_Y_Z
projectName = org.glite
age = Z
vcsroot = 
tag = 
version = 3.X.Y
path = \${projectName}/\${moduleName}/\${version}/\${platformName}/\${packageName}-\${version}-\${age}.tar.gz

[Platform-PLATFORM:StaticDependency]
;<project-name>|<module-name> = <config-name>,<scope>

[Platform-PLATFORM:Property]
package.prefix = /opt/glite

[Platform-PLATFORM:BuildCommand]
checkstyle = None
packaging = None
displayName = None
description = None
doc = None
publish = None
postpublish = None
compile = cd build; echo -e "${gLiteCopyrightText}\n\n${gLiteLicenseText}" > LICENSE; cp LICENSE COPYRIGHT; echo ${version}-${age} > node-version; echo ${platformArch} > arch; echo "NA" > service; echo "NA" > update
init = mkdir -p ./build ${prefix}/release/METAPACKAGE_NAME
install = cp -fp build/* ${prefix}/release/METAPACKAGE_NAME
prepublish = None
test = None
clean = None
configure = None

[Platform-PLATFORM:DynamicDependency]
DEPENDENCY_LIST

[Platform-PLATFORM:TestCommand]
;init = None
;clean = None
;displayName = None
;description = None
;test = None

[Platform-PLATFORM:Environment]
;env1 = value1

[Platform-PLATFORM:VcsCommand]
displayName = None
description = None
tag = None
branch = None
commit = None
checkout = echo "NO CHECKOUT"

[Hierarchy]

Note that vcsroot and tag are left empty.

DEPENDECY_LIST should be replaced by a list of the form project.name|package.name = R. You should add one entry like this per dependency. In gLite, we basically have packages from three ETICS projects: org.glite, externals and vdt. If you are not sure whether a package belongs to one or another project, you can surf the ETICS project tree or contact the Integration Team. All the dependencies in the metapackage should be Runtime dependencies, that's why the R is used. i.e.:

org.glite|bdii = R
org.glite|edg-mkgridmap = R
externals|fetch-crl = R
org.glite|org.glite.security.voms-api = R
org.glite|glite-version = R
org.glite|org.glite.yaim.core = R
org.glite|glue-schema = R
org.glite|lcg-vomscerts = R
externals|mysql-server = R
vdt|vdt_globus_essentials = R

The ETICS commands you can use to upload the component and the configuration in ETICS are (we assume you have the ETICS client installed, otherwise please check ETICS client HowTo):

etics-workspace-setup
etics-get-project org.glite
etics-module add -i component.ini --parent org.glite.node org.glite.node.${metapackage}
etics-commit
etics-configuration add -i /tmp/configuration.ini
etics-commit

You may get errors when trying to upload the configuration if any of the specified dependencies do not exist in ETICS or in the specified project. Make sure you don't have any typos and contact the Integration Team in case of problems.

When the configuration is uploaded in ETICS, you can check it in the web interface by editing it in the workspace and selecting Dependencies. This will show the list of package dependencies. All of them are defined as Runtime dependencies and are marked as using default configuration.

Prepare the metapackage in ETICS

Once a configuration is created for the metapackage, it will be used to prepare the metapackage with ETICS. The dependencies have been defined as Runtime and using default configuration, which means that when preparing the metapackage in ETICS, a project configuration will be used against which the version of each dependency will be solved.

The org.glite configuration that defines the version of each package is called glite_3_X_cert and this is the configuration that has to be selected when building the metapackage, as it will be explained later. The Integration Team is responsible for maintaining glite_3_X_cert. Every time a patch is Verified, the Integration Team will update the versions of the relevant subsystem configurations. In this way, product teams will always get the last verified versions of other packages provided by other product teams.

glite_3_X_cert contains Runtime Dependencies and Build dependencies which means that Product Team can also build their binaries using glite_3_X_cert. Product teams are therefore responsible for providing also the set of build dependencies that should be part of the subsystem configuration to be able to build.

If a product team wants to use a specific version of a component different from the version available in glite_3_X_cert, they can manually specify the needed version using a static dependency.

The steps to create a new version of a metapackage are:

• Clone an existing configuration or Create a new one.
  • If you want to create a new configuration follow the steps described in The metapackage configuration.
• Apply any manual modifications to the dependency list:
  • To release a new version of a component:
    • if the component is in a different subsystem than the metapackage: use static dependencies to specify the new version.
    • if the component is in the same subsystem than the metapackage:
      • you can use default dynamic dependencies, create a new version of the subsystem and build the subsystem. This will create the metapackage with the new version of the component.
      • you can use static dependencies and build the metapackage.
  • To specify a component version different than the version provided by glite_3_X_cert, use also static dependencies.
  • Remove any dependencies, if needed.

Then you will create the metapackage either building your subsystem or building directly your metapackage.

• Launch the ETICS build for the subsystem by selecting the following items:
  • Propagate environment and properties from glite_3_X_cert (3_X should be replaced with 3_1 for 3.1 and 3_2 for 3.2)
  • Use custom checkout behaviour possibly build from binary
  • Checkout run-time dependencies as well
  • Generate YUM repository
  • --urlname patch_number
  • Host selection:
    • Scientific Linux 4 (ia32) with gcc 3.2.3 for gLite 3.1/32bits patch
    • Scientific Linux 4 (x86_64) with gcc 3.4.6 for gLite 3.1/64bits patch
    • Scientific Linux 5 (x86_64) with gcc 4.1.2 for gLite 3.2/64bits patch

If you use the etics client command line interface, then you should run:

Checkout: etics-checkout --frombinary --config org.glite.subsystem_X_Y_Z --project org.glite --project-config glite_3_X_cert --runtimedeps org.glite.subsystem
Local Build:     etics-build --config org.glite.subsystem_X_Y_Z --target postpublish org.glite.subsystem
Remote Build: etics-submit build --config org.glite.subsystem_X_Y_Z --target postpublish --yum org.glite.subsystem

NOTE: The created metapackages contain dependencies

>= x.y.z

without containing the age of the package, as it's the case now with our metapackages. In order to include the age, define the following property in your metapackage configuration package.versioneddepswithage = true.

Once the metapackage is built with ETICS, together with the rest of new packages, the product team fills in the RPM names and gLite subsystem tag(s)/ETICS configurations in the savannah patch and changes its status to Ready for certification, to trigger the certification of the patch.

Note on static dependency management in ETICS

Please, take into account the following example:

If in glite_3_X_cert yaim-core v. 4.0.11-2 is specified but you want to use yaim-core v. 4.0.10-2, in your metapackage you should specify a static dependency on glite-yaim-core_R_4_0_10_2. This will imply:

• The repository created by ETICS will contain yaim-core v. 4.0.10-2.
• The metapackage will have a dependency on yaim-core >= 4.0.10-2.

Specifying a dependency such as: org.glite.yaim.core with version >=4.0.10-2 brings to a different behavior:

• The repository created by ETICS will contain the yaim-core version specified in glite_3_X_cert (that is 4.0.11-2 in our example)
• The metapackage will have a dependency on yaim-core >= 4.0.10-2

Note on Metapackage dependency management in ETICS

Due to a bug in ETICS, dependencies that are not defined under the metapackage may end up in the YUM repository with a version different from the DEFAULT one specified in the project configuration. This is the case for second level dependencies that may have been locked against a different project configuration.

For example.

glite-TURBO -> glite-depA

glite-depA -> glite-depB (locked pointing to glite_depB_1)

The project configuration myProjectConfig contains glite_depB_2.

When building glite-TURBO against myProjectConfig, glite_depB_1 ends up in the YUM repository.

A bug fix is being implemented to avoid this. In the meantime, the workaround is to lock the metapackage before building it.

When launching an ETICS build, it's possible to create at the same time a YUM repository, as explained in the instructions before. The ETICS YUM repository should be used to install the metapackage and start its certification.

OS/DAG/jPackage repositories (if there are external dependencies) should be also enabled and protected because external dependencies should be installed from those repositories. A repository can be protected by adding the line 'protect=1' in the .repo file. The yum-protectbase rpm is required to make yum honour the protect tag

The command to test the metapackage:

yum install METAPACKAGE

Remember that the YUM repository URL has to be included in the Savannah patch to be able to start the certification. In this case, it should always be the YUM repository of a registered build. No volatile repositories are allowed for the certification of a patch.

Note on permanent YUM repo files

There is a bug in ETICS when a build is submitted with registration. The repository stores the report and the YUM repo only if there is at least one new package that was not registered yet in the permanent area.

A bug fix is being implemented to allow YUM repos to be registered even if the build does not register anything new.

Workarounds until the bug is fixed:

• Change the metapackage age, so that it gets newly registered
• Ask the ETICS team to remove one package (i.e. the metapackage) from the permanent area so that the build will re-register it storing also the YUM repo
• Ask the ETICS team to remove the previous registered submission from the permanent area so that all the packages will be re-registered together with the metapackage and its YUM repo.

An internal patch is a patch that contains packages developed by product team A and that are used by other product teams as well. These packages are either:

• not part of metapackages produced by product team A (i.e. GFAL and lcg_utils in Data Management)
• part of metapackages produced by product team A, but none of those metapackages are scheduled for release soon, and therefore the internal patch is used to make the new packages available for the other product teams. (i.e. voms-api packages in VOMS)

When a product team wants to release an internal patch, they have to decide which platforms are going to be affected by the change and open a new patch PER PLATFORM in Savannah. The initial state of the patch is With Provider. The following fields are mandatory:

• Status: With provider
• Assigned to: responsible in the product team
• Summary: Internal patch for OS/ARCH package name
• Release Path: Internal
• gLite release: gLite 3.X
• Operating System: SLX
• Architecture: i386 or x86_64
• YUM repo file URL: URL to download the YUM repo file to be able to install the packages released in the patch
• gLite subsystem tag(s)/ETICS configurations: glite configuration names of all the packages released in the patch
• RPM names: list of URLs of the released packages
• Configuration changes: description if any
• Release notes: description
• List of attached bugs fixed in the patch

The main differences to patches opened before the product team model are:

• RPM names: The package names are provided with the URL from where they can be downloaded. Normally a URL in the ETICS repository when the packages are built in ETICS, otherwise, any other valid URL. The URLs can be easily copy and pasted from the ETICS build report -> Package List -> Package type: rpm and Download as text. Note that due to GGUS ticket 56917, the volatile repo appears in the URL instead of the permanent repo. This has to be changed manually until the problem is fixed.
• RPM names: Do not specify dependencies available in OS and DAG repositories (JPackage repository as well for 3.1).
• Release Path: It is used to mark the patch as an internal patch. Its value should be Internal.
• gLite subsystem tag(s)/ETICS configurations: it's very important to fill in this field since this is what the Integration Team will use to maintain a project configuration with the versions of Verified packages. A subsystem configuration should always be specified.
• YUM repo file URL: URL to download the YUM repo file to be able to install the packages released in the patch. The YUM repository specified here should be the one created after a registered build is launch. No volatile repositories are allowed for the certification of a patch. Use any of the URLs specified in the ETICS report where [protect=0].

The following fields are no longer needed but are still kept to acces the information in old style patches:

• The field Metapackage Changes is no longer needed. Set it to None.
• The Affected metapackages is no longer needed. Set it to None.

Product teams already know how to build packages with ETICS. The only thing to take into account that is different from the previous process, is that the Integration team is no longer creating patch repositories. Product teams should launch the ETICS build creating a yum repository and making sure Checkout run-time dependencies as well is also selected. This is done by selecting in the Build menu:

• • Generate YUM repository
  • --urlname patch-number (optional)
  • Checkout run-time dependencies as well

Once the packages are built in ETICS, the product team can move the patch into Ready for Certification

Remember that the YUM repository URL has to be included in the Savannah patch to be able to start the certification. In this case, it should always be the YUM repository of a registered build. No volatile repositories are allowed for the certification of a patch.

Note on permanent YUM repo files

There is a bug in ETICS when a build is submitted with registration. The repository stores the report and the YUM repo only if there is at least one new package that was not registered yet in the permanent area.

A bug fix is being implemented to allow YUM repos to be registered even if the build does not register anything new.

Workarounds until the bug is fixed:

• Change the metapackage age, so that it gets newly registered
• Ask the ETICS team to remove one package (i.e. the metapackage) from the permanent area so that the build will re-register it storing also the YUM repo
• Ask the ETICS team to remove the previous registered submission from the permanent area so that all the packages will be re-registered together with the metapackage and its YUM repo.

When launching an ETICS build, it's possible to create at the same time a YUM repository. The repo file is specified in the build report. Use the specified repo file together with the OS/DAG/jPackage repositories (if there are external dependencies) that will be used to certify the internal packages.

Please, note that OS/DAG/jPackage repositories (if there are external dependencies) should be protected because external dependencies should be installed from those repositories. A repository can be protected by adding the line 'protect=1' in the .repo file. The yum-protectbase rpm is required to make yum honour the protect tag

Then run: yum install METAPACKAGE

How to define a static dependency in ETICS

If you use the web interface:

• Open the configuration of the metapackage version you want to use in the workspace.
• Select and edit the Dependencies
• Browse through the org.glite project to select the component configuration you would like to use.
  • Make sure you select a component configuration and not just the component !
• When you click on the selected component configuration, a dialog box is opened with information about that component configuration.
• Click on the icon Add as a dependency displayed on the left corner of the box.
• This will add a new dependency on your metapackage configuration.
• Leave the dynamic box empty and define as Runtime.
• Save your changes

If you use the command line interface:

• Please, check the section Metapackage configuration, to know how to create an ini file defining your metapackage configuration.
• For the dependency section of the ini file, use the following syntax:

[Platform-default:StaticDependency]
PROJECT_NAME|COMPONENT_NAME = CONFIGURATION_NAME,R

Example:

[Platform-default:StaticDependency]
org.glite|glite-info-provider-ldap = glite-info-provider-ldap_R_1_2_4_1,R
org.glite|bdii = bdii_R_5_0_6,R

Obsoleting Packages

Sometimes you change the name of a package, or resesign a component and need to obsolete the old name / rpms.

In a normal .spec file you'd do this with Obsoletes:      bar. The ETICS equivalent is:

Add a property in the config

package.obsoletes

(from an ETICS Support email)