ATLAS Release Policies
Author: David Quarrie
24 Nov 2003
Revised: 22 Mar 2005
Table of contents
1. Introduction
ATLAS Software Development utilizes a multi-tier hierarchy of releases:
- Nightly Releases
- Developer Releases
- Production Releases
- Bug-fix (or Patch) Releases
These, together with the policies associated with each one, are described in the following sections. Information about the Nightly Releases is available online at the Nightly Builds (NICOS) Page and for the Developer, Production and Bug-fix releases on the Offline Releases Pages.
Distribution kits are created for many of these releases, and this document also covers policies for those.
2. Platforms
ATLAS currently supports the following platforms and compilers:
Scientific Linux CERN 3 (SLC3) with gcc 3.2.3
RedHat 7.3 with gcc 3.2
This list is reviewed and is liable to change. It is recognized that it is currently inadequate. Earlier attempts to build on Solaris were aborted when it became clear that the compiler was (at that point) inadequate.
In general policy is to be consistent with the set of platforms and compilers that is used for testing of LCG products, but to support a subset of those. This set is arrived at by discussions with all LHC experiments in the LCG Architects Forum. The currently proposed ATLAS subset is:
RedHat 7.3 with gcc 3.2 (to be dropped Summer 2005)
SLC3 with gcc 3.2.3
SLC3 with gcc 3.4.3 (32-bit mode) (available Summer 2005)
SLC3 with gcc 3.4.3 (AMD64 64-bit mode) (available Autumn 2005)
Mac OS X gcc 3.3 (ongoing at a low level)
Current SLC3/gcc3.2.3 is the primary platform. The situation and timescales with other platforms is given above. Only unofficial efforts are currently underway to build on the Mac OS X platform.
The set of platforms and compilers is revisited every 12 months or so.
It should be noted that binary releases support a broader range of platforms and compilers (e.g. SUSE Linux), but are determined by the ICB and the CMB.
The following compiler options are available:
debug (dbg) Debug symbols, unoptimized (-O0???)
optimized (opt) Optimized (-O2)
profiled (pro) Optimized (-O2) and profile information (-pg)
3. Releases
3.1 Nightly Releases
A total of 7 nightly releases are created, with each one having a lifetime of 7 days before it is overwritten. The releases are created for the debug and optimized compiler configurations on the list of supported platforms and compilers as listed above.
The list of packages and their associated tags for the nightly builds is determined by the ATLAS Tag Collector. In normal operations, a single set is used, corresponding to the forthcoming "open" release (being either a Developer or Production release as described in the following sections). However, following a Production release, one or more Bug-Fix releases may optionally be built, in which case packages and tags can be associated with any of the currently opened releases.
Because of inconsistencies in tagging, the Container packages (e.g. Event, Database) are not used to determine the set of packages and tags for the nightly releases, but the set is taken directly from the Tag Collector.
Depending on the timing within a Developer or Production Release cycle, some of the packages will be "locked" within the Tag Collector, meaning that their tags can only be updated by the Release Coordinator (see later section) or Librarian, and not by the package author.
In order to allow for remote sites to anticipate the need for the installation of new external software, the External packages are always locked, and permission must be obtained from the Release Coordinator or SIT Coordinator in order to either add new external packages, or new versions of existing external packages.
Nightly builds are fully automatic, and there is no human intervention if the build fails, either because of a package figuration problem, or an infrastructure problem (e.g. AFS error). To maximize the functionality of the nightly releases, it is the developer responsibility to only put into the nightly code that has been checked to compile and run against the most recent nightly. Backwards incompatible changes should be coordinated with the clients before they are committed, and with the Release Coordinator if more than one top level package is involved.
One of the purposes of nightly releases is to allow integration testing and migration. Thus is not normally expected that nightly releases will be fully functional, and there will be periods when major components are broken (e.g. when migrating core packages to a new API when the clients have not all been updated accordingly). However, this philosophy will be modified somewhat in the lead up to a Production release (see later section).
Nightly releases have names atlrel_0 through atlrel_6 and are located at:
/afs/cern.ch/atlas/software/dist/nightlies/rel/<release>
in the CERN AFS filesystem.
If multiple branches are open, the number of nightly releases on some branches might be reduced from the default of 7 if not enough disk space is available.
3.2 Developer Releases
Developer releases occur approximately every three (3) weeks, driven by the set of packages and tags in the Tag Collector. In contrast to the Nightly Releases, the information from Container packages is used, and consistency in the packages and tags is enforced between the between the set derived directly from the Tag Collector, and the set derived from the Container packages.
A subset of the Container packages, the so-called "Core" packages, is locked approximately 1 week prior to the Developer Release in order to provide additional stability for other packages that depend upon this Core set.
A best effort is made to keep to the 3 week cycle, delays being exceptional. In general, only in the event of an infrastructure problem (e.g.AFS or network problems) will a build attempt be repeated. As a consequence, to maximize the functionality of developer releases, developers are strongly advised to commit major changes into the nightly releases towards the beginning of a developer release cycle, and restrain themselves to only bug fixes in the last few days before the developer release is built. This is especially important for packages with many clients.
Developer releases are subjected to limited QA testing.
Developer releases have names of the form i.j.0, where i and j are positive integers (e.g. 6.1.0, 7.2.0). They are located at:
/afs/cern.ch/atlas/software/dist/<release>
in the CERN AFS filesystem.
From release 10.0.0 onwards, the strategy for developer releases has been changed such that each developer release is a snapshot of a nightly release, taken at 3-4 week intervals and that no distribution kit is created for such releases. The other main difference is that, as container packages are not present in the nightlies, they are also not present for the new developer releases.
3.3 Production Releases
Production releases occur approximately every 4-6 months, normally (but not necessarily) being targetted towards high level milestones such Data Challenges, Physics Workshops or Testbeam operation. The build procedure is essentially identical to that of the Developer Releases, the major differences being that there is more flexibility in delaying the release build if required functionality is missing, and in the amount of QA testing that the release is subjected to. Acceptance of the release is dependent on a positive report card from the relevant communities, typically the Physics Coordination, or High Level Trigger etc.
While there is more flexibility in delaying the release if required functionality is missing, the Release Coordinator has the ultimate authority to proceed.
The general philosophy for nightly releases becomes more restrictive in the lead up to a Production release. Thus, in the last week or so (at the discretion of the Release Coordinator) every attempt should be made to converge on a fully functional nightly release, leading up to the Production Release.
Production releases have names of the form i.0.0, where i is a positive integer (e.g. 6.0.0, 7.0.0). They are located at:
/afs/cern.ch/atlas/software/dist/<release>
in the CERN AFS filesystem.
A distribution kit (see last section) is built for each Production Release, which allows it to be installed and utilized at remote sites without AFS access.
3.4 Bug-Fix Releases
Bug fix releases can be associated with production releases if subsequent extended testing uncovers problems of a severity that the production release is deemed unusable for serious production. In this case, a bug fix release (or set of such releases) may be created. These are supported in the Tag Collector, being associated with their base release. In general only specific packages and tags are accepted, under the authority of the Release Coordinator.
Bug-fix releases have names of the form i.0.k, where i and k are positive integers, and i corresponds to the Production release (e.g. 6.0.1, 7.0.2). They are located at:
/afs/cern.ch/atlas/software/dist/<release>
in the CERN AFS filesystem.
It should be noted that until now bug-fix releases have only been associated with production releases. However, because of the time constraints associated with DC-2 and combined testbeam, it is expected that what are nominally developer releases will need to be used for production processing associated with these, in which case it is possible that bug-fix releases will be associated with developer releases. This is an exception to the normal policy however, and a repeat of this should be reviewed before being embarked upon.
A distribution kit (see last section) is built for each Bug-fix Release, which allows it to be installed and utilized at remote sites without AFS access.
4. Release Coordinator
The Release Coordinator has overall responsibility for the release builds, having a term of duty that extends over a single production release cycle (i.e. from one production release until the next one). They have ultimate decision making authority over whether to delay a release or reject late submissions, etc.
The Release Coordinator should make sure the (developer and production) releases deliverables are indeed delivered, by staying in contact with the developers and coordinators involved. Given the importance of functionality of the nightly builds and developer releases for the success of a production release, the Release Coordinator should regularly monitor the functionality of nightly release to make sure no serious problem remains unfixed. They should coordinate major changes to minimize disruption. They makes sure developers are informed about major changes, and about the status/usability of developer releases.
5. Distribution Kits
For each Production and bug-fix releases of ATLAS software, a distribution kit will created, that allows sites that don't have access to CERN AFS to install the software locally. Distribution kits are described in detail in the instructions for installing the s/w.
The ATLAS policy is that any application should function identically within the context of CERN AFS, and when built from the distribution kit. Every effort will be made to ensure equivalent functionality, and any divergence is treated as a bug.
6. Release Deletion Policy
DRAFT
This policy uses the concept of a stable production release. This is a production (e.g. 10.0.0) or bug-fix release (e.g. 10.0.2) that forms the culmination of a major development cycle and which is used for production purposes. In general it corresponds to the last bug-fix release in a bug-fix branch, but there are occasions when multiple such bug-fix releases are used for slightly diferent purposes.
In general nightly releases are deleted automatically during the normal release cycle. In exceptional circumstances a particular nightly release may be put aside and saved for a restricted period of time. Such nightlies will be deleted as soon as the next production stable release is available. No archive will be made of such releases.
Developer releases (i.j.0) will become candidates for deletion six months after the following stable production release is available.
The deletion policy for a production release (i.0.0) depends on whether any bug-fix releases are asssociated with it. If one or more bug-fix releases are associated with it (i.e. it's not a stable production release), the deletion policy is the same as for developer releases, and it will become a candidate for deletion six months after the availability of the corresponding stable production release.
The deletion policy for bug-fix releases (i.0.k) that have been superceded by later bug-fix releases (e.g. 9.0.1 when 9.0.4 is the stable production release) is the same as for production releases.
Production stable releases become candidates for deletion 12 months after they have been superceded by a subsequent production stable release.
Once a release becomes a candidate for deletion, it will be addded to a list of such candidates that is presented to the CMB every six months. They can request that the deletion be deferred until the next such presentation.
A CASTOR archive will be made of all releases prior to deletion apart from nightly releases.