Change Management Policy

1. Introduction

This document describes the EMI policy to be followed to manage software changes in EMI software products.

There is a project deliverable describing the Software Maintenance and Support Plan, DSA1.1 [R1]. This policy is kept synchronised with DSA1.1.

2. Change Management Process

2.1. EMI Releases

The EMI distribution will be organized in periodic major releases, tentatively delivered once a year, providing a good balance between the conflicting requirements of stability and innovation.

An EMI major release is characterized by well-defined interfaces, behavior and dependencies for all included products, available on a predefined set of platforms. What is included in a new EMI major release is defined by the PTB and the implementation of the plan is coordinated by JRA1.

Backward-incompatible changes to the interface or to the behavior of a product that is part of the EMI distribution can be introduced only in a new EMI major release. Changes to interfaces that are visible outside the node where the product runs (e.g. a WSDL) need to be preserved even across major releases, according to end-of-life policies to be defined on a case-by-case basis.

The availability of a new major release of EMI does not automatically obsolete the previous ones and multiple major releases may be supported at the same time according to their negotiated end-of-life policies.

An EMI distribution includes all the EMI software products that are developed within the project and that have reached production quality. Within an EMI major release, only one major version of a given product is maintained. Four types of releases have been identified for a given product:

• Major Release: A major release for a product is characterized by a well-defined interface and behavior, potentially incompatible with the interface or behavior of a previous release. New major releases of a product can be introduced only in a new major release of EMI. The contents of a new major release are endorsed by the PTB and included in the project technical plan. The implementation is coordinated by JRA1.
• Minor Release: A minor release of a product includes significant interface or behavior changes that are backwards-compatible with those of the corresponding major release. New minor releases of a product can be introduced in an existing major release of EMI. The contents of a new minor release are endorsed by the PTB. The implementation is coordinated by JRA1. If the release is going to be introduced in an existing major release of EMI, the implementation is also supervised by SA1 in order to guarantee that the production quality and the backwards-compatibility are preserved.
• Revision Release: A revision release of a product includes changes fixing specific defects found in production and represents the typical kind of release of a product during the lifetime of an EMI major release.
• Emergency Release: An emergency release of a product includes changes fixing only Immediate-priority defects found in production, typically security-related. The PTB is always informed about the preparation of an emergency release and has the right to turn it down if it does not consider it an emergency.
• Repackaged release in the context of EMI 2: A repackaged release, for those products that have the same version in EMI 1 and EMI 2. In this case there is still the need to have the testing and certification, as the packages are created in the context of EMI 2, having the "environment" of EMI 2 with eventually different dependencies.

The list of current EMI software products is in section 4 of DNA1.3.2 - Technical Development Plan [R2]

2.1.1. EMI Versioning Schema

The following numbering convention must be followed to define the version of EMI products in EMI releases. Versions must be expressed in the form of major.minor.revision(-[age]), where:

• An increment in the major number reflects a change in the component interface or behavior, that can be backwards-incompatible.
• An increment in the minor number reflects a change in the component interface or behavior backward-compatible. It can include also bug fixes.
• An increment in the revision number reflects bugs fixes, with no new features.
• An increment in the age number reflects a rebuild due to change in the external dependencies or an emergency release for component-x.y.z, when the version component-x.y.z+1 is already available.

2.2. Release Tasks

Release tasks track new versions of EMI software products for each supported EMI major release and platform. They contain general information about the new version of the product and the list of changes that are introduced. Changes must be tracked in RfCs or development tasks. For more details about changes check section 2.3.

A single release task is used to track a new product release for all the supported platforms.

Note that in the case of EMI updates, new product versions can only be released if there are Immediate or High Priority RfCs.

2.2.1. Release Task Tracking

Release Tasks are tracked in the EMI Release Tracker [R3] in Savannah and they are created by the Release Manager upon request from the PTs. PTs must prepare the list of changes they want to include in the release task so that they are approved in the EMT. See section 2.3 for more details.

Release tasks should contain the following information:

• Unique identifier: This is automatically created by Savannah when a new task is generated.
• Should Start On: Ignore this field within the EMI context. It's a field that always appears in a Savannah task template. PTs can use for it internal purposes if they want to.
• Should be Finished On: The date by which the new product version should be released. This field should be defined only once and can't be modified.
• Postponed Until: In case a task needs to rescheduled, it's the new date by which the new product version should be released.
• Category: Type of tracker entry. Values can be Component or Release. Component refers to tasks tracking a single product version. Release refers to a set of new product versions. Release tasks are only used by the release manager to help organise the release work.
• Technical Area: Lists one or more of the four EMI technical areas that are relevant to the product/release, that is: Compute Area, Data Area, Infrastructure Area and Security Area.
• UMD Capability: It specifies one or more of the UMD capabilities provided by the product/release. For more information please check the UMD Roadmap [R4].
• Supported Platform: Platform supported by the new product version. For EMI2, in general, the value All - EMI2 should be used, as only one release task will be opened for all the platforms.
• Priority: By default is Normal unless it's an emergency release. In that case it should be set to Immediate.
• Release Type: Type of product release (Major, Minor, Revision, Emergency, Repackaged) as described in section 2.1.
• Product Team Free text, to be completed by the RM with the PT name, for easier searching by PT products.
• Status: see next section.
• Testbed Result: Result of the deployment of the new product version in the EMI Testbed. The values of this field are PASS/FAIL and it's defined by the Testbed Manager.
• Assigned to: savannah user name of the PT responsible person for the product. If it's a release, it's assigned to he release manager.
• Open/Closed: field that tracks whether the task is open and closed. It's automatically managed by Savannah.
• Discussion Lock: ignore this field. It's a field that is always set to unlocked and it's defined by the tracker manager.
• Release: EMI major release.
• Name: Name of the product or release.
• Component Version: version of the product/release.
• List of elements: A product/release can include several elements that are listed in the DNA1.3.2 - Technical Development Plan [R2].
• List of RfCs: URLs to the different RfCs and/or development tasks [R7] describing which defects are fixed or which new features are implemented in this release. They should follow this format:

              Features: 
               - Backward Incompatible:
               - Major:
               - Other:
  
              Bug Fixes:
               - Immediate
               - High
               - Medium
               - Low
  

  Where Backward incompatible and Major features should contain links to the development tasks, Other can contain internal RfCs not mentioned in the development tracker, less important. Ideally all RfCs features should be present in the development tracker, but exceptions are accepted.

• Package list: list of packages affected by the change and developed by the PT who owns the product. All supported formats must be listed. (i.e. source and binary packages and tarballs for all the supported platforms). In order to automate the QC verification, the following syntax has to be taken into account:

              Binaries:
               - SL5:
               - SL6
               - Deb6:
  
              Binary tarballs:
              Sources:
              Source tarballs:

  • Use # to differentiate real packages from comments in the field.
  • Package names possible syntax:
    • Complete package name: i.e.

                  SL5:
                  sherpa-1.0.0-1.sl5.x86_64.rpm
                  sherpa-1.0.0-1.tar.gz
                  sherpa-1.0.0-1.sl5.src.rpm
                  sherpa-1.0.0-1.sl5.x86_64.tar.gz
      
                  SL6:
                  sherpa-1.0.0-1.sl6.x86_64.rpm
                  sherpa-1.0.0-1.tar.gz
                  sherpa-1.0.0-1.sl6.src.rpm
                  sherpa-1.0.0-1.sl6.x86_64.tar.gz
      
                  Debian 6:
                  sherpa_1.0.0-1_amd64.deb
                  sherpa_1.0.0.orig.tar.gz
                  sherpa_1.0.0-1.debian.tar.gz
                  sherpa_1.0.0-1.dsc
                  sherpa_1.0.0-1_amd64.tar.gz
                 

    • Complete URL to the package.
• Documentation: links to the relevant documentation. Please, check the Documentation Policy [R5] to know which documents are mandatory and must be provided.
• Component Release Notes: Check the Documentation Policy [R5] to know what must be provided.
• License: link to the file describing the license under which the product is released.
• Extended Release Notes: link to the web page, if any, where an extended and more detailed version of the release notes can be found.
• Test Plan Link: link to the test plan used to test the product.

The information in the release task should be always valid for all the supported platforms. So before moving the task to certified, for example, you have to make sure that the product being released has been certified for all the platforms and that all the required information is available in the task for the verification step.

Although release Tasks are created in Savannah by the release manager, PTs must fill in the remaining fields as described in this section.

2.2.2. Release Task state transition diagram

The following diagram represents the set of states in the life of a release task:

• Open: the Release manager is responsible for creating the release tasks in Savannah to track the different scheduled releases. This should be done with the assistance of the JRA1 leader making sure the different development plans are fully covered in the release schedule. Check the Release Management Policy for more details on this. All release tasks should be either planned according to the different development plans, or in case they are needed because an unplanned change needs to be introduced, new release tasks will be created in Savannah at any time by the Release manager. At this moment, the Release manager together with the PTs, defines the Should be Finished on field and some other mandatory fields.
• Open to Certified: the PT moves the release task to Certified once the development, testing and certification of the new product version has finished. Before this transition happens, the PT has to make sure all the fields in the release task described above are properly filled in and a complete certification report has been attched to the tracker entry.
• Certified: the work of the PT has finished at this stage. This stage prompts the QC task force to evaluate the release task from the QA point of view.
• Certified to Ready for Testbed: The QC task force evaluates the release task. The QC task force includes the result of its evaluation in a verification report attached to the release task. At this stage, the Production Release Criteria [R6] is checked. If all mandatory checks are OK or only minor things are missing, the QC task force informs the PT to provide them and moves the task to Ready for Testbed.
• Certified to Open: The QC task force evaluates the release task. The QC task force includes the result of its evaluation in a verification report attached to the release task. At this stage, the Production Release Criteria [R6] is checked. If some of the mandatory checks fail, the QC task force puts the task back to Open informing the PT.
• Ready for Testbed : This stage prompts the Release Manager to prepare the repositories so that the Testbed Manager can deploy the new product version in the EMI Testbed. The communication between the Release Manager and the Testbed Manager is done using GGUS.
• Ready for Testbed to Deployed on Testbed: The Testbed Manager moves the task to Deployed on Testbed once he finishes deploying and testing the new packages in the EMI Testbed. If the testing is successsful he also changes the field Testbed Result to PASS; Otherwise he changes the Testbed Result to FAIL.
• Ready for Testbed to Open: The Testbed Manager moves the task to Open if the deployment of the new product version fails.
• Deployed on Testbed: This stage triggers once more the QC task force to verify that everything is correct from the QA point of view. If minor things were missing, they are checked at this step.
• Deployed on Testbed to Open: Even if mandatory verification criteria is checked by QC before, it could be the case that at this point some of the mandatory checks still fail. In this case, the QC task force puts the task back to Open informing the PT.
• Deployed on Testbed to Verified: The QC task force moves the release task to Verified after doing the evaluation of the information contained in the release task. If needed, because minor things were missing, the QC task force includes once more the result of its evaluation in another verification report attached to the release task.
• Verified: the work of the QC task force has finished at this stage. This stage prompts the release manager to continue the preparation of the release.
• Verified to Released: The release manager copies the packages in the EMI production repository and prepares the release pages.
• Released: The new product version is available in the EMI production repository and the release pages are now online. This state automatically closes the Savannah task. Note that at this point no modifications should be added in the tasks, not even comments.

2.3 Changes

Changes can be tracked in both RfCs and development tasks. In both cases, the URL to the corresponding change should be included in the corresponding release task as explained in the previous section.

2.3.1. Request for Change (RfC)

RfCs are tracked in different internal trackers selected by PTs. RfCs are motivated by:

• GGUS tickets where users report incidents or make requests.
• PTs when detecting defects that have been found internally or when introducing minor unplanned improvements.

During the maintenance of an EMI major release, in order to have a common view of the status of open RfCs, XML files exporting information from the different trackers are generated and processed by the SA2.3 task in order to provide a unique report. This report summarises the status of Immediate and High priority RfCs and it is produced every week for the EMT as explained in the Release Management Policy [R8]. For more information about the different trackers and the XML files, please check the Appendix on EMI Tracker Mappings.

An RfC should be evaluated as soon as possible by the PTs to accept it or reject it. A period of two weeks has been defined to carry out this evaluation. Immediate or High priority RfCs must be then approved by the EMT before they can be included in a release task.

If the same RfC is going to be applied to different EMI major releases, then an RfC is created for each EMI major release.

2.3.1.1. RfC Tracking

A set of fields have been defined to provide information about an RfC. PTs must define these fields in the corresponding tracking tools. When this is not possible, a XML file can be generated containing the needed fields. This is described in the Metrics twiki page.

The mandatory fields are:

• Unique identifier/URL: a URL pointing to the RfC. The URL acts as a unique identifier for the RfC.
• Associated GGUS ticket: If applicable, one or more URLs pointing to GGUS tickets that have caused the opening of this RfC.
• Affected Product: According to the list of products defined in DNA1.3.2 - Technical Development Plan [R2].
• Affected EMI major release: EMI major release affected by the RfC
• Affected Platforms: Whether the RfC is platform-specific and, if so, which are the affected platforms. Possible values are SL 5, SL 6, Debian 6, All Linux, All, Noarch, Other.
• Priority of the change. The priority of the change is based on severity, impact, urgency and cost.
  • Immediate: The RfC needs to be addressed as soon as possible, in all affected EMI major releases. A release containing immediate- priority changes can contain only immediate-priority changes. Multiple immediate-priority changes can be included in the same release, provided that any change does not delay the release significantly.
  • High: The RfC will be addressed in a next planned release of the affected product, in all affected EMI major releases.
  • Medium: The RfC will be addressed in the release of the affected product that will be shipped with the next EMI major release. If Medium priority RfCs are ready by the time High priority RfCs are going to be released, they can also be released together as part of an update to an existing EMI major release.
  • Low: There is no target date for addressing the RfC.
• Defect vs Feature. To differenciate between software bugs and new features.
• Status of the change. It shows in which stage of the Change Management Process the RfC is. See the Change Status section below.
• Detection Area: The context in which the version of the product affected by the change is available. Possible values are:
  • Production: The RfC is opened after feedback received from the users who have used the product in a production environment.
  • Testing: The RfC is opened after feedback received from the team of people testing the product.
  • Development: The RfC is opened after feedback received from the team of people developing the product.
• Target EMI major release: The target EMI major release where the RfC will be available.
• Associated Test: Whether there is a test associated with the change. Possible values are Yes, No, NA.
  • If the change is a Defect, this field refers to a regression test.
  • If the change is a Feature, this field refers to a functionality test.

2.3.1.2. RfC state transition diagram

The following diagram represents the minimum set of states that should be present in any of the tracking tools chosen by the PTs:

• Open: The RfC is opened after all the necessary clarifications have been made. This may include discussions in a GGUS ticket to understand if there's an actual problem, internal discussions within the PT after a defect has been found or after an improvement has been proposed; etc.
• Open to Accepted: The clarification is complete and the RfC is accepted to be implemented.
• Open to Rejected: The clarification is complete and it's been decided in the end not to implement the RfC.
• Accepted: The RfC is ready to be implemented by the PT. This state triggers the implementation of the RfC within the PT.
• Rejected: The RfC won't be implemented and the PT should explain why it has been rejected.
• Accepted to Fixed: The implementation of the RfC has finished, that is, the code has been committed.
• Fixed: The code is committed. Once the packages are ready within the PT, testing of the new release can start, including the testing of all RfC within the release.
• Fixed to Not Tested: The PT is not able to test the RfC.
• Not Tested: The RfC can't be tested and the PT should explain why.
• Fixed to Tested: The PT tests the RfC by running the relevant tests. If the tests are succesful, the RfC can be moved to Tested.
• Tested: The RfC has been succesfully tested.
• Fixed to Accepted: The PT tests the RfC but the tests fail. This means the new feature/bug hasn't been properly fixed.
• Tested to Closed and Not Tested to Closed: Once the corresponding task where the RfC is included is released to the EMI production repository, the PT closes the RfC.
• Closed: The RfC is now available in the EMI production repository.

2.3.1.3. RfC for Security Vulnerabilities

In case of security vulnerabilities assessed as such by the EGI-SVG or by the PT - the RfC MUST reference the EGI-SVG internal private RT ticket. No description or details regarding the vulnerability should be present in publicly available trackers.

2.3.2. Development Tasks

Development tasks are tracked in the Development Tracker [R7]. They are maintained by the PTB and assigned to the different PTs. They track:

• High level user requirements.
• Technical objectives defined in the technical area work plans.

PTs addressing a development task must include a reference to it in the corresponding release task by adding the URL of the development task in the List of RfCs field.

2.3.2.1. Development Task Tracking

Development tasks are created by the PTB. Most of the fields are filled in by the PTB but PTs are responsible for keeping the status of the development task up-to-date by modifying the Planned to be finished and Percent complete fields and by posting short textual updates as comments, as the task implementation progresses. Additionally PTs need to modify the Status field as explained in the following section.

The field Associated Test has to be filled to show whether there is a test associated with the change. The possible values are Yes, No, NA.

2.3.2.2. Development Task state transition diagram

The following diagram represents the most important set of states in the life of a development task:

• Open: The development task is created by the PTB and assigned to the corresponding PT. The corresponding value in the Status field at this point is Not started.
• Not started to Ongoing: Status field change performed by the PT when the PT starts working in the implementation of the development task.
• Ongoing: The PT is working in the implementation of the development task.
• Ongoing to Done: Status field change performed by the PT when the PT commits the new code implementing the development task in the VCS system.
• Done: The code implementing the development task is committed in the VCS system.
• Closed: The development task is moved to Closed by the PTB once the corresponding release task has been released to production.

3. Contacts

EMI SA2

4. Table of References

Reference	URL
R1	DSA1.1. Software Maintenance and Support Plan https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDSA11
R2	DNA1.3.2 - Technical Development Plan https://twiki.cern.ch/twiki/pub/EMI/DeliverableDNA132
R3	EMI Release Tracker https://savannah.cern.ch/task/?group=emi-releases
R4	UMD Roadmap https://documents.egi.eu/public/ShowDocument?docid=272
R5	EMI Documentation Policy https://twiki.cern.ch/twiki/bin/view/EMI/EMISa2DocumentationPolicy
R6	EMI Production Release criteria https://twiki.cern.ch/twiki/bin/view/EMI/ProductionReleaseCriteria
R7	EMI Development tracker https://savannah.cern.ch/task/?group=emi-dev
R8	EMI Release Management Policy https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2ReleaseManagementPolicy
R9	EMI Metrics https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2MetricsGuidelines
R10	EMI Persistent Bug Tracking Collection https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2MetricsGuidelines#Persistent_Bug_Tracking_Collecti

5. Logbook