LCG Grid Deployment - gLite Pre Production Services - Pre Production Coordination

---

Author: PPS Coordination
First Published: 14-Feb-2007
Last Update: 2010-01-05 by Main.unknown
Last Content Review: 22-Jan-2009
Expiration: 31-Dec-2009 (after this date please contact the author)

---

---

This document is expired: please refer to the new ReleaseProcedure

---

PPS Release Procedures

Release Cases

Rule: Ideally each new update to the gLite middleware is supposed to be released in PPS first, to stay there for a given amount of time, available to the users; then to be delivered into production.

The amount of time the software spends in PPS is not strictly defined

• If no activity is scheduled or in progress on a service the service is moved to production with the next release
• If user testing is ongoing and producing useful results the stage in PPS can be stretched
• The components of one PPS update are treated independently, so delays in processing one component don’t affect the others (see picture below)

Inner details of the PPS stage of the release

Exception(s): In occasion of exceptional events, the standard flow can be overruled. We have categorized the "special" releases as follow:

• "Urgent" releases: (E.g. fixes for a disclosed vulnerability). These patches are released at the same time both in production and PPS (with priority to the production service)
• "High-priority" releases: (E.g. fixes for a broken service). These patches are released directly in production, upon the notion that, if a service is already broken, things cannot get worse. High-priority releases are generally issued following-up operational problems. The PPS is synchronised soon after.

Release Procedures

Certification → PPS (Standard Releases)

Summary:

The PPS Release Team gives a "green light" to the gLite Release Team for the update of the repository. The gLite Release Team updates the repository and sends out release notes. The release is processed by PPS (deployment test, internal distribution), A new "green light" is given for the next update.

The dates and times in the following procedure are given for reference only, in order to estimate how much the whole process may request. As the release to PPS is sequenced with a 'semaphore' approach, relevant actions can be done by the players as soon as the preconditions are verified.

Day-0 (Friday)

1. 10:00 :PPS-Support creates a GGUS ticket (template here) to track the PPS-update (this ticket will be referred to as 'GT-PPS').
2. 10:10 :PPS-Support sends a 'kick-off' email to the gLite Release Team with CC to GGUS. (This e-mail is now referred to as 'PPS-Update E-mail')
             NB: To update the GT-PPS ticket correctly the string GGUS-Ticket-ID: #nnnnn must be in the subject, where nnnnn is the number of the GT-PPS ticket.
   
   

Day-1 (Monday)

1. 10:00 : The gLite Release Team updates the CERN PPS middleware repository
2. 10:30 : The gLite Release Team send the URL to release notes by a Reply-All to the PPS-Update E-mail, with CC to pps-deployment-test@cern.ch; pps-apt-repository-update@cern.ch (sites doing the pre-deployment in PPS + administrators of the PPS middleware repository).
3. 10:45 : PPS-APT-Repository-Update checks the CERN middleware repository for inconsistencies and upon success triggers the pre-deployment test team to start by Reply-All to the PPS-Update E-mail.
4. 11:00 : The pre-deployment test team in PPS installs the release and reports about possible issues with release notes or software. GGUS tickets possibly opened should be sent in cc to pps-support@cernNOSPAMPLEASE.ch with mention of the PPS-update number somewhere in the body of the ticket. Any problem tickets possibly raised are linked to the GT-PPS ticket by PPS support
   
   • The final report should be sent as Reply-All to the PPS-Update E-mail, stripping the previous content and adding yaim-contact@cern.ch in CC
5. Out of the tickets and the report, PPS-support will
   • create Savannah bugs where appropriate, to be reviewed before the release is moved forward to production.
   • include/link on GT-PPS the list of open bugs related with the issues encountered during the Update.
   • link the open bugs to the released patches to this Update
   • insert bug numbers are inserted in the pre-deployment test summary report
   • communicate bugs to the gLite Release Team using this template

Day-2 (Tuesday)

1. 11:00 : PPS-Support requests the mirroring of the repository at CNAF using this template
2. 11:10 : PPS-Support creates the PPS delivery bulletin (including pre-deployment test reports) and checks it.
   • The delivery bulletin is a web page containing information previously included in the broadcast such as release notes, how to provide feedback via GGUS and so on. The bulletin is started automatically via a script, which creates also a text file with the broadcast to be sent later on, already updated with the correct links
   • log to lxplus
   • copy the two text files with the test reports attached to the report e-mail to your home directory
   • > /afs/cern.ch/project/gd/egee/www/preproduction/release/scripts/preparePPSBulletin.sh update_number summary_report_file complete_report_file
     e.g.
     
   • > /afs/cern.ch/project/gd/egee/www/preproduction/release/scripts/preparePPSBulletin.sh 32 ~/summary-report-glite_3_0_2-r31.txt ~/complete-report-glite_3_0_2-r31.txt
   • You may want to write something of your own (e.g. comments on the pre-deployment tests) in the letter.htm file available in /afs/cern.ch/project/gd/egee/www/preproduction/release/bulletin/gLite3.0.2-UPDATEnn
3. 12:00 : PPS-APT-Repository-Update will mirror the middleware repository to CNAF and when this is completed they will Reply-All to the PPS-Update E-mail.
4. 12:20 : PPS-Support publishes the PPS delivery bulletin and checks it.
   • log to lxplus
   • > /afs/cern.ch/project/gd/egee/www/preproduction/release/scripts/publishPPSBulletin.sh update_number
     e.g.
     
   • > /afs/cern.ch/project/gd/egee/www/preproduction/release/scripts/publishPPSBulletin.sh 32
     1 12:30 : PPS-Support announces the update to the PPS site admins via the CIC Portal broadcast tool. The output of the preparePPSBulletin.sh can be used at this purpose, which is prepared on the basis of (this template). The site admins are requested to check the bulletin to find further information concerning the release. Also publication on the CIC Portal news is requested. Any problem tickets raised are linked to the GT-PPS ticket.
5. 12:40 : PPS-Support copies the link to the bulletin in the CIC Portal into the GT-PPS and assigns it to "OCC"
6. 12:50 : PPS-Support change the status of the patches announced in the PPSReleaseNotes page to "In PreProduction"

Preparation for next iteration

1. 13:00 :PPS-Support opens a new GGUS ticket (template here) to track the next PPS-update (this ticket will be referred to as 'GT-PPS').
2. 13:10 :PPS-Support sends a 'kick-off' email to the gLite Release Team with CC to GGUS. (This e-mail is now referred to as 'PPS-Update E-mail')
             NB: To update the GT-PPS ticket correctly the string GGUS-Ticket-ID: #nnnnn must be in the subject, where nnnnn is the number of the GT-PPS ticket.
   
   

Day-5 (Friday)

1. 10:00 [optional] :PPS-Support announces the update to the PPS Users via the CIC Portal broadcast tool (click here for a template) with glite-announce@cernNOSPAMPLEASE.ch in CC . Also publication on the CIC Portal news is requested. The users are requested that any problems found are reported through GGUS. Any problem tickets raised are linked to the GT-Prod ticket.
2. 10:40 : PPS-Support copies the link to the broadcast in the CIC Portal into the GT-PPS ticket

PPS → Production

Summary:

The PPS Release Team periodically recommends to the gLite Release Team a list of components which are rated mature for production. The list constitutes the starting point for the next release to production.

Thursday

1. 10:00 : PPS-Support creates GGUS ticket 'GT-Prod' (this will record the history of the release). The GT-Prod ticket will be assigned to OCC. Click here for a template. The GT-Prod ticket must be linked to the GT-PPS ticket(s)
2. 10:05 : PPS-Support looks in GGUS for the earliest, open GT-PPS ticket. PPS-Support checks if there are any open, linked problem tickets.
   1. If there are problem tickets linked to the GT-PPS ticket then PPS-Support verifies the status with the submitter(s).
      1. If no response is received from the submitter then it is assumed that a problem with the patch still remains.
3. 11:00 : PPS-Support create an e-mail (click here for e-mail template) containing the list of patches which are proposed to go to production. (This e-mail is from now on referred to as the Prod-Update E-mail). Note that in order for the GT-Prod ticket to be updated correctly, the string "GGUS-Ticket-ID: #nnnnn" must be in the subject of the Prod-Update E-mail (where nnnnn is the number of the GT-Prod ticket).
   
   Monday
4. At the EMT : The list of proposed patches is discussed and finalized.
   1. After the EMT : If there are no problem tickets associated with the patches in the release then PPS-Support sends a confirmation to the gLite Release Team by Reply-All to the Prod-Update E-mail. This template can be used at that purpose.
   2. After the EMT : If there are problem tickets associated with the patches in the release then PPS-Support remove any broken patches from the middleware update, amend the patches in Savannah accordingly (e.g. change status and remove "pps release" label...) and remove the linking of the problem ticket to the GT-PPS ticket _(currently not possible in GGUS)_. PPS-Support does a Reply-All to the Prod-Update E-mail in order to update the GT-Prod ticket.
   3. Unless an explicit roll-back communication is sent out by PPS-support by the end of the day, the list is to be considered implicitely accepted, and the gLite Release Team is authorised to proceed with the release in the following exceptional cases.
      • The EMT is canceled
      • Either the gLite Release Team or PPS-support cannot attend the meeting.
5. The gLite Release Team creates the meta-packages, web pages, etc. for the release of the update to production.
6. 10:00 : The gLite Release Team replies to the Prod-Update E-mail indicating that the 'new' release pages are ready for review. A link to the temporary repository should also be included to allow the release testing team to download the new rpms. A template for the reply can be found here
7. 10:10 : the members of the Release Testing Team acknowledge the request by Reply-All to the Prod-Update E-mail specifying whether they are going to perform the test or not.
   There is no defined timeout for the production of release testing results. However it is reasonable to expect that a test, once started, should not exceed the duration of one working day. The gLite release manager can decide whether to wait for the results or not at his/her discretion.
8. [in parallel]PPS-Support gives feedback on the release pages. Feedback must be given by Reply-All to the Prod-Update E-mail.
   • Timeout for giving feedback is 4 hours after the announce. After that time, if no exceptions were raised, the gLite Release Team is free to proceed with the update.
   • PPS-Support goes to each patch and checks the status of all bugs associated with the patch (i.e. bug is dependent on the patch or vice versa). Any bugs should all be either in "Ready for Review" or "Fixed". If any bugs do not have one of these statuses then an e-mail will be sent to the gLite Release Team requesting that the bug status be updated.
     Before each patch is released to production, PPS-Support must check and agree on the statuses of all bugs associated with the patch.
9. The gLite Release Team publishes the updated release pages, updates the production middleware repository, announces the update to the production middleware via the CIC portal broadcast tool (the template is here).
10. PPS-Support copies and pastes the link to the news in the CIC portal into the GT-Prod ticket. They then close the GT-Prod ticket.

Certification → Production (Fast-tracked Releases)

Summary:

The decision to make a fast-tracked release normally follows a very serious issue affecting the production system (e.g. sever security incident or critical break of functionality). The deployment method of a fast-tracked release is generally undefined, but in general it requires some formal steps in the standard release procedure to be skipped. The Operations Coordination Centre (occ-grid-support@cern.ch) and the gLite Release team (gd-release-team@cern.ch) together are ultimately responsible for the approval of the update as well as of the deployment methods.

1. The representatives of OCC, gLite Release Team and eventually other "experts" meet in order to agree on the need for a fast-tracked release (vs. a roll-back). If the meeting approves, the release continue with the next steps.
2. The gLite Release Team sends a 'fast-release preview' e-mail to PPS-support to inform that some patches are being fast-tracked to production and eventually to request a test to the Release Testing Team.
3. PPS-Support creates GGUS ticket 'GT-Prod' (this will record the history of the release). The GT-Prod ticket will be assigned to OCC. Click here for a template.
4. PPS-support links the GT-Prod ticket to the next available GT-PPS ticket
5. PPS-Support does a reply-all to the 'fast-release preview' e-mail including the original message
   • The template for the reply is available here. It contains the following modifications.
   • The subject is modified to include the GT-Prod ticket ID. In order for the GT-Prod ticket to be updated correctly, the string "GGUS-Ticket-ID: #nnnnn" must be in the subject of the Prod-Update E-mail (where nnnnn is the number of the GT-Prod ticket).
   • The header of the e-mail ('TO:','CC:' and 'BCC:' boxes) is populated using the same addresses defined in the list of patches for production. In particular this triggers the notification to the Release Testing Team
   • The body contains an acknowledment addressed to the Release Testing Team and the gLite Release Team with the request to test this production release and to include the patches in the next PPS release, respectively.
   • This e-mail is from now on referred to as the 'Prod-Update' E-mail).
6. The gLite Release Team publishes the updated release pages. For a fast-tracked release, although a standard release note may be not initially available, there MUST be at least a note in the affected update section of the release pages informing the users of the content and scope of the fast-tracked update (reason of the update, affected services, rpms, special instructions, etc.). Complete release notes have however to be produced within the next regular update.
7. OCC announces the update to the production middleware via the CIC portal broadcast tool (the template is here).
   
8. PPS-Support copies and pastes the link to the news in the CIC portal into the GT-Prod ticket. They then close the GT-Prod ticket.

[PPS internal]: Special actions for fast-tracked releases

Summary: Specific actions to be done for PPS releases corresponding to fast-tracked releases to production

1. PPS-support finds the corresponding GT-PROD ticket and links it to GT-PPS.
2. PPS-support follows the procedure for standard releases starting from day-2, Point 1 (request of mirroring). Actions dealing with pre-deployment testing can be skipped.
   • * IMPORTANT EXCEPTION: waiting for the "prepare" script to be improved, in order to create the release bulletin (day-2, Point 2) PPS-support should use the script
     • > /afs/cern.ch/project/gd/egee/www/preproduction/release/scripts/preparePPSBulletinFastTrack.sh update_number [summary_report_file complete_report_file]
       e.g.
       
     • > /afs/cern.ch/project/gd/egee/www/preproduction/release/scripts/preparePPSBulletinFastTrack.sh 32

Templates and examples for release process

Template for GGUS tracking tickets 'GT-PPS'

E-Mail: pps-support@cern.ch
CC to: pps-support@cern.ch
VO: None
Short Description: gLite Release > PPS: gL 3.x.y PPS-Unn

Describe Your Problem:

This ticket is used to track the history of the release of a middleware upgrade.

To the TPM: This ticket should be assigned to "Pre-Production System"

Best Regards,
        PPS Coordination Team 

Type of problem: Other
Priority: less urgent

PPS Update kick-off e-mail

To: gd-release-team@cern.ch
CC: pps-support@cern.ch; helpdesk@ggusNOSPAMPLEASE.org; pps-deployment-test
Subject: gLite 3.*x* release: PPS Update nn (GGUS-Ticket-ID: #GT-PPS.ID)

Body:

Dear gLite Release Team,

              
The PPS is ready to receive gLite3.n.nUpdate nn.

When you are ready please:
1) proceed with the update of the CERN PPS middleware repository
2) Send the URL with the release notes via Reply-All to this e-mail
   - Please add in CC the pre-deployment team and the PPS middleware repository team
   - in particular make sure that the pre-deployment team had all the references
     needed for testing.


(As the release is tracked in a GGUS ticket and as this e-mail will be automatically attached to that ticket,
when you reply please strip the history out of your e-mail.)


Best regards,
          PPS Coordination Team

Notification of PPS for fast-tracked releases (Opening GT-PPS)

To: pps-support
CC: pps-apt-repository-update; helpdesk@ggusNOSPAMPLEASE.org; pps-deployment-test
Subject: gLite 3.x.y PPS Update nn
Body:

Dear PPS,

the PPS repository has been updated to align the PPS repository with the latest production release (3.x.y) Update kk. 

[optional details]

The details for the release can be found at:
           
[link to release notes]

This e-mail will create a GGUS ticket which should be assigned to "Pre-Production System"

Regards,

    The gLite Release Team

Request to CNAF for PPS Repository update

To: people in REPLY-ALL; pps-apt-repository-update@cern.ch
CC: pps-support@cern.ch; helpdesk@ggus.org
Subject: Re:gLite release: PPS Update nn on DD-MMM-YY (GGUS-Ticket-ID: #GT-PPS.ID)
Body:

Hi [Danilo|Daniele]

Based on the results of the pre-deployment tests and of the preliminary checks on the middleware repository,
We reckon that the release can be moved forward to PPS.
              
Please update the PPS middleware repository at CNAF.

When you have done please reply-all to this e-mail

For sake of readability it is preferable that you strip the copy of previous message from you reply.
The history of this release is however logged in the GGUS ticket referenced in the subject

Best regards,
          PPS Coordination Team

List of bugs opened after pre-deployment test

To: gd-release-team@cern.ch
CC: pps-support; helpdesk@ggusNOSPAMPLEASE.org; pps-deployment-test
Subject: gLite Release > Production: gL 3.0 Update nn:bugs from PPS (GGUS-Ticket-ID: #GT-PROD.ID)
Body:

Dear gLite Release Team,

Please check the new version of the pre-deployment testing summary report

http://www.cern.ch/pps/release/testreports/gLite3.x.y-UPDATEnn/summary-report-glite_3_0_2-rnn.txt

updated with the links of new related bugs opened


Best regards,
          PPS Coordination Team

Template for GGUS tracking tickets 'GT-Prod'

Short Description: gLite 3.x Unn > Production:
Describe Your Problem:

 This ticket is used to track the history of the release of a middleware upgrade.

To the TPM: This ticket should be assigned to "OCC"

Best Regards,
        gLite Release Team 

E-Mail: occ-grid-support@cern.ch
CC to: pps-support@cern.ch ; gd-release-team@cern.ch ;
VO: None
Type of problem: 'Deployment-other'
Does it affect the whole:(nothing) Priority: Less urgent

Template for announcing the release of the update to the PPS (using the CIC Portal broadcast tool)

TO: CIC-on-Duty, OSG, ROC managers, PPS site admins,
CC: project-eu-egee-pre-production-service@cern.ch, glite-announce@cern.ch
SUBJECT: Release of new update to PPS: gLite3.0.2-PPSu nn and gLite3.1.0-PPSu nn

Publication in the news

MESSAGE:

Dear PPS site Administrators,

PPS-update nn is now available for pre-production sites

please find all the details in the bulletin
http://www.cern.ch/PPS/index.php?dir=./release/bulletin/gLite3.0.2-UPDATEnn/&
 
Best regards,
        PPS Coordination Team

Template for announcing the update of the PPS to users (using the CIC Portal broadcast tool)

TO: CIC-on-Duty, ROC managers, VO managers, ALICE VO users, CMS VO Users, Pre-Production site admins
CC: support-eis@cern.ch, project-eu-egee-pre-production-service@cernNOSPAMPLEASE.ch, glite-announce@cern.ch
publish in the CIC Portal news
SUBJECT: gLite 3.02 update nn available in PPS

MESSAGE:

Dear Users of the EGEE Grid Pre-Production Service,

PPS-update nn is now available in PPS

The release notes with list of fixed bugs and new features are available in 
 (copy the link from the PPS-Update E-mail or the GT-PPS ticket)

    https://twiki.cern.ch/twiki/bin/view/EGEE/PPSReleaseNotes_302_PPS_Update_NN

Please note that the upgrade at the following PPS sites is not completed yet at the time of this announcement, but it will be as quickly as possible:

* site
* site
* site

You are welcome to submit via GGUS tickets comments and issues you may have about the service.

Best regards,
	PPS Coordination Team

List of patches to go to production

To: gd-release-team@cern.ch ; helpdesk@ggus.org ; project-eu-egee-middleware-emt
CC: pps-support ; occ-grid-support ; sa1-release-test
BCC: project-eu-egee-pre-production-service
Subject: gLite3.x Unn: patches  to go to production (GGUS-Ticket-ID: #GT-Prod.ID)
Body:

Dear gLite Release Team,
              
Based on feedback from PPS sites and users, we think that the following
patches can go to production with the next release:

# available on the linked GT-PPS ticket(s)
nn1: patch description 1 - 3.0.2 PPS Update NN 
nn2: patch description 2 - 3.0.2 PPS Update NN
nn3: patch description 3 - 3.0.2 PPS Update NN
...

***
Please reply-all to this message as soon as possible indicating a proposed date for the roll-out of this update.
***


Please while editing the release notes, have a close look at the installation/configuration issues found in PPS by the pre-deployment test team while testing the corresponding PPS update(s).
The test reports can be found in
http://www.cern.ch/pps/index.php?dir=./release/testreports/&

# remove if not relevant
Please take also into account the open issues and recommendations pointed out during the pilot "wrap-up" meeting 
http://indico.cern.ch/conferenceDisplay.py?confId=xxxxx

PPS SITE MANAGERS:
If you have a problem with any of these patches moving to production,
please let us know before tomorrow 16:00 UTC (17:00 Swiss local time).

During the EMT meeting, next Monday, possible issues with this list will be
discussed.  If the EMT meeting is canceled or you don't receive any
further communication on this subject by today, please feel free to
proceed with the preparation of the release.


Best regards,
        EGEE Grid Operations team

List of patches to go to production (confirmation)

To

gd-release-team@cern.ch; helpdesk@ggusNOSPAMPLEASE.org; project-eu-egee-middleware-emt

CC

pps-support@cern.ch

Subject

RE: gLite release: patches to go to production on DD-MMM-YY (GGUS-Ticket-ID: #GT-Prod.ID)

Body

Dear gLite Release Team,

After the discussion in the EMT meeting we confirm the validity of the list previously given.

Please proceed with the packaging.

Best regards,
Grid Operations

Template for announcing the release of the update to the production service (using the CIC Portal broadcast tool)

TO:

 Publish in the CIC Portal news, CIC-on-Duty, OSG, ROC managers, VO managers, Production site admins,
support-eis@cern.ch,glite-announce@cern.ch 

SUBJECT:

 Release of UPDATE nn to gLite 3. x. Priority: [Normal/HIGH/**URGENT**] 

MESSAGE:

Dear members of the EGEE Grid Production Service,

UPDATE nn for gLite 3.x has been released to the EGEE Grid production service.
The priority of the updates is: [Normal/HIGH/**URGENT**]

All details of the update can be found in:
  http://glite.web.cern.ch/glite/packages/R3.x/updates.asp
(services supported on 32b architectures)
and
http://glite.web.cern.ch/glite/packages/R3.x/x86_64/updates.asp
(services supported on 64b architectures)

The updated middleware services with new version numbers are given on the Updates web page.


For more detailed information including fixed bugs, updated RPMs, configuration changes and
how to deploy, please go to the 'Details' link next to each service on the 'Updates' web page.


** PLEASE ALSO READ THE NOTES AND 'KNOWN ISSUES' SECTION FOR EACH SERVICE. **


Note that the UI/WN tarball is available from the individual service pages
(click on the service name on the main page). 

Remember to report any issues with this set of updates using GGUS:  www.ggus.org



Best regards,
    The gLite Middleware Teams

Delivery of a preview of the next release to production

To: pps-support ; sa1-release-test
CC: occ-grid-support ; project-eu-egee-middleware-emt ; helpdesk@ggus.org
Subject: RE:gLite3.x Unn: Update to go to production on DD-MMM-YY (GGUS-Ticket-ID: #GT-Prod.ID)
Body:

Dear all,

please find a preview on the production release notes here:
http://glite.web.cern.ch/glite/packages/R3.x/updates_new.asp
http://glite.web.cern.ch/glite/packages/R3.x/x86_64/updates_new.asp

To get the details of the update, please use "Details" in the service node listing. 
You will not see any update on the service node main page.

the according RPM packages can be found here:
http://linuxsoft.cern.ch/EGEE/gLite/R3.x_new/
(this location will be renamed when the update is officially released)

Any input/feedback provided as a reply to this mail would be valuable and is therefore welcome. 

Regards,
The gLite Release Team

Template for notification of fast-tracked release to production

To: pps-support
Subject: gLite3.x Unn: Fast tracked release to go to production on DD-MMM-YY
Body:

Dear all,

A fast-tracked release to production was decided in agreement with the OCC due to the following issue observed in production
----------------------------------------
-- describe the issue here if possible --
----------------------------------------

the updated packages [already deployed in production] are referenced by the following patches:
----------------------------------------
-- list of patches here                      --
----------------------------------------

-- REMOVE IF NOT RELEVANT --
The release testing the can find a preview on the production release notes here:
http://glite.web.cern.ch/glite/packages/R3.x/updates_new.asp
http://glite.web.cern.ch/glite/packages/R3.x/x86_64/updates_new.asp

To get the details of the update, please use "Details" in the service node listing. 
You will not see any update on the service node main page.

the according RPM packages can be found here:
http://linuxsoft.cern.ch/EGEE/gLite/R3.x_new/
(this location will be renamed when the update is officially released)

Any input/feedback provided as a reply to this mail would be valuable and is therefore welcome. 
----------------------------------

Regards,
The gLite Release Team

Acknowledge to a fast-tracked release

To: gd-release-team@cern.ch ; helpdesk@ggus.org ; project-eu-egee-middleware-emt
CC: pps-support ; occ-grid-support ; sa1-release-test
BCC: project-eu-egee-pre-production-service
Subject: Re:gLite3.x Unn: Fast tracked release to go to production on DD-MMM-YY(GGUS-Ticket-ID: #GT-Prod.ID)

Body:

Dear gLite Release Team,

The PPS acknowledge the need for this update ot be fast-tracked to production. 

Please make sure that the relevant pacthes are included in the next update to PPS, namely gLite3.x.y  PPS-Updatenn

-- REMOVE IF NOT RELEVANT --
*SA1 Release Testing Team*:

 Please go on with the test of this update if it is relevant for you.
----------------------------------

Regards,

The PPS Coordination Team

> -----Original Message-----

Please include the ORIGINAL MESSAGE here

Template for announcing a fast-tracked update to the production service (using the CIC Portal broadcast tool)

TO:

 Publish in the CIC Portal news, CIC-on-Duty, OSG, ROC managers, VO managers, Production site admins,
support-eis@cern.ch,glite-announce@cern.ch 

SUBJECT:

 Fast-Tracked Release of UPDATE nn to gLite 3. x. Priority: [HIGH|/**URGENT**] 

MESSAGE:

Dear members of the EGEE Grid Production Service,

A fast-tracked release to production was decided in agreement with the OCC due to the following issue observed in production
--------------------------------------------------------------------------------------------------------
-- describe the issue here taking it from the initial notification e-mail or the GT-PROD GGUS ticket --
--------------------------------------------------------------------------------------------------------

The priority of the updates is: [HIGH/**URGENT**]

All details of the update can be found in:
  http://glite.web.cern.ch/glite/packages/R3.x/updates.asp
(services supported on 32b architectures)
and
http://glite.web.cern.ch/glite/packages/R3.x/x86_64/updates.asp
(services supported on 64b architectures)


-- REMOVE IF NOT RELEVANT --
the updated packages are referenced by the following patches [|distributed with the following rpms]:
----------------------------------------
-- list of patches/rpms here                      --
----------------------------------------



-- REMOVE IF NOT RELEVANT --
For more detailed information including fixed bugs, updated RPMs, configuration changes and
how to deploy, please go to the 'Details' link next to each service on the 'Updates' web page.

-- REMOVE IF NOT RELEVANT --
Note that the UI/WN tarball is available from the individual service pages
(click on the service name on the main page). 

Remember to report any issues with this set of updates using GGUS:  www.ggus.org



Best regards,
    The gLite Middleware Teams