This website is no longer maintained. Its content may be obsolete. Please visit http://home.cern/ for current CERN information.

How to prepare the export kits for an Anaphe release

Author: Dino Ferrero Merlino

Date: 18 Sept 2001

1. Introduction

Whenever an Anaphe release is produced, "export kits" must be prepared for distribution to external sites. For each officially supported platform an export kit is made available via AFS.

An export kit consists of:

A set of compressed tar files containing all the packages that are part of the release (including the commercial packages).
A set of scripts to install the packages and update the symlinks.
Documentation, i.e. a README.txt file containing instruction for the installation and the release notes in HTML format.

The export kits are located in the /afs/cern.ch/sw/lhcxx/export area and are further organized by platform:

> ls -l /afs/cern.ch/sw/lhcxx/export

drwxrwxrwx   5 root     root         2048 May  4 10:37 dec
drwxrwxrwx   6 root     root         2048 May  4 10:37 hp
drwxrwxrwx   5 root     root         2048 May  4 10:38 ibm
drwxrwxrwx   8 root     root         2048 May  4 10:38 lnx
drwxrwxrwx  10 root     root         2048 May  4 10:39 nt
drwxrwxrwx  29 root     root         2048 Jul 19 11:05 redhat61
drwxrwxrwx   5 root     root         2048 May  4 10:41 sgi
drwxrwxrwx   6 root     root         2048 May  4 11:56 share
drwxrwxrwx  12 root     root         2048 Jul 19 15:17 sol7
drwxrwxrwx  16 root     root         2048 May  4 10:41 sun

This is the content of a typical export kit:

> ls -l /afs/cern.ch/sw/lhcxx/export/redhat61/3.6.2/

-rw-r--r--   1 dinofm   dl          10081 Jul 19 14:21 3.6.2.html
-rw-r--r--   1 dinofm   dl           5365 Jul 19 14:21 README.txt
-rw-r--r--   1 dinofm   dl        5123149 Jul 19 12:32 heshared.tar.gz
-rw-r--r--   1 dinofm   dl       27701058 Jul 19 14:06 hespec.tar.gz
lrwxr-xr-x   1 dinofm   dl             20 Jul 19 15:18 nagC.tar.gz -> ../3.6.1/nagC.tar.gz
lrwxr-xr-x   1 dinofm   dl             20 Jul 19 15:18 objy.tar.gz -> ../3.6.1/objy.tar.gz
lrwxr-xr-x   1 dinofm   dl             22 Jul 19 15:18 ospace.tar.gz -> ../3.6.1/ospace.tar.gz
lrwxr-xr-x   1 dinofm   dl             20 Jul 19 15:19 pdom.tar.gz -> ../3.6.1/pdom.tar.gz
-rwxr-xr-x   1 dinofm   dl            577 Jul 19 14:21 sharedinst
-rwxr-xr-x   1 dinofm   dl           2777 Jul 19 14:21 specinstall

Notice how symlinks are used to avoid wasting space when commercial packages do not change among close releases.

Preparing the export kit actually means several steps:

Update the documentation (README.txt file and release notes in HTML)
Update the installation scripts if necessary (e.g. if a new tar file is added to the release).
Prepare the set of tar files (including the so called HEP distribution, a tarball of all non-commercial components that can be downloaded directly from the Web).
Test the kit by making a local installation and checking a subset of the packages.
Check the ACLs in the export area.
Update the Web documentation in the Anaphe site (including the download of HEP distribution).

Each step will be described in more detail after the next section, that describes the LHCXX package that stores in CVS all the information to build releases and export kits.

2. The LHCXX package

This package stores in CVS all the scripts, configuration files, makefiles and documentation related to the Anaphe releases and their export kit. This is the package structure:

-rw-r--r--   1 dinofm   dl          17554 Sep 19 11:07 GNUmakefile
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 admin
drwxr-xr-x   3 dinofm   dl           2048 Sep 19 14:43 config
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 doc
drwxr-xr-x   3 dinofm   dl           2048 Sep 19 11:07 etc
drwxr-xr-x  15 dinofm   dl           2048 Jul 27 15:12 export
drwxr-xr-x   3 dinofm   dl           2048 Sep 19 11:07 scripts

The config subdirectory contains configuration files for each release:

> ls config

2.0.0.mk  3.0.0.mk  3.2.0.mk  3.4.0.mk  3.5.5.mk  3.5.8.mk       3.6.0-pre2.mk  3.6.2.mk       3.6.3.mk  SOL7.mk
2.0.1.mk  3.0.1.mk  3.2.1.mk  3.5.0.mk  3.5.6.mk  3.5.9.mk       3.6.0.mk       3.6.3-pre1.mk  3.7.0.mk  pre-3.0.0.mk
2.1.0.mk  3.1.0.mk  3.3.0.mk  3.5.4.mk  3.5.7.mk  3.6.0-pre1.mk  3.6.1.mk       3.6.3-sec.mk

Every configuration file contains the list of package versions that make up the release, e.g.:

> cat config/3.6.3.mk 

NAGVERS=6.0
OBJY_VERS=6.1
OSPACE_VERS=2.1.0.1
QTVERS=2.2.4
CLHEPVERS=1.6.0.0
HEPODBMVERS=0.3.2.12
CONDB_VERS=0.2.2.1
HEPUTILVERS=0.6.0.0
HTLVERS=1.3.1.2
GEMINIVERS=1.3.0.1
HEPFITVERS=1.3.0.1
FMLVERS=1.2.0.0
CHBOOKVERS=1.0.0.0
NTUPTAGVERS=1.5.0.0
NTUPTAGHBVERS=1.1.0.0
NTUPTAGHEVERS=1.1.0.0
A_INTERFVERS=1.4.0.0
QPLOTTERVERS=1.5.0.3
A_ANALYZERVERS=1.2.0.3
A_ANNOTATIONVERS=1.3.0.1
A_FITTERVERS=1.2.0.5
A_FMLVERS=1.2.0.0
A_HTLVERS=1.3.0.7
A_HBKVERS=1.1.0.0
A_NTUPLEVERS=1.3.5.0
A_PLOTTERVERS=1.3.0.0
A_SCATTERVERS=1.2.0.2
A_VECTORVERS=1.5.0.0
LIZARDVERS=1.3.0.5
OTHLIBVERS=1.0.0.1

The export kits are built starting from such files.

The other important directory of the package, in this contest, is export, which contains a subdirectory for each Anaphe production release:

> ls -l export

drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 2.0.1
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 2.1.0
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 3.0.0
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 3.0.1
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 3.2.0
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 3.4.0
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 3.6.0
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 3.6.1
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 3.6.2
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 alpha00
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 may00
drwxr-xr-x   3 dinofm   dl           2048 Jul 27 15:12 obsolete

In all subdirectories we find a set of files related to that version of the export kits, e.g.:

> ls -l export/3.6.2

-rw-r--r--   1 dinofm   dl          10081 Jul 19 15:16 3.6.2.html
-rw-r--r--   1 dinofm   dl           7530 Jul 19 15:16 GNUmakefile
-rw-r--r--   1 dinofm   dl           5365 Jul 19 15:16 README.txt
-rwxr-xr-x   1 dinofm   dl            577 Jul 19 15:16 sharedinst
-rwxr-xr-x   1 dinofm   dl           2777 Jul 19 15:16 specinstall

These are, respectively

HTML release notes
A GNU makefile to produce the tar files
README.txt file (installation information)
Two shell scripts to install the kit

3. Updating the LHCXX package to produce new export kits

The first step in the procedure is to create a subdirectory in export to hold the data for the new set of kits. A reasonable rule of thumb is to start from the latest release and adapt it. So for instance to make the kits for Anaphe 3.6.3 we start from the subdirectory of 3.6.2:

cd  LHCXX/export
mkdir  3.6.3
cp  3.6.2/*  3.6.2
mv  3.6.3/3.6.2.html  3.6.3/3.6.3.html

Notice how the release notes html file has been renamed. At this point it's wise to update the CVS repository:

cvs add 3.6.3
cvs add 3.6.3/*
cvs commit 3.6.3

4. Updating the documentation

Release notes

Change all occurences of the release name (e.g all 3.6.2 should become 3.6.3.)
Verify the consistency of the Supported Platforms and Dependencies sections.
Update the Content section with the proper version numbers (taken from the Versions file of the release).
Verify the consistency of the Obsolete section.
Update the Release notes section with the relevant changes. Don't forget to update the HTML link to the previous version.
Check that all links in the page work properly.

`README.txt` file

Change all occurences of the release name (e.g all 3.6.2 should become 3.6.3.)
Verify the size estimates are still reasonable
Verify the consistency of the Requirements section.
Update the Kit content section with the proper version numbers (taken from the Versions file of the release).
If new platforms/compilers are used, update consistently all the hints in the file (e.g. ASIS-like platform names).

5. Updating the installation scripts

The kit contains two installation scripts:

sharedinst to install the shared tree (source code, scripts)
specinstall to install the specific tree (header files, librairies, binaries.

While the former doesn't change so often, you may need to modify specinstall when there are platform changes (new platform or compiler) or new tar files are added to the kit.

6. Prepare the set of `tar` files

This activity is repetead for each platform's kit. Basically the tar files are created by targets in the GNUmakefile makefile mentioned beforehand:

>ls -l export/3.6.3/GNUmakefile 
-rw-r--r--   1 dinofm   dl           7530 Sep 19 16:16 export/3.6.3/GNUmakefile

The most important targets are:

allkit (prepare all tar files and compress them)
linkkit (prepare tar files for non-commercial components and compress them)
sharedtar (prepare the tar file for the shared tree)
spectar (prepare the tar file for the specific tree)

6.1 Updating targets

Whenever packages are added/dropped the makefile targets should be modified. Most of the times this means adding/removing commands to the sharedtar and spectar targets. As an example, version 3.6.3 introduced three new packages:

CHBook
NtupleTag_HBook
NtupleTag_HepExp

The sharedtar target has been modified in this way:

	cd $(LHCXXTOP); $(TAR) rf $(SHAREDTARFILE) \
	share/CHBook/$(CHBOOKVERS)
	cd $(LHCXXTOP); $(TAR) rf $(SHAREDTARFILE) \
	share/NtupleTag_HBook/$(NTUPTAGHBVERS)
	cd $(LHCXXTOP); $(TAR) rf $(SHAREDTARFILE) \
	share/NtupleTag_HepExp/$(NTUPTAGHEVERS)

The spectar target was modified as well:

	cd $(LHCXXTOP); $(TAR) rf $(SPECTARFILE) \
	specific/$(SYSN)/CHBook/$(CHBOOKVERS)
	cd $(LHCXXTOP); $(TAR) rf $(SPECTARFILE) \
	specific/$(SYSN)/NtupleTag_HBook/$(NTUPTAGHBVERS)
	cd $(LHCXXTOP); $(TAR) rf $(SPECTARFILE) \
	specific/$(SYSN)/NtupleTag_HepExp/$(NTUPTAGHEVERS)

6.2 The `allkit` and `linkkit` targets

In order to build the whole kit (including commercial components like Objectivity/DB, ObjectSpace and NAG C) use the allkit target:

gmake allkit "KITVERS=3.6.3"

(notice how the release number is passed in explicitly as the KITVERS variable).

Since the public domain components typically have a life-cycle that spans multiple Anaphe releases (e.g. NAG C cycle ~ 18 months), it would be a waste of space to duplicate the corresponding tar files everytime. In this case the tar files are prepared once and recycled in following releases using symlinks, e.g.:

> ls -l  3.6.2/  
total 64160
-rw-r--r--   1 dinofm   dl         10081 Jul 19 14:21 3.6.2.html
-rw-r--r--   1 dinofm   dl          5365 Jul 19 14:21 README.txt
-rw-r--r--   1 dinofm   dl       5123149 Jul 19 12:32 heshared.tar.gz
-rw-r--r--   1 dinofm   dl       27701058 Jul 19 14:06 hespec.tar.gz
lrwxr-xr-x   1 dinofm   dl            20 Jul 19 15:18 nagC.tar.gz -> ../3.6.1/nagC.tar.gz
lrwxr-xr-x   1 dinofm   dl            20 Jul 19 15:18 objy.tar.gz -> ../3.6.1/objy.tar.gz
lrwxr-xr-x   1 dinofm   dl            22 Jul 19 15:18 ospace.tar.gz -> ../3.6.1/ospace.tar.gz
lrwxr-xr-x   1 dinofm   dl            20 Jul 19 15:19 pdom.tar.gz -> ../3.6.1/pdom.tar.gz
-rwxr-xr-x   1 dinofm   dl           577 Jul 19 14:21 sharedinst
-rwxr-xr-x   1 dinofm   dl          2777 Jul 19 14:21 specinstall

In this case four tar files are reused via symlinks to the real files in the previous release.

For exactly this reason we have a special make target that does not pack these components:

gmake linkkit "KITVERS=3.6.3" "PREVKIT=3.6.2"

The public domain tar file is built by this target (since it's required to build the HEP componts tar file), but it is possible to replace the file with a symlink afterwards.

6.2 Customize the export kit by platform

It might be necessary to exclude packages from the export kits on some platforms (e.g. if it's not available). The spectarOS target could be used for this purpose. As an example let's see how to include the ConditionsDB package on Linux and not on Solaris:


ifeq ($(OS),Linux)
spectarOS:
	cd $(LHCXXTOP); $(TAR) rf $(SPECTARFILE) \
	specific/$(SYSN)/ConditionsDB/$(CONDB_VERS); \
	$(TAR) rf $(SPECTARFILE) specific/$(SYSN)/ConditionsDB/README-VERSIONS
else

# All other platforms...

spectarOS:

endif

7. Test the export kit

Once the kit is installed in the export area it is possible to test it (at least partially). The test script requires a scratch area large enough to contain all the release and will run the HTL tests plus an exaample from the Qplotter package that will test visualization and basic fitting.

Be aware this is not a full test, just a quick check that packages are properly installed, libraries are accessible and that utilities in the bin subdirectory can be reached.

7.1 Pre-check

Before running the test, verify that all the tar files are there:

/afs/cern.ch/sw/lhcxx/export/redhat61/3.6.3 > ls -lL
total 123959
-rw-r--r--   1 dinofm   dl           8540 Sep 21 18:50 3.6.3.html
-rw-r--r--   1 dinofm   dl           5570 Sep 21 18:50 README.txt
-rw-r--r--   1 dinofm   dl       10129863 Sep 21 17:18 heshared.tar.gz
-rw-r--r--   1 dinofm   dl       28514464 Sep 21 18:23 hespec.tar.gz
-rw-r--r--   1 dinofm   dl        3583806 Jun 13 16:41 nagC.tar.gz
-rw-r--r--   1 dinofm   dl       56884531 Jun 13 16:40 objy.tar.gz
-rw-r--r--   1 dinofm   dl        5681559 Feb  9  2001 ospace.tar.gz
-rw-r--r--   1 dinofm   dl       22117182 Sep 21 18:28 pdom.tar.gz
-rwxr-xr-x   1 dinofm   dl            577 Sep 21 18:50 sharedinst
-rwxr-xr-x   1 dinofm   dl           2790 Sep 21 18:50 specinstall

7.2 Kit test

The testKit.sh test script is located in LHCXX/admin. It accepts 4 mandatory parameters to identify respectively:

The place the release should be installed (scratch)
The platform name in the export area
The release name
The Anaphe platform (typically platform/compiler)

This is an example on how to use it:

LHCXX/admin > ./testKit.sh ~/hepexplorer/scratch redhat61 3.6.3 redhat61/gcc-2.95.2
Removing log
Reset environment
Running  /afs/cern.ch/user/d/dinofm/hepexplorer/scratch/anaphe/share/LHCXX/3.6.3/install/sharedstart.sh
Symlink to Objectivity/DB
HTL test
Test of binary directory
Qplotter & HepFitting/Gemini test
cleaning in /afs/cern.ch/project/asddat/hepexplorer/scratch/anaphe/share/Qplotter/1.5.0.3/Qplotter/examples/histoFit
Check the file  /afs/cern.ch/user/d/dinofm/hepexplorer/RH61/LHCXX/admin/histofit.ps
and the logfiles:
/afs/cern.ch/user/d/dinofm/hepexplorer/RH61/LHCXX/admin/testKitInst.log
/afs/cern.ch/user/d/dinofm/hepexplorer/RH61/LHCXX/admin/testKitRun.log
Check the file  /afs/cern.ch/user/d/dinofm/hepexplorer/RH61/LHCXX/admin/histofit.ps

Follow the script instructions to check the log files and the output Postscript file.

8. Set up the ACLs in the export tree

Once the kit is tested, it can be frozen to avoid overwriting by mistake. There's another script in LHCXX/admin to do the job and it's named setKitAcl.sh . The script contains comments explaining the policy to setup ACLs in the export kit, so have a look... In order to use it:

LHCXX/admin > ./setKitAcl.sh 3.6.3 redhat61

9. Update the Web documentation in the Anaphe site

The Releases page

Copy the release notes in HTML format to the web area:

/afs/cern.ch/sw/lhcxx/share/WWW/anapheweb/releases

Modify the page: http://anaphe.web.cern.ch/anaphe/releases.html to add the link to the latest release and shift the former production release into the Older releases section.

The Download page

Modify the page: http://anaphe.web.cern.ch/anaphe/download.html to include in the table the HEP distribution of the latest release.

The News and Home pages

Modify the pages: http://anaphe.web.cern.ch/anaphe/news.html
and
http://anaphe.web.cern.ch/anaphe/anaphe.html to annouce the release.

CERN Accelerating science