This website is no longer maintained. Its content may be obsolete. Please visit http://home.cern/ for current CERN information.

How to make a testsuite for Anaphe packages

Author: Dino Ferrero Merlino

Date: 15 Sept 2001

1. Introduction

Every Anaphe package should include a comprehensive testsuite in order to perform regression tests and memory leakage/overwriting analysis. The testsuite can consist of a single testprogram (e.g. HepUtilities) or multiple executables, each one exercising part of the package (as in the HTL case, where transient and persistent libraries are tested separately).

There are several requirements which should be met in order to have a manageable test procedure. These are the ones identified so far:

Package testing should be possible using libraries/header files both from the CVS checkout area (to validate new versions of the package) and from the release area (testing of a new Anaphe release).
The ouput of the testing should be concise (just passed/failed diagnostics) so to improve the readability of the logs. This information should be directed to the standard output stream (std::cout).
Additional information should be directed to the error stream (std::cerr) so to allow redirection "from the outside".
No user interaction is required to carry out the test.

2. Structure

To ease the automatic testing of packages, it is important that the testsuite can be located, built and executed in a homogeneous way. These are the rules of thumb to follow:

The testsuite is located in the test subdirectory of the package.
If the testsuite consists of several distinct executables, it should be structured in subdirectories under the test directory, each subdirectory containing one testprogram.
gmake is used to build and execute the test. The makefile should contain at least the test target that cleans, builds and executes the testsuite.
The default behaviour of the testsuite is to run using the package parts installed in the current Anaphe release (as defined by the environment variable LHCXX_REL_DIR).
To test the package using the CVS checkout area, the gmake variable LOCALTEST should be set, e.g.:
```
  gmake test "LOCALTEST=1"
```
The name of the testprogram is kept in the gmake variable APPL
The testprogram is built in the $OS subdirectory (i.e. Linux on RedHat).

Existing testsuites which do not fulfill these requirements should be adapted.

3. Helper makefile

The Anaphe etc directory contains a helper makefile named test.mk. This file includes goodies to help implementing "local testing" (i.e. in the CVS checkout area), such as filters to locate libraries/header files (especially useful if they're scattered around in several subdirectories), to setup automatically the LD_LIBRARY_PATH and to optionally list the shared libraries used in the run (via the ldd command). See the next

4. An example of test makefile: HepUtilities

This section describes the makefile of a simple testsuite. This should be enough for most packages. To see a more complex example, checkout the HTL package and have a look at its testsuite.

# Standard Anaphe makefile
include $(LHCXX_REL_DIR)/etc/lhcxx.mk
# Special stuff for testing
include $(LHCXX_REL_DIR)/etc/test.mk

ifdef LOCALTEST
C_INCLUDES += -I../../ 
TEST_LIBS  += -L../lib/${OS} -lHepUtilities 
LIBPATH    = `find ../lib -name *.$(SHLIB_EXT)|awk $(AWK_LIBS_PATH)`
else
C_INCLUDES  += -I${LHCXX_REL_DIR}/include 
TEST_LIBS   += -L${LHCXX_REL_DIR}/lib -lHepUtilities 
endif

# Application name, source files on objects
SRC     = testutil.cpp CutList.cpp 
APPL    = testutil
OBJS    = $(addprefix $(OS)/, $(SRC:.cpp=.$(OBJ_EXT)))

# make APPL the default target
all: $(APPL)

test: clean all run

$(APPL): GNUmakefile $(OBJS)
	$(C++) $(C_FLAGS) -o $(OS)/$(APPL) $(OBJS) $(TEST_LIBS) \
	$(DYNA_LIBS)

run: testrun

Local test setup

The LOCALTEST variable defines whether the test uses the local checkout area or the release area. In the former case the path for header files and libraries is defined locally and the LIBPATH variable holds the path to the directory containing the libraries (it will be prepended to the current LD_LIBRARY_PATH before running the test). Otherwise the release directories for header files and libraries are used (ore details on the weird `find ../lib -name... later on).

If the testsuite requires other Anaphe components, the advice is to add after the if statement something like that:

# All other includes and libraries
C_INCLUDES += -I$(LHCXX_REL_DIR)/include
TEST_LIBS  += -L${LHCXX_REL_DIR}/lib -lCLHEP

Application name, source files on objects

This section implements some of the requirements stated beforehand (name and location of the executable testprogram).

Targets

The mandatory test target implements the whole test (building and running the testsuite). The run target relies on a predifined testrun implemented in the helper makefile:

ifdef VERBOSE
showlibs:
	@(LD_LIBRARY_PATH=$(LIBPATH)$(LD_LIBRARY_PATH);export LD_LIBRARY_PATH;\
	$(LDD) $(OS)/$(APPL))

else
showlibs:
endif

testrun: showlibs
	@(LD_LIBRARY_PATH=$(LIBPATH)$(LD_LIBRARY_PATH);export LD_LIBRARY_PATH;\
	$(OS)/$(APPL))

This target prepends the path to local libraries to LD_LIBRARY_PATH (if necessary) and shows the loaded shared libraries if the VERBOSE variable is set.

Output

Using the test target to verify the package as is in the release:

gmake test
cleaning in /afs/cern.ch/user/d/dinofm/hepexplorer/RH61/HepUtilities/test
creating platform directory...
compiling testutil
compiling CutList
g++  -I/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/3.6.3-pre1/include   -ansi -pedantic -Wall -Wno-long-long 
-Wpointer-arith -Wnested-externs -Woverloaded-virtual -Wbad-function-cast -Wconversion -fnonnull-objects  -pthread 
-o Linux/testutil Linux/testutil.o Linux/CutList.o 
-L/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/3.6.3-pre1/lib -lHepUtilities -ldl

gmake[1]: Entering directory `/afs/cern.ch/project/asddat/hepexplorer/RH61/HepUtilities/test'
gmake[1]: Leaving directory `/afs/cern.ch/project/asddat/hepexplorer/RH61/HepUtilities/test'
Test 1 : passed
Test 2 : passed
Test 3 : passed
Test 4 : passed
Test 5 : passed
2000-Jan-1 0:0:0.0 (GMT) 
Test 6 : passed

Verbose output and local test

By switching on the VERBOSE option we can get additional information on how the testsuite was compiled and what set of libraries was used to run the test. In this example the test runs with the local set of libraries and the VERBOSE option shows the list of shared libraries confirming we're really testing the package using the shared objects in the CVS checkout area.

gmake test "LOCALTEST=1" "VERBOSE=1"

including /afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/3.6.3-pre1/etc/unix_defaults.mk
including /afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/3.6.3-pre1/etc/Linux.mk
including /afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/3.6.3-pre1/etc/deps.mk
cleaning in /afs/cern.ch/user/d/dinofm/hepexplorer/RH61/HepUtilities/test
creating platform directory...
compiling testutil
g++  -c testutil.cpp   -I../../   -ansi -pedantic -Wall -Wno-long-long -Wpointer-arith -Wnested-externs 
-Woverloaded-virtual -Wbad-function-cast -Wconversion -fnonnull-objects  -pthread -o "Linux/testutil.o"
compiling CutList
g++  -c CutList.cpp   -I../../   -ansi -pedantic -Wall -Wno-long-long -Wpointer-arith -Wnested-externs 
-Woverloaded-virtual -Wbad-function-cast -Wconversion -fnonnull-objects  -pthread -o "Linux/CutList.o"
g++  -I../../   -ansi -pedantic -Wall -Wno-long-long -Wpointer-arith -Wnested-externs -Woverloaded-virtual 
-Wbad-function-cast -Wconversion -fnonnull-objects  -pthread -o Linux/testutil Linux/testutil.o Linux/CutList.o 
-L../lib/Linux -lHepUtilities -ldl 
        libHepUtilities.so => ../lib/Linux/libHepUtilities.so (0x2aac0000)
        libdl.so.2 => /lib/libdl.so.2 (0x2aae5000)
        libstdc++-libc6.1-2.so.3 => /usr/local/gcc-alt-2.95.2/lib/libstdc++-libc6.1-2.so.3 (0x2aae9000)
        libm.so.6 => /lib/libm.so.6 (0x2ab31000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x2ab4e000)
        libc.so.6 => /lib/libc.so.6 (0x2ab61000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x2aaab000)

gmake[1]: Entering directory `/afs/cern.ch/project/asddat/hepexplorer/RH61/HepUtilities/test'
g++  shTest.cpp -c -I../../.. -I. -fPIC -I/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/HepODBMS/0.3.2.11/include 
-I/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc-2.95.2/Objectivity/6.1/include -o shTest.o
g++  -shared -o  shTest.so -ldl    \
  shTest.o 
gmake[1]: Leaving directory `/afs/cern.ch/project/asddat/hepexplorer/RH61/HepUtilities/test'
Test 1 : passed
Test 2 : passed
Test 3 : passed
Test 4 : passed
Test 5 : passed
2000-Jan-1 0:0:0.0 (GMT) 
Test 6 : passed

5. More complex use of helper makefile features: HTL

The HTL testsuite is the most complex so far. It consists of three separate testprograms and local testing is more complex due to the location of header files and libraries in different directories. In order to facilitate the search for local libraries/header files, the helper makefile provides awk filters to transform the result of a UNIX find search in a format suitable for a makefile. This is an excerpt of the HTL test makefiles:

ifdef LOCALTEST 
# prepend local headers (include files from code and utility dirs)
C_INCLUDES += `find ../../[cu]* -name *.h|awk $(AWK_INCS_FILTER)`
# figure out local libraries
TEST_LIBS  = `find ../../code/T_ -name *.$(SHLIB_EXT)|awk $(AWK_LIBS_FILTER)`
LIBPATH    = `find ../../code/T_ -name *.$(SHLIB_EXT)|awk $(AWK_LIBS_PATH)`
else
# use release libraries
TEST_LIBS   = -L$(LHCXX_REL_DIR)/lib -lT_Histoxx
endif

The first find looks for header files in HTL subdirectories starting with c (code) or u (utility). The output of the filter would be someting like that:

-I../../utility -I../../code/Common/ -I../../code/T_/

The second find looks for shared libraries in HTL subdirectory ../../code/T_. The output of the filter would be someting like that:

-L../../code/T_/HTL/Linux/ -lT_Histoxx

Finally the third filter would produce a library path that can be prepended to the environment variable LD_LIBRARY_PATH:

../../code/T_/HTL/Linux/:

CERN Accelerating science