Appendix: EMI Test Report Template

1. Introduction

This twiki page presents a template for the Test Report produced after testing new releases of EMI software products.

2. Test Report Template

The following template must be used by PTs to report about the result of their testing. The use of a common template will simplify the work of the QC activity.

*********************************
EMI Test Report Template 
*********************************

- Product: 

- Release Task:

- ETICS Subsystem Configuration Name:

- VCS Tag:

- EMI Major Release:

- Platforms:

- Author: 

- Date:

- Test Report Template : v. 3.2

*************
Summary 
*************

1. Deployment tests: 
   1.1. Clean Installation - PASS/FAIL
   1.2. Upgrade Installation - PASS/FAIL/NA 
2. Static Code Analysis - PASS/FAIL/NA
3. Unit Tests Execution - YES/NO
4. System tests:
  4.1. Functionality tests - PASS/FAIL
  4.2. Regression tests - PASS/FAIL/NA
  4.3. Standard Conformance tests - PASS/FAIL/NA
  4.4. Performance tests - PASS/FAIL/NA
  4.5. Scalability tests - PASS/FAIL/NA
  4.6. Integration tests - PASS/FAIL/NA

REMARKS:

*************************** Detailed Testing Report ***************************************

1. Deployment log 
************************

1.1. Clean Installation
-----------------------------
- YUM/APT Testing Repo file contents:
- YUM/APT Install command:
- YUM/APT log:
- Configuration log:

1.2. Upgrade Installation
--------------------------------
- YUM/APT Production Repo file contents:
- YUM/APT Install command:
- YUM/APT Testing Repo file contents:
- YUM/APT Upgrade command:
- YUM/APT log:
- Configuration log:

2. Static Code Analysis
******************************
- URL where static code analysis results can be accessed

3. Unit Tests
*****************
- URL pointing to the results of the Unit Tests.
- Code Coverage %, if available.

4. System tests 
*********************
- URL where the tests/testsuite can be accessed:
- URL where the test results can be accessed:

OR 

please, use the template below to include the test results in this document:

--------------- System Test Summary ------------------
For each Basic Functionality test, please include the following summary:
* Description of the test:             
* Result: PASSED/FAILED:               

For each Regression test, please include the following summary:
* Description of the test:             
* Test Unique ID/RfC unique ID:                        
* Result: PASSED/FAILED:               

For each of Performance and Scalability tests, please include the following summary:
* Description of the test:             
* Description of the specific context: 
* Result: PASSED/FAILED:               

For each Standards Compliance/Conformance test, please include the following summary:
* Description of the test:             
* Adopted Standard:                     
* Result: PASSED/FAILED:               

For each Integration test, please include the following summary:
* Description of involved components and testbed setup:
* Description of the test:
* Description of the specific context:
* Result: PASSED/FAILED:

4.1. Basic Functionality tests
*************************************
For each test:
COPY & PASTE TEST OUTPUT.

4.2. Regression tests
*****************************
For each test:
COPY & PASTE TEST OUTPUT.

4.3. Standard Conformance tests
******************************************
For each test:
COPY & PASTE TEST OUTPUT.

4.4. Performance tests 
*******************************
For each test:
COPY & PASTE TEST OUTPUT.

4.5. Scalability tests 
**************************
For each test:
COPY & PASTE TEST OUTPUT.

4.6. Integration tests 
**************************
For each test:
COPY & PASTE TEST OUTPUT.

---------------End of System Test Summary ------------------

3. Test Report in detail

3.1. Product

The Test Report must specify the name of the tested product as specified in the DNA1.3.2 - Technical Development Plan.

3.2. Release Task

The Test Report must specify the unique id for the Release Task in the EMI release tracker where the new product version is being tracked.

3.3. EMI major release and Platform

The Test Report must specify the EMI major release and the supported platforms of the new product version.

3.4. Author

The Test Report must contain the name of the PT members who have performed the testing of the new product version.

3.5. Summary

In order to have a quick summary of the results of the tests, PTs must fill in the summary section of the template, where PASS means all the tests have passed; FAIL means that at least one test has failed; NA means the test is not applicable for the product.

It's important to note, that in the summary of the report, questions applicable to more than one platform, have to be answered taking into account all the applicable platforms.

Remarks about the result of the tests can be added in the summary.

3.6. Deployment log

3.6.1. Clean Installation

The Test Report must contain a test for a clean installation, that is an installation of the new product version including the change on a clean machine.

Don't forget to include:

• the testing yum/apt repo file with the details of the yum/apt repository that is used to do the installation.
• the yum/apt command used to install the product.
• the yum/apt output.
• configuration output (YAIM log or any other tool or set of commands run to configure the product).

Verify that the packages included in the change have been correctly installed.

3.6.2. Upgrade installation

The Test Report must contain a test for an upgrade from production. That is an installation of the product from the production repository on a clean machine, and then an upgrade to the new product version including the change. Upgrade tests must be provided for minor releases and across major releases.

Don't forget to include:

• the production yum/apt repo file with the details of the yum/apt repository that is used to do the installation.
• the yum/apt command used to install the production version of the product.
• the testing yum/apt repo file with the details of the yum/apt repository that is used to do the upgrade.
• the yum/apt command used to upgrade to the new product version.
• the yum/apt output.
• configuration output (YAIM log or any other tool or set of commands run to configure the product)

Verify that the packages included in the change have been correctly updated.

3.7. Static Code Analysis

If the product has executed Static Code Analysis plugins in ETICS, include a link to the plugin results. If you have used any other tool, you can optionally include a URL with the results as well.

3.8. Unit Tests

The Test Report must report on the results of Unit Tests. Include a URL to the results of the Unit Tests. You can optionally report about the unit test coverage in % for the product.

3.9. System tests

The Test Report must report on the results of the system tests defined in the Test Plan of the product.

Don't forget to include:

• URL of the tests/testsuites (with the revision number in case of tests maintained on a VCS)
• When tests are automated, a link to the tool-specific report, otherwise, please specify:
  • a short description of the tests run (1-2 lines should be enough)
  • the outcome of the tests, that is either PASSED or FAILED.
  • specific command to run the test and output of the test.

3.9.1. Basic functionality tests

The Test Report must contain the result of running the Basic Functionality tests described in the test plan of the product.

3.9.2. Regression tests

The Test Report must contain the result of running the existing tests associated to software defects (bugs) tracked in specific RfCs, as it should be specified in the test plan of the product.

Don't forget to include:

• The RfC unique ID for which the regression test is written.

3.9.3. Performance and scalability tests

The Test Report may contain the result of running performance and scalability tests, as defined in the test plan of the product.

Don't forget to include:

• A description of the context, user case or scenario for the tests. This should include the amount of time used to run the test, stress conditions, interactions with other services, etc.
• any relevant log, CPU usage statistics, etc that proves the good performance of the service.

3.9.4. Standard conformance tests

The Test Report may contain the result of running standard compliance/conformance tests, as defined in the test plan of the product.

Don't forget to include:

• a reference to the standard adopted.

3.9.5. Integration tests

The Test Report may contain the result of running integration tests, as defined in the test plan of the product.

4. Test Report Name

In order to automate the QC task, the name of the Test Report attached to the release task must contain the words test and report. The case is not sensitive, so both upper and lower case is allowed. Some valid examples are: VOMS_test_report_1.1.txt or ReportTest_CREAM1.2.txt.

5. Test Report Format

In order to automate the QC task, at least the summary section of the Test Report must be provided in txt format and must be attached to the release task with the name specified as described in the previous section 4. Test Report Name, and it has to be a single file taking into account all the platforms, as described in the section 3.5 Summary.

The remaining part of the test report can be provided in any other format as an extra attachment if this is more convenient for the PT, but in that case, the name should not contain the words test and report. It can be provided also in independent files for each platform.