Standard Compliance Testing

This page describes the available and planned test suites for standard compliance testing in EMI.

Available test suites

Testsuite	Standard tested	Short description
litmus	HTTP and WebDAV	litmus is a WebDAV server test suite, which aims to test whether a server is compliant with the WebDAV protocol as specified in RFC2518
nfs connectathon	NFS4.1/pNFS	Programs that can be used to test an implementation of the NFS Protocol. The tests run on a UNIX client and test server and client functions
pynfs	NFS4.1/pNFS	CITI maintains pynfs, a suite of several Python tools for NFS4, useful for testing purposes. Its primary components are a server tester, useful for testing server response to unusual error conditions and/or a badly behaving client
GlueValidator	GLUE 2.0	The GLUE Validator is a simple python-based unit test suite that validates information against test cases. It works against both LDIF files and LDAP servers. In addition, a GLUE 2.0 LDAP schema validator is available
EMI CAR validator	EMI CAR	Test suite for validating EMI Compute Accounting Records
!StARval	EMI StAR	Test suite for validating EMI Storage Accounting Records
EMI ES Creation and Management Ports Testsuite	EMI ES Management and Creation port types	EMI ES Management and Creation testsuite based on the UNICORE Commandline Client (UCC)
EMI ES Delegation Port Testsuite	EMI ES Delegation port type	EMI ES Delegation testsuite
arcemiestest	EMI ES Information port type	EMI ES Information testsuite based on ARC
CANL	OCSP, SHA2, others	Auth validators for individual CANL packages

EMI 3 standard compliance testing

The table below list the MANDATORY standard compliance tests to be preformed by the product teams.

Description

• Test id: ID
• Product: The product to be tested.
• Test suit: The test utility to be used.
• Fields to be filled by PTs:
  • Test page: Link to the detailed test plan or a dedicated test area, if available. If special configurations of the testbed are needed, these should be stated here.
  • Comment: Any relevant information to share. E.g. note here if the test is run automatically e.g. via a unit testing or similar framework.
  • Test outcome: For executed tests this field should be either FAILED or SUCCESS. If available, please provide URLs of problem tickets (FAILED) or URL to the test report (SUCCESS).

Test id	Product	Test suite	Test page	Comment	Test outcome
S1	UNICORE CE	EMI-ES delegation		Test written by lisa.zangrando@pdNOSPAMPLEASE.infn.it There were 4 failures because UNICORE does NOT implement the following operations: getInterfaceVersion(), getTerminationTime(), getVersion(), renewProxyReq()	FAILED Test report: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/standards_test_emi_delegation_UNICORE_report.txt
S2	CREAM	EMI-ES delegation		Test written by lisa.zangrando@pdNOSPAMPLEASE.infn.it	SUCCESS Test report: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/DelegationService_report_CREAM.txt
S3	ARC CE	EMI-ES delegation		Test written by lisa.zangrando@pdNOSPAMPLEASE.infn.it	SUCCESS Test report:https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/standards_test_emi_delegation_ARC_report.txt
S4	UNICORE CE	EMI-ES Info	http://svn.nordugrid.org/trac/nordugrid/browser/arc1/trunk/src/hed/acc/EMIES	Test written by aleksandr.konstantinov@fysNOSPAMPLEASE.uio.no	SUCCESS Test report: https://savannah.cern.ch/file/EMI-ES%20ResourceInfo%20validation.txt?file_id=31610
S5	CREAM	EMI-ES Info	http://svn.nordugrid.org/trac/nordugrid/browser/arc1/trunk/src/hed/acc/EMIES	Test written by aleksandr.konstantinov@fysNOSPAMPLEASE.uio.no	SUCCESS Test report: https://savannah.cern.ch/file/EMI-ES%20ResourceInfo%20validation.txt?file_id=31610
S6	ARC CE	EMI-ES Info	http://svn.nordugrid.org/trac/nordugrid/browser/arc1/trunk/src/hed/acc/EMIES	Test written by aleksandr.konstantinov@fysNOSPAMPLEASE.uio.no	SUCCESS Test report: https://savannah.cern.ch/file/EMI-ES%20ResourceInfo%20validation.txt?file_id=31610
S7	UNICORE CE	EMI-ES Crt&Mgmt		Test written by b.schuller@fz-juelichNOSPAMPLEASE.de	SUCCESS https://savannah.cern.ch/task/download.php?file_id=31620 https://savannah.cern.ch/task/download.php?file_id=32018
S8	CREAM	EMI-ES Crt&Mgmt		Test written by b.schuller@fz-juelichNOSPAMPLEASE.de	SUCCESS https://savannah.cern.ch/task/download.php?file_id=32015 https://savannah.cern.ch/task/download.php?file_id=32017
S9	ARC CE	EMI-ES Crt&Mgmt		Test written by b.schuller@fz-juelichNOSPAMPLEASE.de	SUCCESS https://savannah.cern.ch/task/download.php?file_id=31621 https://savannah.cern.ch/task/download.php?file_id=32016
S10	dCache	litmus-webdav			RUN - basic tests pass, some advanced tests fail. See report for details: http://svn.dcache.org/build/view/emi_certification/job/litmus%20-%20WebDAV%20tests/11/console
S11	DPM	litmus-webdav			SUCCESS: https://savannah.cern.ch/task/index.php?34046
S12	StoRM	litmus-webdav		The test is considered successfull since PROPPATCH and LOCK operations have not been implemented because not of interest to StoRM users	SUCCESS: https://savannah.cern.ch/task/index.php?34065
S13	dCache	NFS4.1			SUCCESS http://svn.dcache.org/build/view/emi_certification/job/nfs4.1_Tests/14/
S14	DPM	NFS4.1			SUCCESS: https://savannah.cern.ch/task/index.php?34046
S15	StoRM	NFS4.1		Test suite not suitable, StoRM uses GPFS	NOT RUN
S16	Dcache	STARval			SUCCESS
S17	DPM	STARval			SUCCESS: https://savannah.cern.ch/task/index.php?34046
S18	StoRM	STARval		Feature is not implemented yet	NOT RUN
S19	UNICORE CE	CAR validator	https://github.com/guarise/emicar	Test run manually. CAR records are not stored on disk as text files normally, so the input was taken from the log dumps which can be turned on on the rus-service.	SUCCESS Test report: https://savannah.cern.ch/task/download.php?file_id=31548
S20	CREAM	CAR validator	https://github.com/guarise/emicar	A 'minor' fail. If Processors or NodeCount information is not present, APEL publishes '0' instead of omitting the tag.	FAIL
S21	ARC CE	CAR validator	https://github.com/guarise/emicar	Generate CAR records with the JURA from the ARC CE reports and run the emicar validator.	SUCCESS Test report: ARC_CE_CAR_validation
S22	CANL JAVA	auth validator	http://csrc.nist.gov/groups/ST/crypto_apps_infra/documents/PKITS.pdf	All tests are run by JUnit, so are guaranteed to be passed. There is one real std. testsuite implemented: NIST http://csrc.nist.gov/groups/ST/crypto_apps_infra/pki/pkitesting.html (Path validation). Some optional parts which are unsupported by canl are not implemented. There are some functional tests for OCSP and SHA2 - linked too, though are not standards compliance tests.	SUCCESS NIST report: https://savannah.cern.ch/task/download.php?file_id=31557 OCSP: https://savannah.cern.ch/task/download.php?file_id=31559 SHA2: https://savannah.cern.ch/task/download.php?file_id=31558
S23	CANL C	auth validator	https://twiki.cern.ch/twiki/bin/view/EMI/CaNlC-partEMI3FinalRC#C_Part	OCSP and SHA2 use case tests run as a part of the normal caNl test Suite. The do not constitute a full standards compliance test, though.	DONE
S24	CANL C++	auth validator	http://svn.nordugrid.org/trac/workarea/browser/caNl%2B%2B/trunk/src/canl-c%2B%2B/test/	Test written by weizhong.qiang@fysNOSPAMPLEASE.uio.no. SHA2 test is included in test suite CredentialsTest.cpp; OCSP and OCSP stapling are included in test suite CredentialsTest.cpp and OCSPStaplingTest.cpp. NIST certificates suite has not been completely supported by canl c++, so the test suite ( NIST_PKITSTest.cpp) is not supposed to succeed for all of the NIST certificates.	SUCCESS (for SHA2 and OCSP/OCSP stapling), test report: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/sha2_and_ocsp_canlxx https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/ocsp_stapling_canlxx
S25	UNICORE	glue2 validator		4 failing tests: three because of 'open enumeration' values that are wrongly checked by the validator	FAIL, report https://savannah.cern.ch/task/download.php?file_id=32014
S26	DPM	glue2 validator			FAIL, report and ticket at https://its.cern.ch/jira/browse/LCGDM-934
S27	dCache	glue2 validator			RUN for glue1 and glue2 see reports: http://svn.dcache.org/build/view/emi_certification/job/glue-validator%20tests/7/console
S28	StoRM	glue2 validator			RUN, report: glue_validator_storm_new.txt
S29	FTS3	glue2 validator		FTS3 is still prototype	NOT RUN
S30	CREAM	glue2 validator		Modified the types.py for extending the default values of Capability_t, InterfaceName _t and Platform_t GLUE2 types. This is needed for supporting the GLUE2 extensions introduced by the EMI-ES specification (e.g.'org.ogf.glue.emies.activityinfo', 'org.ogf.glue.emies.activitycreation' ... interface names, etc).	SUCCESS, report: glue2_validator_report_CREAM.txt
S31	WMS	glue2 validator			SUCCESS, report: glue_validator_wms.txt
S32	ARC CE	glue2 validator	https://wiki.nordugrid.org/index.php/Testing/ARC-CE#CE-F15:_infopublishing:_glue2_LDAP_schema	Test fails because of missing hardcoded open enumerations; I think this is wrong, it should just give a warning. Some open enumerations are new and are not part of the validator code	FAILED but false positives - see log: https://twiki.cern.ch/twiki/pub/EMI/StandardComplianceTestSuites/S32_glue_validator_ARC_CE.log
S33	VOMS	glue2 validator			SUCCESS,on SL6 report: glue_validator_voms.txt
S34	ARGUS	glue2 validator		The glue-validator tests failed, but the errors are in the test themself, not in Argus. Reports: argus-EMI3_RC-glue-validator-SL6.txt argus-EMI3_RC-glue-validator-SL5.txt	RUN
S35	L&B	glue2 validator	https://twiki.cern.ch/twiki/bin/view/EGEE/TestReportForLBEMI3FinalRC#Glue_Validator	Fixes were required in L&B (fixed immediately) and Info Providers (https://savannah.cern.ch/bugs/?100126). All OK now.	SUCCESS

-- JonKerrNilsen - 11-Nov-2011