VOMS core troubleshooting

Wrong host certificate subject in the vomses file

It is possible that after renewing a host certificate, the host certificate subject changes and the vomses file containing the VOMS server information is not updated accordingly.

The client side message is like in the following example:

                  bash-2.05b$ voms-proxy-init -voms mysql_vo1 -userconf ~/vomses 
                  Your identity: /C=CH/O=CERN/OU=GRID/CN=Maria Alandes Pradillo 5561 Enter GRID pass phrase:
                  Creating temporary proxy ....................................... Done
                  Contacting  lxb0769.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=lxb0769.cern.ch] "mysql_vo1" Failed

                  Error: Could not establish authenticated connection with the server.
                  GSS Major Status: Unexpected Gatekeeper or Service Name GSS Minor Status Error Chain:

                  an unknown error occurred

                 Failed to contact servers for mysql_vo1.

The server log file contains the following lines:

                 Wed Aug 16 11:04:48 2006:lxb0769.cern.ch:vomsd(4341):ERROR:REQUEST:AcceptGSIAuthentication
                 home/glbuild/GLITE_3_0_0_final/org.glite.security.voms/src/socklib/Server.cpp:259):Failed to establish 
                 security context (accept):.GSS Major Status: General failure.GSS Minor Status Error 
                 Chain:..accept_sec_context.c:305:gss_accept_sec_context: Error during delegation: Delegation protocol 
                 violation

In this case it's good that you check whether the vomses file contains the correct host certificate subject. To check what's your VOMS host certificate subject, run the following command:

                 [root@lxb0769 root]# openssl x509 -in /etc/grid-security/hostcert.pem -noout -subject
                 subject= /C=CH/O=CERN/OU=GRID/CN=host/lxb0769.cern.ch

And check in the vomses file that the certificate subject is correct:

                 bash-2.05b$ more vomses
                 ...
                 "mysql_vo1" "lxb0769.cern.ch" "15001" "/C=CH/O=CERN/OU=GRID/CN=host/lxb0769.cern.ch" "mysql_vo1"
                 ...

Wrong port number in the vomses file

It is possible that after a VOMS server reconfiguration, the post number changes and the vomses file containing the VOMS server information is not updated accordingly.

The client side message is like in the following example:

                [lxb1420] /afs/cern.ch/user/m/malandes > voms-proxy-init -voms dteam
                Enter GRID pass phrase:
                Your identity: /C=CH/O=CERN/OU=GRID/CN=Maria Alandes Pradillo 5561
                Cannot find file or dir: /afs/cern.ch/user/m/malandes/.glite/vomses
                Creating temporary proxy .................................... Done
                Contacting  lxb2176.cern.ch:15012
                [/C=CH/O=CERN/OU=GRID/CN=host/lxb2176.cern.ch] "dteam" Failed
 
                Error: Could not connect to socket. Connection refused
 
                None of the contacted servers for dteam were capable
                of returning a valid AC for the user.

The server log file doesn't contain any message since it couldn't be reached.

In this case it's good that you check whether the vomses file contains the correct port. You can confirm this infomation by checking the configuration information of the VOMS server in: https://:8443/voms//webui/config

VOMS server is down: edg-voms process

When the VOMS server is down, more specifically, the edg-voms process has died in the VOMS server, when trying to get a proxy you will get the same error as in the previous scenario:

                Error: Could not connect to socket. Connection refused

If you check that your vomses files contains the right port, then the error is very likely to be because the VOMS server is down.

VOMS server is down: middleman process

When the VOMS server is down, more specifically, the middleman process has died in the VOMS server, when trying to get a proxy you will get the following error:

                 [lxb1420] /afs/cern.ch/user/m/malandes > voms-proxy-init -voms dteam -userconf ~/vomses-lxb2176
                 Cannot find file or dir: /opt/glite/etc/vomses
                 Enter GRID pass phrase:
                 Your identity: /C=CH/O=CERN/OU=GRID/CN=Maria Alandes Pradillo 5561
                 Cannot find file or dir: /opt/glite/etc/vomses
                 Creating temporary proxy ................................. Done
                 Contacting  lxb2176.cern.ch:15002 [/C=CH/O=CERN/OU=GRID/CN=host/lxb2176.cern.ch] "dteam" Failed

                 Error: dteam: User unknown to this VO.

                 None of the contacted servers for dteam were capable
                 of returning a valid AC for the user.

If you are pretty sure you are a member of the VO, then it is very likely that the VOMS server is down.

You can check you are actually a member of the VO with the following voms-admin command:

bash-2.05b$ voms-admin -host -vo= list-users

or in the following URL: https://:8443/voms//webui/admin/users/list

There is a bug for this error message which is misleading: https://savannah.cern.ch/bugs/?func=detailitem&item_id=18367

Outdated or missing VOMS CA in the /etc/grid-security/certificates directory of the UI

It's possible that the CA that signs the host certificate of the VOMS server you want to contact is not present in /etc/grid-security/certificates or that you forgot to update its files.

The client side message is like in the following example:

                  bash-2.05b$ voms-proxy-init -voms mysql_vo1 -userconf ~/vomses 
                  Your identity: /C=CH/O=CERN/OU=GRID/CN=Maria Alandes Pradillo 5561 Enter GRID pass phrase:
                  Creating temporary proxy ....................................... Done
                  Contacting  lxb0769.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=lxb0769.cern.ch] "mysql_vo1" Failed

                  Error: Could not establish authenticated connection with the server.
                  GSS Major Status: Unexpected Gatekeeper or Service Name GSS Minor Status Error Chain:

                  an unknown error occurred

                 Failed to contact servers for mysql_vo1.

The server log file contains the following lines:

                 Wed Aug 16 11:04:48 2006:lxb0769.cern.ch:vomsd(4341):ERROR:REQUEST:AcceptGSIAuthentication
                 home/glbuild/GLITE_3_0_0_final/org.glite.security.voms/src/socklib/Server.cpp:259):Failed to establish 
                 security context (accept):.GSS Major Status: General failure.GSS Minor Status Error 
                 Chain:..accept_sec_context.c:305:gss_accept_sec_context: Error during delegation: Delegation protocol 
                 violation

The solution is to update the CA files, check that the APT string of the CAs repository is correct and apply the necessary updates.

Missing user CA in the /etc/grid-security/certificates directory of the VOMS server

If for some reason, the user CA is not present in the VOMS server /etc/grid-security/certificates directory, the client side message is like in the following example:

                 [lxb1420] /afs/cern.ch/user/m/malandes > voms-proxy-init -voms dteam -userconf ~/vomses-lxb2176 
                 -cert ~malandes/test-certs/test_user_1_cert.pem -key ~malandes/test-certs/test_user_1_key.pem
                 Cannot find file or dir: /opt/glite/etc/vomses
                 Enter GRID pass phrase:
                 Your identity: /C=CH/O=CERN/OU=GD/CN=Test user 1
                 Cannot find file or dir: /opt/glite/etc/vomses
                 Creating temporary proxy .............................. Done
                 Contacting  lxb2176.cern.ch:15002 [/C=CH/O=CERN/OU=GRID/CN=host/lxb2176.cern.ch] "dteam" Failed

                  Error: Could not establish authenticated connection with the server.
                     globus_gss_assist token :-1: read failure: unknown


                 None of the contacted servers for dteam were capable
                 of returning a valid AC for the user.

The server log file contains the following lines:

                Wed Jan 17 12:21:44 2007:lxb2176.cern.ch:vomsd(9318):ERROR:REQUEST:AcceptGSIAuthentication 
                (/home/glbuild/GLITE_3_0_3_RC1/org.glite.security.voms/src/socklib/Server.cpp:262):Failed to establish 
                security context (accept):.GSS Major Status: Authentication Failed.GSS Minor Status Error Chain:
                ..accept_sec_context.c:170: gss_accept_sec_context: SSLv3 handshake problems.globus_i_gsi_gss_utils.c:881: 
                globus_i_gsi_gss_handshake: Unable to verify remote side's credentials.globus_i_gsi_gss_utils.c:854: 
                globus_i_gsi_gss_handshake: SSLv3 handshake problems: Couldn't do ssl handshake.OpenSSL Error: 
                s3_srvr.c:1816: in library: SSL routines, function SSL3_GET_CLIENT_CERTIFICATE: no certificate 
                returned.globus_gsi_callback.c:351: globus_i_gsi_callback_handshake_callback: Could not verify 
                credential.globus_gsi_callback.c:429: globus_i_gsi_callback_cred_verify: Can't get the local trusted CA 
                certificate: Cannot find issuer certificate for local credential with subject: /C=CH/O=CERN/OU=GD/CN=Test user 1

The solution is to update the CAs in /etc/grid-security/certificates and install the one that it's missing.

User interface machine that had a terrible drift of more than 500000 sec

Should be resynchronize the time by /usr/sbin/ntpdate -b pool.ntp.org otherwise you will see the same message as

                  Error: Could not establish authenticated connection with the server.
                     globus_gss_assist token :-1: read failure: unknown

Expired user CA CRL in the /etc/grid-security/certificates directory of the VOMS server

It's possible that in the VOMS server, the fetch-crl cron job has failed to update your user CA CRL in /etc/grid-security/certificates.

The client side message is like in the following example:

                 [lxb1420] /afs/cern.ch/user/m/malandes > voms-proxy-init -voms dteam -userconf ~/vomses-lxb2176 
                 -cert ~malandes/test-certs/test_user_1_cert.pem -key ~malandes/test-certs/test_user_1_key.pem
                 Cannot find file or dir: /opt/glite/etc/vomses
                 Enter GRID pass phrase:
                 Your identity: /C=CH/O=CERN/OU=GD/CN=Test user 1
                 Cannot find file or dir: /opt/glite/etc/vomses
                 Creating temporary proxy .............................. Done
                 Contacting  lxb2176.cern.ch:15002 [/C=CH/O=CERN/OU=GRID/CN=host/lxb2176.cern.ch] "dteam" Failed

                  Error: Could not establish authenticated connection with the server.
                     globus_gss_assist token :-1: read failure: unknown


                 None of the contacted servers for dteam were capable
                 of returning a valid AC for the user.

The server log file contains the following lines:

                 Wed Jan 17 11:48:20 2007:lxb2176.cern.ch:vomsd (5943):ERROR:REQUEST:AcceptGSIAuthentication 
                 (/home/glbuild/GLITE_3_0_3_RC1/org.glite.security.voms/src/socklib/Server.cpp:262):Failed to establish security 
                 context (accept):.GSS Major Status: Authentication Failed.GSS  Minor Status Error Chain:..accept_sec_context.c:170: 
                 gss_accept_sec_context: SSLv3 handshake problems.globus_i_gsi_gss_utils.c:881: globus_i_gsi_gss_handshake: 
                 Unable to verify remote side's credentials.globus_i_gsi_gss_utils.c:854: globus_i_gsi_gss_handshake: SSLv3 handshake 
                 problems: Couldn't do ssl handshake.OpenSSL Error: s3_srvr.c:1816: in library: SSL routines, function 
                 SSL3_GET_CLIENT_CERTIFICATE: no certificate returned.globus_gsi_callback.c:351: 
                 globus_i_gsi_callback_handshake_callback: Could not verify credential.globus_gsi_callback.c:477: 
                 globus_i_gsi_callback_cred_verify: Could not verify credential.globus_gsi_callback.c:
                 769: globus_i_gsi_callback_check_revoked: Invalid CRL: The available CRL has expired

In that case you have to update your CRL.