Accessing CERN Linux machines via Kerberos
Kerberos is shared-secret networked authentication system. Its use at CERN serves a dual purpose:
- user convenience: once signed in, authentication is handled "automagically" on the users' behalf.
- security: by reducing the need to enter passwords into potentially compromised machines, the impact of such a compromise can be greatly reduced.
In the past both Kerberos version IV and 5 were supported: Kerberos IV support has been stopped in 2010.
: This document does not aim to explain Kerberos. For an overview and introduction into the concepts, please see man kerberos, MIT's site or the Kerberos FAQ. Several books exist on the subject as well.
Client-side configuration
In order for Kerberos to work, the user needs to be registered in a central database (KDC). In CERN parlance, this means a CERN account is required for the user.
Kerberos client binaries are part of the default install of many operating systems (such as Mac OS-X, BSD, Linux, Solaris, ..), or are at least available as add-ons. This document does not cover how to install these binaries on a particular operating system.
Linux clients must be registered with "LINUX" as the "Operating System" in the network database (go to https://network.cern.ch to check the registration). The precise "Version" of the "Operating System" does not matter for this purpose.
Trying it out
To test whether a client is configured correctly, please use kinit and klist. These commands should live in your default PATH, otherwise look into /usr/kerberos/bin or /usr/heimdal/bin (please avoid using kinit from a Java JRE, these often have problems such as echoing the password).
$ kinit afsusername@CERN.CH afsusername@CERN.CH's Password: <enter your password> $
- no message means no error
- your password should not get displayed on the terminal
- On a correctly configured machine inside the CERN domain, kinit afsusername would be sufficient.
$ klist Credentials cache: FILE:/tmp/krb5cc_USERID_somenumber Principal: afsusername@CERN.CH Issued Expires Principal Sep 28 15:20:24 Sep 29 16:20:23 krbtgt/CERN.CH@CERN.CH Sep 28 15:20:24 Sep 29 16:20:23 afs@CERN.CH Kerberos 4 ticket cache: /tmp/tkt0 klist: You have no tickets cachedIn this example, only Kerberos 5 credentials for the CERN realm were obtained as Kerberos 4 is no longer supported, and the expiry time is in the future (at the time of this writing). Looks good.
In order to get access to AFS after a successful authentication, you will need to get a "forwardable" TGT, and instruct your ssh client to actually transfer it over to the server (who will then use it to obtain an AFS token for your session). Use klist -f to check whether your TGT is "forwardable" - usually it then has a flag "F":
Issued Expires Flags Principal Sep 29 13:24:41 Sep 30 13:24:03 FPR krbtgt/CERN.CH@CERN.CH
To continue testing, use ssh -v afsusername@hostname.cern.ch klist -f\; tokens\; touch .sshtest (this should log you in, show your credentials - both Kerberos and AFS, and try to actually write something into your AFS home directory) - if all succeeds, you are set to use passwordless Kerberos. If you get error messages from these commands or are being asked for your passwords, the "verbose" output should allow you to pinpoint what went wrong. You may need to explicitly set some of the ssh client options.
Client configuration - Kerberos
In order to use CERN authentication, the name of the CERN KDC(s) need to be specified explicitly.
To get this done, a working config file (with CERN as default realm) can be copied from here. Default location for this configuration file is /etc/krb5.conf on most Linuxes and Mac OS X (here Library/Preferences/edu.mit.Kerberos would work as well), on some systems (Solaris) it is in /etc/krb5/.
You can edit the krb5.conf file using the following syntax, which is accepted by both the MIT and Heimdal Kerberos implementations.
As an alternative to editing the krb5.conf directly, the below information can typically also be entered via any Kerberos-configuring user interface your vendor may provide. If you do not have "root" privileges on the client machine, you can instruct MIT Kerberos to look at a different file by setting the KRB5_CONFIG environment variable.
[realms] ; AD : This Kerberos configuration is for CERN's Active Directory realm. [libdefaults] default_realm = CERN.CH ticket_lifetime = 25h renew_lifetime = 120h forwardable = true proxiable = true ;Next two lines MAY be needed - depending on Kerberos implementation, ;do not uncomment unless you see preauthentication errors on your client. ;default_tkt_enctypes = arcfour-hmac-md5 aes256-cts aes128-cts des3-cbc-sha1 des-cbc-md5 des-cbc-crc ;allow_weak_crypto = true ... [realms] ... CERN.CH = { default_domain = cern.ch kdc = cerndc.cern.ch } ... [domain_realm] ... .cern.ch = CERN.CH ...
Once you can get TGTs from the CERN realm as described above, the
client-side Kerberos configuration is assumed to work.
Client configuration - SSH
Two flavours of the SSH protocol exist, which support Kerberos differently:
- SSH1: ancient and theoretically less secure, it had nevertheless direct support for Kerberos4, Kerberos5 and AFS. Typically, SSH programs nowadays needs to be specifically patched/recompiled to understand Kerberos4 and AFS.
- SSH2: the modern successor that does not "speak" Kerberos directly, but uses an intermediate mechanism called GSSAPI ("Generic Security Services API", which itself uses Kerberos5). It is blissfully unaware of AFS, but widely available.
A common implementation of these protocols is found in OpenSSH, widely used in the Linux world, and also part of CERN Linux distributions.
ssh clients may need some help to try Kerberos for authentication. The following options should be enabled (via the command line or in ~/.ssh/config, see man ssh_config):
- SSH1 - since separate patches are involved, not all
clients understand all of these options,
consult the man page of the version you are using.
- KerberosAuthentication: as the name implies, tells the ssh client to try Kerberos (both Kerberos4 and/or Kerberos5)
- KerberosTgtPassing: tell the client to forward the Kerberos TGT to the remote side (kerberos5 TGTs need to be marked as "forwardable" for this to work).
- AFSTokenPassing: tell the client to forward the AFS credentials.
- SSH2:
- GSSAPIAuthentication: try Kerberos5 authentication
- GSSAPIDelegateCredentials: tell the client to forward the Kerberos5 credentials to the remote side
debug1: Trying Kerberos v5 authentication. debug1: Kerberos v5 authentication accepted. debug1: Kerberos v4 authentication accepted. debug1: Kerberos v4 challenge successful. debug1: Kerberos v4 TGT forwarded (afsaccount@CERN.CH). debug1: AFS token for cell cern.ch forwarded. debug1: Next authentication method: gssapi-with-mic debug1: Delegating credentials debug1: Authentication succeeded (gssapi-with-mic).
Some common SSH errors:
- no credentials on the client:
debug1: Kerberos v5: krb5_mk_req failed: No credentials cache found debug1: Kerberos v4 krb_mk_req failed: Couldn't access ticket file (tf_util) debug1: Miscellaneous failure No credentials cache found
(this should get fixed by re-running kinit) - credentials not delegated: (this appears to effect Mac OS Tiger (and newer?)).
Symptons: You can login without having to specify a password, but you get
fs: You don't have the required access rights on '/afs/cern.ch/user/j/jouser' hepix: E: /usr/bin/fs returned error, no tokens?
Make sure you use the following options either in your local ~/.ssh/config, or in your local systemwide /etc/ssh_config (could also be at /etc/ssh/ssh_config):HOST lxplus* GSSAPITrustDns yes # not needed (even incompatible) with 10.4 to 10.6(?) vs SLC5 GSSAPIAuthentication yes GSSAPIDelegateCredentials yes
- Problem with IP aliasing (this effects Mac OS Lion).
Possible symptons:
- You are asked for a password, despite having a valid token, or
- you are logged in via your key, but you get no afs token.
debug1: Next authentication method: gssapi-with-mic debug1: Miscellaneous failure (see text) Server (host/lx64slc5.cern.ch@CERN.CH) unknown while looking up 'host/lx64slc5.cern.ch@CERN.CH' (cached result, timeout in 853 sec) debug1: An invalid name was supplied unknown mech-code 0 for mech 1 2 752 43 14 2
Here the client does a reverse lookup of the host, but does not find anything for the alias used. The only solution is to add the following options to the ssh_config.HOST lxplus* GSSAPITrustDns yes
for all clusters that use the IP alias round-robin for loadbalancing (lxplus, lxadm,...).Please read the ssh_config man pages before modifying one of the config files.
While you are at it you might want to add the
ForwardX11 yes
for convenience in the HOST lxplus* section, but in that case do not forget to put theForwardX11Trusted no
there as well, for not making your setup too insecure. It is not recommended to put the 'GSSAPIDelegateCredentials yes' or the 'ForwardX11 yes' in the global section of the systemwide ssh_config file. - Buggy GSSAPI call inside openssh (this
e.g. affects MacOS ssh
and most "vendor" ssh'es when connecting to one of the DNS-round-robin
clusters such as LXPLUS, it will fall through to password
authentication in such cases):
debug1: Next authentication method: gssapi-with-mic debug2: we sent a gssapi-with-mic packet, wait for reply debug1: Miscellaneous failure Generic error (see e-text)
ordebug1: Delegating credentials debug1: An invalid name was supplied No error
A symptom is that the credentials cache ("klist") contains a service ticket (host/lxplus123.cern.ch@CERN.CH) that does not match the machine ssh actually connected to:debug1: Connecting to lxplus [137.138.4.22] port 22.
(in this case lxplus == 137.138.4.22 == lxplus207 != lxplus123).
This bug has been patched in the CERN SLC ssh client and in some other Linux distributions, it is being tracked upstream at the OpenSSH bugzilla. If you are lucky, your man ssh_config knows about a GSSAPITrustDNS parameter - in this case please try ssh -oGSSAPITrustDNS=yes .....Other workarounds include using the SSH-1 protocol (ssh -1 ... - if your version still supports this, and supports Kerberos over SSH-1), or connecting to an individual node out of the cluster, instead of to the DNS alias (i.e. use ssh lxplus202.cern.ch instead of ssh lxplus.cern.ch). Obviously you will miss the benefits of the DNS alias - occasionally your target may be down, overloaded or simply no longer there.
Client configuration - other applications
It is possible to use Kerberos for local authentication (this is how CERN Linux machines are generally set up). However, such a setup only makes sense on machines that are permanently connected to the network, and have permanent connectivity to the CERN KDCs. In such cases, TGTs are typically obtained directly on login and get renewed via screensavers. For CERN Linux machines, a variant of Red Hat's pam_krb5 is being used to achieve this.
Besides SSH, other network services can also be made Kerberos-aware. Examples including web services (via the SPNEGO/GSSAPI authentication mechanism), ORACLE or CERN tools such as the Quattor CDB client.
Server-side configuration and troubleshooting
Kerberos is a shared-secret authentication system. This implies that both the users and the machines need to be registered at a central place, the KDC. These shared secrets are versioned, and both the machine and the KDC need to be using the same version of the secret. Most CERN Linux machines should get registered at install time, but in case of authentication problems the following may help to troubleshoot.
- Check that /etc/krb5.keytab is present, non-empty,
owned by "root" and readable only for "root".
$ ls -l /etc/krb5.keytab -rw------- 1 root root 197 Jan 14 2005 /etc/krb5.keytab
If file is missing, run cern-get-keytab as "root" to (re-)create them. It will only work on the CERN network.
Note: cern-get-keytab is available only for CERN supported Linux versions: Scientific Linux 5,6, .. (and Red Hat Enterprise Linux). If you run different system please see: msktutil paragraph below.
This program (re-)creates the machine's secret on the KDC and stores a matching version inside these files. By running it, you invalidate existing service tickets for the machine in question (users will need to re-run kinit before they can connect again). - If these files exist, verify that the "key version" number (KVNO) is correct: On the server, run
$ klist -k -t -e Keytab name: WRFILE:/etc/krb5.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 3 01/14/13 11:17:02 host/hostname.cern.ch@CERN.CH (arcfour-hmac) 3 01/14/13 11:17:02 host/hostname.cern.ch@CERN.CH (aes128-cts-hmac-sha1-96) 3 01/14/13 11:17:02 host/hostname.cern.ch@CERN.CH (aes256-cts-hmac-sha1-96)
On a client, use$ kvno host/hostname.cern.ch@CERN.CH host/hostname.cern.ch@CERN.CH: kvno = 2
to get the KDC's idea of the current KVNO. If they are different (as in the example), remove /etc/krb5.keytab and recreate them as per above.This is a frequent problem on multi-boot systems when you use the same hostname for several systems. When the keytab is recreated on one of the systems, it invalidates the keytab files of the other systems, requiring you to use a password. In /var/log/secure you will see messages like
Sep 27 14:08:55 myhost sshd[25361]: pam_krb5[25361]: authentication fails for 'jouser' (jouser@CERN.CH): Authentication failure (Success) Sep 27 14:08:55 myhost sshd[25361]: pam_krb5[25361]: TGT failed verification using keytab and key for 'host/myhost.cern.ch@CERN.CH': Key version number for principal in key table is incorrect
One possible work-around is to copy the modified keytab file to the other systems on that node using the same hostname. - Check /var/log/messages for hints why a given service cannot use Kerberos, for example if it has trouble accessing the keytab files. For sshd, you might want to run in debug mode (sshd -d -d -d -p someport) for further hints.
msktutil: acquiring keytab file on unsupported linux system.
Note:For CERN supported Linux systems please always use cern-get-keytab utility.In order to acquire keytab from CERN KDC msktutil version 0.4.2 (or newer) is needed. It may be included in your Linux distribution, if not, sources are available from msktutil site.
Obtaining initial keytab using msktutil
- configure your /etc/krb5.conf as described in this document above.
- check in Network Database if your account is authorized to manage the system for which you want to acquire the keytab.
- As root on your system execute:
# curl -k --local-port 600-700 https://lxkerbwin.cern.ch/LxKerb.asmx/ResetComputerPassword?service=host
(--local-port option is available in curl 7.15.2 or newer only) Note: This will RESET current Kerberos host identity (if any), invalidating already existing keys/keytabs. - above curl command will return a result alike:
<success>true</success> <principalname>host/HOSTNAME.cern.ch</principalname> <hostname>HOSTNAME</hostname> <samaccountname>SAMACCOUNTNAME$</samaccountname> <computerpassword>COMPUTERPASSWORD</computerpassword>
- wait ~ 20 seconds (to allow resynchronization of all Domain Controllers).
- Make sure that there are no user Kerberos credentials cached, clearing current ones:
# kdestroy; klist
- As root on your system execute:
# msktutil --update --dont-expire-password --server cerndcXX.cern.ch --computer-name HOSTNAME --service host --old-account-password COMPUTERPASSWORD (--verbose)
(please note: msktutil will reset computer account password in Active Directory to an random value, since this password is not needed anymore, for subsequent runs the existing keytab will be used to authenticate) - (to find exact name of KDC (cerndcXX), please use
$ dig -t SRV _kerberos._tcp.cern.ch
, do not use cerndc.cern.ch - that will fail) - New system keytab will be created by msktutil as:
/etc/krb5.keytab
(please verify permissions and selinux security context if SELinux in use, correct permissions are:# ls -l /etc/krb5.keytab -rw-------. 1 root root 861 Feb 25 11:17 /etc/krb5.keytab
correct selinux context is:# ls -Z /etc/krb5.keytab -rw-------. root root system_u:object_r:krb5_keytab_t:s0 /etc/krb5.keytab
use chown, chmod, chcon to change if needed )
Keytab recreation/acquiring additional service principal names using msktutil
Once initial keytab has been created (and is still valid) it can be regenerated using msktutil:- As root execute:
# msktutil --update --dont-expire-password --server cerndcXX.cern.ch (--verbose)
(existing keytab will be used for authentication).
- As root execute:
# msktutil --update --dont-expire-password --server cerndcXX.cern.ch --service HTTP (--verbose)