Bearer Tokens

Bearer tokens are a topic which has gained a lot of traction within WLCG. Eventually, the goal is for bearer tokens to replace X509 authentication in the (hopefully not-so-far) future.

This page describes what bearer tokens are in Gfal2 and how they are used.

Why bearer tokens?

In the simplest words, a bearer token is a series of bytes which grants the holder of the token certain permissions on a server. The client presents the token to the server as means of authorization, when performing a given operation. It doesn't matter what is in the token or whether the client can read it. What matters is that the storage performs authorization based on the token.

This brings a level of simplicity and mobility on the client's part. No complex cryptography or requirements of agreed-upon authorities is involved, as is the case for X509 proxy certificates. More so, delegation becomes very easy. If you have write access to a storage, passing your token to another entity allows them to perform the write operation. If that token also contains some identity information, combined with the proper instructions, the entrusted entity can write the file and announce it is doing so on your behalf. Which, in fact, is what happens during HTTP-TPC.

Bearer tokens can be issued either directly by the Storage Element (colloquially called macaroons, albeit sometimes incorrectly, as there are other types available) or by a dedicated Identity and Authorization entity (as is the case for OIDC tokens).

Gfal2 and bearer tokens

Gfal2 offers bearer tokens support for the HTTP protocol. This is done with the help of a credentials map. Gfal2 performs HTTP operations by using the Davix library, preparing the request parameters and passing them to the library for the actual HTTP request. There are many request parameters (many of them configurable), including some related to authentication. Gfal2 uses the following algorithm to set up the authentication parameters:

• Load the X509 credentials (proxy or user/key pair)
  • Will always be used as fallback
• Identify if Cloud endpoint by URI protocol
  • Load the cloud credentials from the Gfal2 environment
• Search for bearer tokens in the credentials map for the given path
  • If not found, search for bearer tokens in the credentials map for the given host
  • If still not found, attempt to retrieve bearer token from Storage Element (requires gfal2 >= v2.20.0)

The Gfal2 credentials map

The Gfal2 credentials map is a mechanism that allows Gfal2 users to associate a type of credentials to a particular path prefix.

Notice the type of credential.
This means that a path may have multiple types of credentials associated to it. The available types are: X509_CERT, X509_KEY, BEARER, USER, PASSWORD.

Notice the path prefix.
Retrieving from the credentials map takes as input a type of credential and a desired path. As the paths stored in the credentials maps are prefixes, they will attempt to match your path, starting from the longest match towards the smallest.

More so, there are 3 important authentication variables that may be loaded in the Gfal2 environment, namely [X509] CERT=<cert>, [X509] KEY=<key> and [BEARER] TOKEN=<token>. If no specific credential object is retrieved, then the credential is loaded from the Gfal2 environment, if available.

Some things are better explained with a (Python) example:

# Set general [BEARER] TOKEN option
context.set_opt_string("BEARER", "TOKEN", "<token>")

# Create Gfal2 credential objects
token_cred = gfal2.cred_new("BEARER", "<bearer-token>")
macaroon_cred = gfal2.cred_new("BEARER", "<macaroon>")

# Set credential objects in the credentials map
gfal2.cred_set(context, "https://grid-storage/path/dir/file", token_cred)
gfal2.cred_set(context, "https://grid-storage/", macaroon_cred)

# Retrieves the <token_cred> object
gfal2.cred_get(context, "BEARER", "https://grid-storage/path/dir/file")

# Retrieves the <macaroon_cred> object
gfal2.cred_get(context, "BEARER", "https://grid-storage/path/dir/")

# Retrieves the general "[BEARER] TOKEN=<token>" value
gfal2.cred_get(context, "BEARER", "https://other-grid-stroage/path/")

In the example, we have the [BEARER] TOKEN=<token> loaded as a Gfal2 environment option and 2 user-set credential objects:

• The first request, for the full path, we get the longest match, which is the token_cred object
• Second request, for the containing directory, we get the macaroon_cred, which was set for the full storage, but is in fact the longest user-set match for us
• The third time, we request credentials for a different storage, which has no user-set credentials associated to it. Therefore, we receive the token from the Gfal2 environment

Important to note, there is a correlation between the Gfal2 environment and the shell environment. As soon as one is loaded, the others are not processed anymore.

Automatically requesting Storage Element issued bearer tokens

This feature is available starting from version 2.20.0. It allows Gfal2 to request a bearer token from the storage element, when no such token could be retrieved from the credentials map. This operation is transparent to the user and the lifecycle management of these tokens is handled by Gfal2.

To enable this feature, the following configuration must be set in the Gfal2 HTTP plugin config file (generally /etc/gfal2.d/http_plugin.conf):

RETRIEVE_BEARER_TOKEN=true

Important to note, when Gfal2 is contacting the Storage Element to request a bearer token, X509 certificate is used for the authentication.

Implementation details

When requesting SE-issued bearer tokens, sooner or later, you'll run into the scenario where Gfal2 performs a read operation on <path>, no token is available, thus a<read-bearer-token> is obtained, but later Gfal2 has to do a write operation on the same <path>, where it will find the <read-bearer-token> in the credentials map.

To work around this problem, Gfal2 keeps an additional token access map, in the form of <token, access>. Once a token is retrieved from the credentials map, the read/write permissions are checked against the token access map. If the permissions are not sufficient (namely, read token for write operation), the token is discarded and a new token with the needed permissions is obtained for the path.

In case a token is retrieved from the credentials map but this token cannot be found in the token access map, then the token is assumed to be user-set and will be used without any additional checks.

Note: A previous version of Gfal2 relied on the x509-scitokens-issuer-client package for the token retrieval. Now, the functionality is fully implemented in the Gfal2 HTTP-plugin.

Examples for Gfal2 and SE-issued tokens

Make sure gfal2 v2.20.0, python2/3-gfal2 v1.11.0 and python2/3-gfal2-util v1.7.0 are installed. The packages are available via the DMC RC or DMC production repositories.

Prerequisites:

• Set RETRIEVE_BEARER_TOKEN=true in the Gfal2 HTTP-plugin config file (default /etc/gfal2.d/http_plugin.conf)
• Make sure you have a valid proxy certificate

Submit a transfer

$ gfal-copy <http_src> <http_dst> [-vvv --logfile=gfal2.log]
# (Optional) Check the log file for mentions of `SEToken` (requires debug logging)

Obtain SE-issued token via Python bindings

import gfal2

ctx = gfal2.creat_context()

# retrieve_token(<path>, <issuer> (default=""), validity (default=60), [list of capabilities])
ctx.retrieve_token("<http_storage_endpoint>", "", 60, ["LIST", "DOWNLOAD"])

# Out of convenience, we offer a second method signature with write_access=<True/False>
# write_access=False <==> ["LIST", "DOWNLOAD"]
# write_access=True  <==> ["LIST", "UPLOAD", "MANAGE", "DELETE"]
# retrieve_token(<path>, <issuer> (default=""), validity (default=60), write_access (default=False))

Obtain SE-issued token from the command line

$ gfal-token --validity 60 <http_storage_endpoint> READ,LIST,MANAGE

What about FTS?

Until now, FTS was the service requesting the tokens from the Storage Elements and passing them to Gfal2. It can still do this, as v2.20.0 is backwards compatible with FTS versions. However, starting with FTS v3.11.0, a new server configuration RetrieveSEToken=<true|false> is introduced to enable or disable FTS requesting SE-issued tokens.

A log message is printed at INFO level:

INFO    Mon, 20 Sep 2021 17:20:00 +0200; Configured to skip retrieval of SE-issued tokens

If DEBUG level is enabled, search for SEToken mentions:

DEBUG   Mon, 20 Sep 2021 17:20:00 +0200; (SEToken) Set bearer token in credential_map[https://eospps.cern.ch/eos/opstest/dteam/file.test] (access=read) (validity=180)
DEBUG   Mon, 20 Sep 2021 17:20:00 +0200; (SEToken) Set bearer token in credential_map[https://prometheus.desy.de:2443/VOs/dteam/file.test] (access=read) (validity=180)
DEBUG   Mon, 20 Sep 2021 17:20:01 +0200; (SEToken) Found token in credential_map[https://prometheus.desy.de:2443/VOs/dteam/file.test] (access=read) (needed=read)
DEBUG   Mon, 20 Sep 2021 17:20:01 +0200; (SEToken) Set bearer token in credential_map[https://prometheus.desy.de:2443/VOs/dteam] (access=read) (validity=180)
DEBUG   Mon, 20 Sep 2021 17:20:01 +0200; (SEToken) Found token in credential_map[https://eospps.cern.ch/eos/opstest/dteam/file.test] (access=read) (needed=read)
DEBUG   Mon, 20 Sep 2021 17:20:01 +0200; (SEToken) Found token in credential_map[https://prometheus.desy.de:2443/VOs/dteam/file.test] (access=read) (needed=write)
DEBUG   Mon, 20 Sep 2021 17:20:01 +0200; (SEToken) Invalidating token for path=https://prometheus.desy.de:2443/VOs/dteam/file.test because write access is missing
DEBUG   Mon, 20 Sep 2021 17:20:02 +0200; (SEToken) Set bearer token in credential_map[https://prometheus.desy.de:2443/VOs/dteam/file.test] (access=write) (validity=36)
DEBUG   Mon, 20 Sep 2021 17:20:05 +0200; (SEToken) Found token in credential_map[https://prometheus.desy.de:2443/VOs/dteam/file.test] (access=write) (needed=read)

Please report any bugs, inconsistencies, unexpected behavior or documentation improvements at:

• dmc-devel@cern.ch or DMC JIRA
• fts-devel@cern.ch or FTS JIRA