A third-party transfer is where data is transferred directly between two storage systems, without relaying the data through some intermediate service, such as FTS. Currently (almost) all data transfers in WLCG are done using GridFTP. Through the DOMA-TPC working group, WLCG is investigating two alternatives to GridFTP: one based on HTTP (called HTTP-TPC) and another based on xrootd (called xrood-TPC).

dCache supports both HTTP- and xrootd-based third-party transfers.

This page contains information on how to configure your dCache instance to enable these two new TPC protocols.

HTTP-TPC

HTTP third-party transfer (HTTP-TPC) is available in all currently supported versions of dCache.

To enable HTTP-TPC, your dCache cluster MUST run at least one webdav service and MUST run at least one transfermanagers service.

The simplest deployment has a single transfermanager service and one (or more) webdav doors. Currently dCache only supports a single transfermanager service per webdav door, with the association controlled by the webdav.service.transfermanager configuration property.

You MUST have a consistent IPv6 deployment: if you have a DNS AAAA record for nodes hosting your WebDAV doors, then those nodes MUST accept TCP connections using the advertised IPv6 address(es). WebDAV and pool nodes with only IPv4 addresses will still work, but only for transfers where the remote party has an IPv4 address.

You MUST configure your dCache cluster so that macaroons are enabled. This is controlled by the dcache.enable.macaroons configuration property (or webdav.enable.macaroons when the configuration targets just WebDAV doors). Macaroon support is enabled by default.

You MUST NOT configure the WebDAV doors so that they require the client to authenticate via X.509. This is controlled by the webdav.authn.require-client-cert property. The default value for this property is false so, by default, dCache is compatible with HTTP-TPC.

All pools participated in HTTP-TPC transfers MUST have the IGTF trust store installed. Typically, the trust store is installed from packages (RPM or deb) which installs files located in the `/etc/grid-security/certificates` directory. The pool nodes also require up-to-date CRLs, which is normally achieved by running the 'fetch-crl' script periodically.

In general, no changes to dCache's permissions/authorisation are needed.

The DOMA-TPC testbed has an opt-in policy. This means a dCache instance is only included if the site running that instance has agreed explicitly to participate in the DOMA-TPC testbed. If your dCache instance is part of the DOMA-TPC testbed then you MUST allow write access to members of the dteam virtual organisation because the testing is done using the dteam VO.

xrootd-TPC

xrootd third-party copy (xrootd-TPC) is available in all currently supported versions of dCache. Some features are only available in newer dCache versions; see below for details.

xrootd-TPC transfers may be done either in unauthenticated mode, or with GSI (X509) authentication.

xrootd-TPC from dCache to another xrootd server

Very few changes in the dCache door were needed to accomplish this. If dCache is merely to serve as file source, then all that is needed is to update to version 4.2+ on the nodes running the xrootd doors.

xrootd-TPC from another xrootd server to dCache, or between dCache instances

As per the protocol, the destination pulls/reads the file from the source and writes it locally to a selected pool. This is achieved by an embedded third-party client which runs on the pool. Hence, using dCache as destination means the pools must also be running dCache 4.2+.

Changes to dCache configuration for authenticated (GSI) transfers

For dCache as source, gPlazma configuration is identical to that needed for normal two-party reads and writes. The DN of the credential used by the trigger client should as usual be mapped to a user.

To use dCache as xrootd-TPC destination, one necessary additional step needs to be taken. For all pools that will receive files through xrootd-TPC, the GSI service provider plugin must be loaded by including this in the configuration or layout:

         pool.mover.xrootd.tpc-authn-plugins=gsi

Third-party authentication

In dCache 5.0 and 5.1, the full (rendezvous-key) xrootd-TPC protocol is used. There are two ways of providing authentication capability to the third-party client on the pools in these cases:

• Generate a proxy from a credential that will be recognized by the source, and arrange to have it placed (and periodically refreshed) on each pool that may be the recipient of files transferred via xrootd-TPC. The proxy path must be indicated to dCache by setting this property:

         xrootd.gsi.tpc.proxy.path={path-to-proxy}

• If this property is left undefined, dCache will auto-generate a proxy from the hostcert.pem/hostkey.pem of the node on which the pool is running.

With dCache 5.2, full delegation of the user proxy is available; no configuration is necessary to enable this feature, as it will be the preferred mode, falling back to the above-mentioned methods only in the absence of a delegated proxy (for instance, if the client being used is pre-4.9).

Insofar as the authentication of third-party clients is concerned (on a dCache source door):

• in dCache 5.0, mapping the DNs of the (non-delegated) credential used by a third-party client is necessary. For instance, if a robocert is used, its VO primary fqn (such as dteam) will need to be mapped on the dCache end.

• in dCache 5.1, we introduced the ability to accept a rendezvous-key xrootd-TPC connection without further authorization; as long as the issuer for the credential used by the xrootd-TPC client is recognized by dCache, xrootd-TPC will go forward without requiring that its DN be mapped at the dCache source/door.

• in dCache 5.2, with proxy delegation, nothing more than the normal GSI setup for the door should be necessary.

Signed hash verification (kXR_sigver) support

The embedded third-party client will honor signed hash verification if the source server indicates it must be observed.

Starting with dCache 5.0, the dCache door/server will also provide the option to enable signed hash verification.

However, there is a caveat here. Since dCache redirects reads from the door to a selected pool, and since the subsequent connection to the pool is unauthenticated (this has always been the case; the connection fails if the opaque id token dCache gives back to the client is missing), the only way to get signed hash verification on the destination-to-pool connection is to set the kXR_secOFrce flag. This means that the pool will then require unix authentication from the destination and that it will expect unencrypted hashes.

While the usefulness of unencrypted signed hash verification is disputable, the specification nevertheless provides for it, and this was the only way, short of encumbering our pool interactions with yet another GSI handshake, to allow for sigver on the dCache end at all, since the main subsequent requests (open, read, etc.) are made to the pool, not the door.

dCache 5.0 will provide the following properties to control security level and force unencrypted signing:

          dcache.xrootd.security.level={0-4}
          dcache.xrootd.security.force-signing={true,false}

In the case that the latter is set to true, and one anticipates there will be xrootd-TPC transfers between two dCache instances or two dCache doors, one also would need to include the unix service provider plugin in all the relevant pool configurations:

          pool.mover.xrootd.tpc-authn-plugins=gsi,unix

Troubleshooting

• Remote copy failed with status code 0: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
  • webdav.authn.require-client-cert configuration option set to true which is not compatible with HTTP-TPC
• Transfer failed: failure: Failed to select pool: No write pools configured outside of a linkgroup.
  • no spacetoken assigned to directories => transfers must be executed with spacetoken parameter
  • no standard way to specify destination spacetoken for xrootd and webdav protocol
  • necessary to assign corresponding WriteToken to all directories (use chimera pushtag)

-- PaulMillarExCern - 2018-10-29