Xrootd 3rd Party Copy (TPC)

Native Xrootd storage:

Hardware and OS

You need a machine with a fast network connection (10Gbps or faster recommended). This machine should allow inbound traffic to xrootd port 1094, allow unrestricted outbound traffic, and run SL7 x86_64. SL6 is no longer supported. For a non-casual site, several parameters in /etc/security/limits.d need to be bump up:

• set nproc to at least 8192
• set nofile to at least 8192

One could also consider using tcmalloc or jemalloc.

Installation

Grid environment:

Make sure a standard EGI or OSG environment is installed, including

• host certificate (default /etc/grid-security/{hostcert.pem, hostkey.pem})
• CA certificates (default /etc/grid-security/certificates)
• voms certificates (default /etc/grid-security/vomsdir)

copy /etc/grid-security/{hostcert.pem, hostkey.pem} to /etc/grid-security/xrd/{xrdcert.pem, xrdkey.pem}. Keep the permission bits and change ownership of these files to whoever runs the xrootd and cmsd process.

Set up YUM repositories

Please enable the EPEL or OSG repo

Install the RPMs

• Install or upgrade Xrootd rpms

yum install xrootd xrootd-voms voms. This will also added user "xrootd" and group "xrootd" to your system if they don't already exist. Make sure you install Xrootd 5.3.1 or above.

• Files to check

From time to time, you may need to check or change the following files for configuration changes or debugging:

• /etc/xrootd/xrootd-clustered.cfg : Main Xrootd configuration file
• /etc/xrootd/Environment (SL7): Setup environment variables before xrootd and cmsd start. This file by default does not exist (see below)
• /etc/xrootd/auth_file : Xrootd Authentication / Authorization file
• /var/log/xrootd/xrootd.log : xrootd log file
• /var/log/xrootd/cmsd.log : cmsd log file

Configuration

run the following command

systemctl enable xrootd@clustered
# systemctl enable cmsd@clustered

The user and group that runs xrootd and cmsd are defined in /etc/systemd/system/multi-user.target.requires/{xrootd,cmsd}@clustered.service, so you need to adjust in these two files. Once you make change to user/group, make sure owner/group of /var/log/xrootd, /var/spool/xrootd /var/spool/xrootd and their sub-directories are all updated.

One can also add a line

EnvironmentFile=/etc/xrootd/EnvironmentFile

to the [Service] session in the above systemd service files. Then add X509_CERT_DIR and X509_VOMS_DIR definitions to /etc/xrootd/Environment (this file will be "sourced" as a shell script). Please make the permission of this file is 755.

Xrootd on Posix Storage Systems

Main configuration file /etc/xrootd/xrootd-clustered.cfg

## "redirector" should be full qualified DNS name (e.g. hostname -f). 
## If you are running a single xrootd service, leave the "set redirector" line below as it is.
#set redirector = www.google.com
#all.manager $(redirector):1213
all.adminpath /var/spool/xrootd/var/spool
all.pidpath   /var/run/xrootd/var/run

all.export <my.storage.path e.g. /data/dir1>
xrd.port 1094
xrootd.chksum adler32

# TLS:
xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
xrd.tlsca certdir /etc/grid-security/certificates
xrootd.tls capable all

#if $(redirector)
#    all.role manager
#    cms.dfs lookup distrib redirect immed
#    cms.delay startup 30
#    pss.ckslib adler32 /usr/lib64/libXrdPss.so
#else
#    all.role server

    # Enable security
    xrootd.seclib /usr/lib64/libXrdSec.so

    # X509 VOMS security in xroot protocol
    sec.protparm gsi -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:dbg
    sec.protocol /usr/lib64 gsi -dlgpxy:1 -exppxy:=creds -ca:1 -crl:3 -gridmap:/dev/null

    Xrootd TPC
    ofs.tpc fcreds ?gsi =X509_USER_PROXY ttl 60 70 xfr 100 autorm 

    Authorizatoin
    acc.audit deny
    acc.authdb /etc/xrootd/auth_file
    acc.authrefresh 60
    ofs.authorize
#fi

Xrootd Proxy as a DTN

Main configuration file /etc/xrootd/xrootd-clustered.cfg

## "redirector" should be full qualified DNS name (e.g. hostname -f). 
## If you are running a single xrootd service, leave the "set redirector" line below as it is.
#set redirector = www.google.com
#all.manager proxy $(redirector):1213
all.adminpath /var/spool/xrootd/var/spool
all.pidpath   /var/run/xrootd/var/run

all.export <my.storage.path e.g. /data/dir1>
xrd.port 1094
xrootd.chksum adler32

# For DTN only
pss.origin <mybackend.storage.xrootd.door>:<port>

# TLS:
xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
xrd.tlsca certdir /etc/grid-security/certificates
xrootd.tls capable all

#if $(redirector)
#    all.role proxy manager
#    cms.dfs lookup distrib redirect immed
#    cms.delay startup 30
#    pss.ckslib adler32 /usr/lib64/libXrdPss.so
#else
#    all.role proxy server
    ofs.osslib /usr/lib64/libXrdPss.so

    # Enable security
    xrootd.seclib /usr/lib64/libXrdSec.so

    # X509 VOMS security in xroot protocol
    sec.protparm gsi -vomsfun:/usr/lib64/libXrdVOMS.so -vomsfunparms:dbg
    sec.protocol /usr/lib64 gsi -dlgpxy:1 -exppxy:=creds -ca:1 -crl:3 -gridmap:/dev/null

    # Xrootd TPC
    ofs.tpc fcreds ?gsi =X509_USER_PROXY ttl 60 70 xfr 100 autorm pgm /etc/xrootd/xrdcp-tpc.sh

    # Authorization
    acc.audit deny
    acc.authdb /etc/xrootd/auth_file
    acc.authrefresh 60
    ofs.authorize
#fi

The xrootd TPC script

$ cat /etc/xrootd/xrdcp-tpc.sh
#!/bin/sh
set -- `getopt S: -S 1 $*`
while [ $# -gt 0 ]
do
  case $1 in
  -S)
      ((nstreams=$2-1))
      [ $nstreams -ge 1 ] && TCPstreamOpts="-S $nstreams"
      shift 2
      ;;
  --)
      shift
      break
      ;;
  esac
done

src=$1
dst=$2
xrdcp --server $TCPstreamOpts -f $src root://$XRDXROOTD_PROXY/${dst}

Please set permission to 755.

Access control file

The access control file, /etc/xrootd/auth_file should have the following lines:

= atlasprod o: atlas g: /atlas r: production
x atlasprod <my.storage.path> rwildn
g /atlas <my.storage.path> rl
g /cms <my.storage.path> rl
g /dteam <my.storage.path>/dteam/doma rwild

• line 1: define a special compound id "atlasprod" to identify all users within ATLAS VO with "/atlas/Role=production".
• line 2: grant this "atasprod" access permission 'rwildn'
• line 3: grant all other ATLAS users (all of those in VO group /atlas) permission 'rl'
• line 4: grant all /cms users (all users in VO group /cms) permission 'rl'
• line 5: grant all /dteam users (all users in VO group /dteam) permission 'rwild'

Note that Xrootd will stop processing this list once it find a matching rule. Please refer to Xrootd config doc for more info about Xrootd's access control.

Adding HTTP(s) protocol and HTTP TPC (with Macaroon)

In the above two configurations (Xrootd on Posix storage and Xrootd as DTN), one can add the following to the appropriate section of the configuration file (e.g. append to the end) to support HTTP and HTTP TPC using macaroon. In this doc, when HTTP is mentioned, it really means HTTPS.

#if ($redirector)
#  http.desthttps yes
#fi

# HTTP Protocol
if exec xrootd
    xrd.protocol http libXrdHttp.so
fi
http.selfhttps2http no

# X509 VOMS in HTTP(s) protocol
http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
http.secxtractor libXrdVoms.so
http.header2cgi Authorization authz

# Set all.sitename in order to use Macaroon
all.sitename MYSITE

# HTTP TPC, see https://twiki.cern.ch/twiki/bin/view/Main/XRootDoverHTTP#Enable_Third_Party_Copy
http.exthandler xrdtpc libXrdHttpTPC.so
# Macaroons support, see: https://twiki.cern.ch/twiki/bin/view/Main/XRootDoverHTTP#Macaroons_Support
http.exthandler xrdmacaroons libXrdMacaroons.so
macaroons.secretkey /etc/xrootd/macaroon-secret
ofs.authlib libXrdMacaroons.so

Note that the referred URL https://twiki.cern.ch/twiki/bin/view/Main/XRootDoverHTTP is marked as obsoleted. But it still contains useful info with regards to /etc/xrootd/robots.txt and /etc/xrootd/macaroon-secret, Please read carefully (the macaroon secret needs to be identical on all of your severs, including redirector).

Also note that by default, xroot protocol and http protocol share the same port. xroot and http also share the authorization file /etc/xrootd/auth_file.

HTTP TPC misc:

• Tell internet search engines to stay away:

       $ cat robots.txt 
       User-agent: *
       Disallow: / 

• To generate a macaroon secret, run command openssl rand -base64 -out macaroon-secret 64. All nodes, include redirectors should have the same macaoon secret.

Start/Stop services

On SL7 platform, the command to start and stop xrootd and cmsd services are:

systemctl start xrootd@clustered (or stop)
# systemctl start cmsd@clustered (or stop)

Please refer to this page for detail.

To setup a cluster of xrootd proxy, both xrootd and cmsd services need to run on redirector and individual proxies. To setup a single xrootd proxy, only xrootd service is needed.

Credential delegation

The client will delegate its x509 credential to the destination in order to fetch data from the source. (Currently xrootd proxy can not use delegated user credential to access remote file on user's behave. This is still a work in progress. Until that is completed, a credential with read access to VO's files is needed by the xrootd proxy. In the case of ATLAS and CMS, this credential is a X509 proxy with VOMS attributes. The credential needs to be periodically refreshed before it expired).

Test the setup

The easiest way to verify upload/download via xrootd and HTTP(s) protocol is to use gfal-copy. To verify xrootd TPC and HTTP TPC, one can use the FTS client:

fts-transfer-submit --overwrite -s https://fts3-devel.cern.ch:8446 root://source.edu:port//dir/myfile root://destination:port//dir/myfile.new

replace root by https to test HTTP TPC.

For xrootd protocol, one can also use

/usr/bin/xrdcp --tpc delegate only -f root://source.edu:port//dir/myfile root://destination.edu:port//dir/myfile.new

Make sure /usr/bin/xrdcp --version shows version 5.2.0 or above.

TPC with dCache

TPC with DPM

For DPM (>= 1.10) DOME, please refer to DPM deployment guide. For more technical detail on DPM Xrootd TPC configuration, including x509 user proxy delegation, check this DPM manual installation Twiki

TPC with EOS

TPC with CEPH

TPC with Storm

If the storage systems at a Storm site is posix (such as GPFS or Lustre), please refer to the Posix storage section.

Deployment Status

EOS & CASTOR

Both EOS and CASTOR supports Xrootd TPC. X509 robot certificates are not available so transferring data to EOS and CASTOR from outside will not work, if the outside endpoints requires X509 authentication. Transfer between EOS and CASTOR does not require X509 authentication, and therefore works well.

Xrootd & Posix

• SLAC: a cluster of (containerized) Xrootd DTNs to move data in and out of the SLAC xrootd storage.
• NERSC: a single Xrootd DTN to move data in and out of their Lustre filesystem.
• Uni-Bonn: seven Xrootd DTNs with one redirector, data on shared CephFS storage

Integration with FTS and RUCIO

Completed. ATLAS has been able to use RUCIO to drive, through FTS to move data between Xrootd endpoints at EOS, CASTOR, SLAC and NERSC.

Open Tickets / Issues

• gfal2-xrootd plugin
  • Implement performance markers logic as in gridftp plugin (DMC-1016)
• Xrootd
  • The tpcTimeout passed to the XrdCl::CopyProcess is not honored, so the default 1800 sec timeout is always used. ( no ticket open yet but confirmed by the devs)

-- AlessandraForti - 2018-08-01