TapeRestAPI

Introduction

The main goal of this project is to come with new, simple, minimalistic and uniform interface to manage tape transfers. It is going to replace SRM which is very complex with dependencies on old legacy libraries and also proprietary method built into xroot protocol used by CTA. New specification v1 (finished in May 2022) is based on standard HTTP REST API with access authorized by X.509 proxy certificates or WLCG JWT tokens. It was designed by storage (CTA, dCache, StoRM) and transfer clients (gfal2, FTS) developers.

Testbeds

Site	Storage Version	Contact	VO	TAPE REST	Storage WebDAV endpoint	Notes
BNL	dCache/8.2.8	Vincent Garonne		https://dcqos001.usatlas.bnl.gov:3880/api/v1/tape/	https://dcqos001.usatlas.bnl.gov:443/pnfs/usatlas.bnl.gov/qostape/
CERN	CTA	Julien Leduc		https://ctadevfts.cern.ch:8444/api/v1	https://ctadevfts.cern.ch:8444/eos/ctaeos/preprod/	final v1 deployed
DESY	dCache
FNAL	dCache	Dmitry Litvintsev	dteam	https://fndcaitb4.fnal.gov:3880/api/v1/tape	https://fndcaitb4.fnal.gov:2880/pnfs/fnal.gov/usr/dteam
RAL	CTA	Alastair Dewhurst				plan to use in production ~ Q2 2023
INFN-T1	StoRM	Francesco Giacomini				in development
IN2P3 -CC	dCache 9.2.14	Adrien Georget	atlas	https://ccdcamcli06.in2p3.fr:3880/api/v1/tape	https://ccdavatlas.in2p3.fr:2880	deployed and tested with LHCb.

Clients

• Gfal2 2.21.0 ( DMC-1301): released as v2.21.0 ( DMC repo)
  • Brings full support for Tape REST API operations
• FTS ( repo)
  • Release timeline
    • v3.12.0 released
      • Brings support for /release and /archiveinfo functionality
      • Enables full HTTP transfer-in / transfer-out from CTA deployments (RAL CTA and CERN CTA)
    • v3.12.2 ( FTS-1803) release mid-autumn: brings support for /stage operation
      • Brings full support for HTTP Tape operations
  • Authentication support
    • All v3.12.x releases offer token support only for the transfer part (this includes the /release call). In other words: tape operations will use only X.509 authentication
    • Focus on X.509 to be ready for production use during 2023
    • Support for tokens + storage.stage:/ scope comes in future (doesn't yet have defined timeline)
      • Tape is not considered to be used for next WLCG Data Challenge
        • Included in the DC21 to check tape systems are ready for LHC Run3 data rate
  • Known issues
    • versions up to 3.12.8 did not automatically call evict after successful file transfer FTS-1913

TAPE REST API examples

• FTS Workshop presentation with examples how to use REST API
• basic TAPE REST interaction with curl (you can use also gfal-bringonline with https/davs protocol scheme which also trigger TAPE REST calls)

CURLX509="--capath /etc/grid-security/certificates --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u)"
OUTPUT=$(mktemp)
trap 'rm "${OUTPUT}"' EXIT

TAPESRC=https://dcdoor-tape.usatlas.bnl.gov/pnfs/usatlas.bnl.gov/BNLT1D0/dumps/dump_20210926
echo "Source URL: ${TAPESRC}"

[[ $TAPESRC =~ ^([^:]+://[^/]+)(/?.*)$ ]] && URLBASE="${BASH_REMATCH[1]}"
[[ $TAPESRC =~ ^([^:]+://[^/]+)(/?.*)$ ]] && URLPATH="${BASH_REMATCH[2]}"

WELLKNOWN=${URLBASE}/.well-known/wlcg-tape-rest-api
echo "Well known TAPE configuration: ${WELLKNOWN}"

TAPEREST=$(curl -s --capath /etc/grid-security/certificates ${WELLKNOWN} | jq -r '.endpoints | .[] | .uri')
echo "Tape REST API endpoint: ${TAPEREST}"

# status
#gfal-xattr ${TAPESRC} user.status
JSON='{ "paths":["'${URLPATH}'"]}'
STATUS=$(curl -s -w "%{response_code}" ${CURLX509} -X POST -H 'Content-Type: application/json' -d "${JSON}" -o "${OUTPUT}" ${TAPEREST}/archiveinfo)
echo "File status (HTTP ${STATUS}): $(cat ${OUTPUT})"

# create
#gfal-bringonline ${TAPESRC}
JSON='{ "files":[{ "path": "'${URLPATH}'" }] }'
STATUS=$(curl -s -w "%{response_code}" ${CURLX509} -X POST -H 'Content-Type: application/json' -d "${JSON}" -o "${OUTPUT}" ${TAPEREST}/stage)
REQID=$(cat "${OUTPUT}" | jq -r .requestId 2> /dev/null)
echo "Create stage request (HTTP ${STATUS}): $REQID"

# show
STATUS=$(curl -s -w "%{response_code}" ${CURLX509} -X GET -H 'Content-Type: application/json' -o "${OUTPUT}" ${TAPEREST}/stage/${REQID})
echo "Show stage request (HTTP ${STATUS}): $(cat ${OUTPUT})"

# cancel
JSON='{ "paths":["'${URLPATH}'"]}'
STATUS=$(curl -s -w "%{response_code}" ${CURLX509} -X POST -H 'Content-Type: application/json' -d "${JSON}" ${TAPEREST}/stage/${REQID}/cancel)
echo "Cancel stage request (HTTP ${STATUS})"

# release
JSON='{ "paths":["'${URLPATH}'"] }'
STATUS=$(curl -s -w "%{response_code}" ${CURLX509} -X POST -H 'Content-Type: application/json' -d "${JSON}" ${TAPEREST}/release/${REQID})
echo "Release stage request (HTTP ${STATUS})"

gfal2 test results and examples

• CTA July 22, 2022 tests
• dCache July 8, 2022 tests
• dCache July 22, 2022 tests

FTS test results and examples

• stress tests with 10k, 100k and 1M FTS transfers
• CTA and dCache implementation stress tested with 1M FTS staging requests

Automatic TAPE REST tests

• Gfal2 test suite for functionality necessary for FTS tape management prepared by FTS developers
• cover just subset of TAPE REST API (remaining TAPE REST functionality tests on TODO list)

Storage configuration

dCache

• minimum required version 8.2.22
  • important bugfix to support file path length up to 1024 characters (earlier versions supported just 256, database schema update)
  • include updated bulk default configuratoins optimized for WLCG / FTS use-cases
    • dCache limits were tuned for match default values for staging requests in FTS (StagingBulkSize =200, StagingConcurrentRequests =5000)
    • these defaults doesn't take into account multiple FTS servers operated usually by experiments and it may be necessary to increase bulk.limits.max-requests-per-user to multiples of FTS servers
• sites that rely on non-default webdav.root needs at least dCache 9.2.14 with support to specify frontend.root
• dCache frontend service is used to publish tape configuration on well known url
• publish configuration JSON on =frontend.wellknown!wlcg-tape-rest-api.path= - this name was deprecated in 9.2 and replaced by dcache.wellknown!wlcg-tape-rest-api.path, see upgrade notes
  • default location /var/lib/dcache/httpd/wlcg-tape-rest-api.json
  • JSON file content may be inspired by FZK configuration
  • this /.well-known/wlcg-tape-rest-api endpoint must be available on tape WebDAV host
    • the https://host:port part of TURL returned in case your storage currently provide SRM+HTTPS transfers
  • If you define frontend.wellknown!wlcg-tape-rest-api.path in your general dcache.conf, it will work not only for your Frontend(s), but also for your WebDAV doors! There is no webdav.wellknown!wlcg-tape-rest-api.path.
  • different webdav doors can use different frontend.wellknown!wlcg-tape-rest-api.path (this configuration applies not just to frontend service, but also webdav service)
• allow connections to the advertised TAPE REST endpoint uri configured in wlcg-tape-rest-api.json
• dCache bulk service has to be enabled

known issues

• FTS doesn't cleanup requests (by design) and current dCache keeps request in database forever
  • this was fixed in 9.2 branch which automatically archive old requests

CTA

• deployment plans presented in March WLCG BDT meeting

Timeline

Recent update was presented at June DOMA BDT meeting and some sites would like to use this method as soon as in Q1 2023.

Crystal ball predictions for moving production sites to TAPE REST:

• Second half of 2022 will be for testbed instances and software verification.
• Leading-edge T1s will have production instances in 2023 available for tests.
• Site-by-site migration during 2023; ask all sites to finish by winter shutdown 2023/2024?

At the beginning of February 2023 we started with deployment (tests) with CERN CTA and BNL dCache team. Once we see TAPE REST is working fine in production for these early adopter we'll ask all remaining T1 sites to come with deployment plan. Right now we still target winter 2023/24 as our deployment deadline.

• March/April 2023: FZK started with ATLAS LOCALGROUPTAPE configuration GGUS:160869, on April 19 enabled also for DATATAPE + MCTAPE
• March 2023: deployed on CERN CTA, on April 17 ATLAS enabled its production transfers from CERN CTA using new TAPE REST
• May 23 2023: ATLAS asked sites to come with timescale when they will be able to configured TAPE REST (we would like to move all sites before the end of this year)
• June 7 2023: ATLAS asked PIC GGUS:162244 to enable TAPE REST, because PIC already use dCache 8.2.22, but we could not move forward because of missing support for frontend.root

-- PetrVokac - 2022-06-22