StoRM upgrade task force

Introduction

The third-party-copy sub-group of the DOMA working group is dedicated to improving bulk transfers between WLCG sites. All WLCG sites are required to upgrade to the releases of storage implementation which support third-party-copy. For StoRM storage implementation the recommended versions are those which are equal or higher than 1.11.19. Another important feature which has to be provided by the WLCG storage services is enabling of the Storage Resource Reporting (SRR). The task force will follow up migration of the WLCG StoRM sites to version 1.11.19 and higher and enabling of SRR

Initial status

STORM versions of the WLCG StoRM sites as of 27.05.2020

• According to CRIC there are 24 sites which are running StoRM and are used by the WLCG VOs. The number can be higher since for some of sites CRIC is lacking information about storage implementation and implementation versions the sites are running.
• 5 sites are currently running StoRM instances with version 1.11.15 and 2 are running 1.11.17
• There are 3 sites with unknown StoRM version in CRIC and BDII
• 13 sites need upgrade

Current activities

• Ticket sites for upgrade to version 1.11.19 or higher and to enable SRR in case site is already running required version
• Follow up on progress

Site Instructions

Step by step instructions and useful links are listed below. In case you have questions or comments do not hesitate to contact storm-devel@listsNOSPAMPLEASE.infn.it or storm-upgrade-tf@cernNOSPAMPLEASE.ch

General StoRM instructions

• The recommended version for StoRM TPC is v1.11.19.
• If you want to upgrade your StoRM site to the latest version, please read carefully and follow the StoRM latest version upgrading instructions.

SRR

• Instructions to enable SRR
• File should be updated every half an hour

Update information for your site in CRIC

After changes performed on your service, please, update information in CRIC

Authentication & authorization step

• Go to WLCG cric server , click Core (menu on the top of the page) -> Services. Enable filtering, by clicking on the 'Filter' button and select your site. By default , you won't see implementation and implementation version columns in the table. In order to see this info, you need to click on 'Columns' and then select corresponding columns in the drop down list.

• You should be able to list all CRIC entities (sites (GocDB /OIM and experiment-specific ones), federations, pledges, services, storage protocols and queues) without authentication. However, once you would like to see details of any particular entity, you would be asked to login.

• Those who are registered in the CERN DB, please, use SSO authentication. Authentication with certificate is not yet enabled on this instance, will come soon.

• Those who are not registered in the CERN DB would need to ask for CRIC local account. Please, send a mail to

• As soon as you are logged in, you will be able to see details of any CRIC entity, however in order to edit in information, one would need to get specific privileges.
• As soon as you are authenticated, you will see 'Request privileges' on the top of the page next to your login name. Please, click on it and follow up the request procedure which allows to request global admin privileges, site admin privileges or federation admin privileges. Ask for sites admin privileges for your site. You will be shortly informed that your privileges are enabled. Please re-login.

Editing storage info

• Once you login with appropriate privileges, you should be able to edit information about your site. At the moment we are particularly interested in storage info at your site, namely its implementation, implementation version and SRR URL when it enabled.

• CRIC creates virtual storage service per site/per VO/per media/per implementation. By default it creates 1 disk and 1 tape virtual storage for every VO which is served by a given T1 site and 1 disk visrtual storage for T2 site. However, if for a given VO there are storage instances for the same media but different implementation (for example EOS and dCache instances for disk storage for ATLAS), CRIC should create two different disk virtual storage instances for this VO. Unfortunately, for the moment, there is no reliable primary source for this kind of information, so it is highly likely that only a single virtual storage will be created by CRIC in such cases. Would be great if you could correct it using CRIC UI and add other storage virtual instances with their implementation , implementation versions for your site and SRR URL when it is enabled. In the future we hope to get this information through SRR (Storage Resource Reporting).

• In the service table view, click on a particular service name
• You get a form with detailed information about service
• Click on the 'Edit' button under the first block of information
• You get another form. Please, correct 'Version' of your StoRM implementation. In the "Resource Reporting URL" field provide your SRR link
• Click on 'Check input data' and save info

Configuration for Third Party Copy

• Instructions for enabling Third Party Copy

WLCG sites which need upgrade to 1.11.18

Site	StoRM Version (27.05.2020)	Upgrade is planned (date)	SRR	Comments	GGUS ticket	Contacts
mainz	1.11.19			DONE	https://ggus.eu/?mode=ticket_info&ticket_id=147339	etapadmins@listsNOSPAMPLEASE.uni-mainz.de
egee.irb.hr	[u'1.11.14']			ticket has been updated to request upgrade to 1.11.18	https://ggus.eu/?mode=ticket_info&ticket_id=147328	lcg-admin@irbNOSPAMPLEASE.hr
WEIZMANN-LCG2	[u'1.11.19']	DONE, apparently storage of the site is not used any more by WLCG VOs	report.json	ticket has been updated to request upgrade to 1.11.18	https://ggus.eu/?mode=ticket_info&ticket_id=147341	lcg-managers-L@listservNOSPAMPLEASE.technion.ac.il
UKI-SOUTHGRID-SUSX	[None]			According to site admin site does not run StoRM any more. not clear what they are running. Storage is disabled by ATLAS	https://ggus.eu/?mode=ticket_info&ticket_id=147340	grid-admin@sussexNOSPAMPLEASE.ac.uk
INFN-GENOVA	[u'1.11.19']	DONE	report.json		https://ggus.eu/index.php?mode=ticket_info&ticket_id=148755	grid-prod@geNOSPAMPLEASE.infn.it
INFN-MILANO-ATLASC	[u'1.11.19']	DONE	srr.json	Inconsistent information on ATLASLOCALGROUPDISK, also observed at other sites	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148758	grid-prod@miNOSPAMPLEASE.infn.it
INFN-TRIESTE	[u'1.11.19']	Upgrade performed, SRR needs to be fixed. TPC is being debugged	report.json	Inconsistent information on ATLASLOCALGROUPDISK, also observed at other sites	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148763	grid-prod@tsNOSPAMPLEASE.infn.it
INFN-MIB	[u'1.11.19']			DONE. TPC to be checked with CMS	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148757	grid-prod@mibNOSPAMPLEASE.infn.it
INFN-CATANIA	[u'1.11.15']			DONE. Storage has been decomissioned. xRootd instance is running for ALICE	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148754	grid-prod@ctNOSPAMPLEASE.infn.it
INFN-PISA	[u'1.11.14']			No progress and answer so far	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148759	grid-prod@listsNOSPAMPLEASE.pi.infn.it
INFN-BARI	[u'1.11.21']		done		https://ggus.eu/index.php?mode=ticket_info&ticket_id=148753	grid-prod@listsNOSPAMPLEASE.ba.infn.it
IFCA-LCG2	[u'1.11.3']			Claimed to upgrade, though no SRR, not properly checked, CRIC is not updated	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148750	lcg.support@ifcaNOSPAMPLEASE.unican.es
IL-TAU-HEP	[u'1.11.17']			Upgrade is performed, SRR to be enabled	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148752	lcg-managers-L@listservNOSPAMPLEASE.technion.ac.il
BY-NCPHEP	[u'1.1.19']			DONE, but storage is disabled by CMS	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148749	gridadmin@hepNOSPAMPLEASE.by
INFN-TORINO	[u'1.11.9']			No progress so far	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148762	grid-prod@toNOSPAMPLEASE.infn.it
INFN-T1	[u'1.11.19', u'1.11.19', u'1.11.19', u'1.11.19', u'1.11.19']		DONE	Ticket TBD	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148761	t1-admin@listsNOSPAMPLEASE.cnaf.infn.it
NCG-INGRID-PT	[u'1.11.19', u'1.11.19']			DONE.	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148764, https://ggus.eu/index.php?mode=ticket_info&ticket_id=150728,	grid.admin@lipNOSPAMPLEASE.pt
UNI-SIEGEN-HEP	[u'1.11.15']	2020-06-09	storage-descriptor.json	File not updated since 12 June . Status is unclear, new ticket to be created	https://ggus.eu/?mode=ticket_info&ticket_id=147352	lcg-admin@listsNOSPAMPLEASE.hep.physik.uni-siegen.de
TECHNION-HEP	[u'1.11.17']		report.json	Upgrade is planned, not accomplished. File inaccessible with HTTP error 403: permission denied	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148765	lcg-managers-L@listservNOSPAMPLEASE.technion.ac.il
INFN-ROMA2	[1.11.19]			DONE, TPC status to be checked with Alessandra	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148781	grid-prod@roma2NOSPAMPLEASE.infn.it
INFN-ROMA3	[u'1.11.19']			upgrade DONE, TPC test is in progress	https://ggus.eu/?mode=ticket_info&ticket_id=147348; https://ggus.eu/index.php?mode=ticket_info&ticket_id=148760	grid-prod@roma3NOSPAMPLEASE.infn.it
UKI-LT2-QMUL	[u'1.11.19', u'1.11.19', u'1.11.19']			upgrade DONE, TPC tests are ongoing	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148766	edg-site-admin@qmulNOSPAMPLEASE.ac.uk; daniela.bauer.grid@googlemailNOSPAMPLEASE.com; d.colling@imperialNOSPAMPLEASE.ac.uk
IFIC-LCG2	[u'1.11.15']			Apparently upgrade is done, though CRIC is not updated. TPC tests are ongoing	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148751	lcg.support@ificNOSPAMPLEASE.uv.es
INFN-LECCE	[u'1.1.19']			DONE, TPC status to check with Alessandra	https://ggus.eu/index.php?mode=ticket_info&ticket_id=148756	grid-prod@leNOSPAMPLEASENOSPAMPLEASE.infn.it

Initial status when the task force started

STORM versions of the WLCG StoRM sites as of 27.05.2020

• According to CRIC there are 24 sites which are running StoRM and are used by the WLCG VOs. The number can be higher since for some of sites CRIC is lacking information about storage implementation and implementation versions the sites are running.
• 5 sites are currently running StoRM instances with version 1.11.15 and 2 are running 1.11.17
• There are 3 sites with unknown StoRM version in CRIC and BDII
• 13 sites need upgrade

Membership

• Enrico Vianello (StoRM)
• Julia Andreeva (WLCG Operations)
• Dimitrios Christidis (Storage Space Accounting expert)

Mailing lists

-- JuliaAndreeva - 2020-05-27