DPM upgrade task force (Second circle)
- DPM upgrade task force (Second circle)
- Introduction
- Mandate of the task force
- Upgrade instructions
- Test that storage works after upgrade
- Recommended configuration
- Detected problems
- Useful links
- Participants
- First Upgrade Circle
Introduction
WLCG Operations is starting the second upgrade circle of the DPM sites. The goal of this circle is to upgrade DPM instances to the version which is supporting Third Party Copy (TPC) and in which most of the issues detected during TPC tests have been fixed. Version to which DPM instances should be upgraded is 1.14. Apart of upgrade to DPM 1.14, those sites which did note yet enabled DOME have to enable DOME as well as SRR generation and macaroons.Mandate of the task force
- Coordinate the upgrade of the DPM sites to DPM version 1.14 or higher and reconfiguration required to enable DOME, SRR and macaroons.
- Provide guidance and support sites for upgrade and reconfiguration
- Validate SRRs published by DPM sites and make sure that they can be integrated with CRIC and the WLCG Storage Space Accounting system
Sites requiring upgrade to 1.14 and reconfiguration for DOME
Sites requiring upgrade to 1.14.*
Upgrade instructions
- Instructions for the upgrade from the legacy DPM to DPM with DOME documentation. It is important for the sites which did not yet run DPM versions with DOME and did not re-configure for DOME.
- For the sites which are already running DPM versions with DOME but need to upgrade to DPM 1.14.0
- run
yum upgradeon all DPM machines (headnode + disknodes) - enable macaroons
- with puppet
- manual headnode and disknode configuration
- Permit HTTPS access to DPM headnode + disknodes (don't block port 443 on your firewall LCGDM-2910
)
- enable ARGUS banning
- apply your puppet configuration on headnode + disknodes (sites that don't rely on puppet should consider updates described in LCGDM-2903)
- only puppet 5 and 6 supported by DPM 1.14 puppet modules
- if your site did not yet enabled GridFTP redirection than be aware it is now enabled by default with puppet
- sites can ignore OIDC section from the DPM 1.14.0 release notes (only for testbeds participating in WLCG DOMA TPC / OIDC activities)
- run
- DPM 1.14.0 from EPEL is the last version that works with XRootD 4.x, DPM 1.14.1 requires XRootD 5.x
Test that storage works after upgrade
- wget https://raw.githubusercontent.com/paulmillar/http-tpc-utils/master/bin/smoke-test.sh
- chmod 755 ./smoke-test.sh
- voms-proxy-init -voms your_vo
- ./smoke-test.sh -v your_vo https://<storage_head>:<port>/<dpm_path>/your_vo
Recommended configuration
- ATLAS (Rucio)
- Use at least DOME DPM 1.14 + XRootD 4.12.4 + davix 0.7.6 ... latest (stable) versions from EPEL recommended
- dmlite is linked with xrootd packages available at release date and by moving to the latest dmlite it is necessary to use the most recent xrootd packages
- August 2020: xrootd R4 is supported. Xrootd R5 is not supported.
- enable GridFTP redirection: puppet head+disknode configuration option
gridftp_redirect(enabled by default since DPM 1.14) - enable XRootD checksums: puppet head+disknode configuration option
configure_dpm_xrootd_checksum(enabled by default since DPM 1.13) - optionally enable TPC XRootD delegation: puppet disknode configuration option
configure_dpm_xrootd_delegation(enabled by default since DPM 1.13) - to support IPv4 only clients with enabled IPV6 in GridFTP plugin (was default in gfal-2.17.0, but reverted back to IPv4 in following releases) on dualstack DPM
epsv_matchmust be enabled, see LCGDM-2817
- AGIS configuration (example for SE, panda)
- GridFTP is still preferred protocol with priority 0 for tpc activities (requires GridFTP redirection)
- during Autumn/Winter 2020 we'll gradually move ATLAS sites to HTTP-TPC ( ADCINFR-166)
- during Autumn/Winter 2020 we'll gradually move ATLAS sites to HTTP-TPC ( ADCINFR-166
- XRootD for lan and wan read+write (write works only with XRootD checksums enabled)
- rucio mover for panda queues (rucio mover use storage protocols according preferences defined in AGIS)
- each protocol in AGIS should have monitoring enabled to be part of ATLAS SAM tests / ETF check_mk
- EGI sites should also register each SE protocol with GOCDB (example: SRM, GridFTP, XRootD, WebDAV)
- GOCDB storage protocols are used to automatically postpone ATLAS transfers when site storage is in downtime
- fully SRM-less operation requires additional configuration of the Storage Resource Reporting (SRR)
- since DOME DPM 1.13.2 SRR info automatically available via HTTP CGI and cron config is no longer necessary
- modify "Space method" and "Space Usage" for each DDM endpoint in AGIS ( example)
- Space method: storage
- Space Usage: URL of your storagesummary.json
- since DOME DPM 1.13.2 SRR info automatically available
- GridFTP is still preferred protocol with priority 0 for tpc activities (requires GridFTP redirection)
- Configure Argus banning available since DOME DPM 1.14.0
- Use at least DOME DPM 1.14 + XRootD 4.12.4 + davix 0.7.6 ... latest (stable) versions from EPEL recommended
- CMS (PhEDEx)
- Dirac users
- internally use GFAL for transfers (unless you still use deprecated protocols)
- if your DPM supports IPv4 + IPv6 be avare IPv4 only clients can't access data using gsiftp protocol unless you follow instruction in LCGDM-2817
- LFC catalog
- deprecated & EOL - you should think about migration
- full file URL stored in catalog - can't easily switch from SRM protocol
- DFC catalog
- possible to configure non-SRM transfer protocols
- with GridFTP redirection enabled in DPM it should be almost transparent switching from GFAL2_SRM2 to GFAL2_GSIFTP
- internally use GFAL for transfers (unless you still use deprecated protocols
After reconfiguration for DOME make sure that SRR is enabled
How to enable SRRAfter changes performed on your service, please, update information in CRIC
Authentication & authorization step
- Go to WLCG cric server , click "Site topology" (menu on the top of the page) -> Services. Enable filtering, by clicking on the 'Filter' button and select your site. By default , you won't see implementation and implementation version columns in the table. In order to see this info, you need to click on 'Columns' and then select corresponding columns in the drop down list.
- You should be able to list all CRIC entities (sites (GocDB /OIM and experiment-specific ones), federations, pledges, services, storage protocols and queues) without authentication. However, once you would like to see details of any particular entity, you would be asked to login.
- Those who are registered in the CERN DB, please, use SSO authentication. Authentication with certificate is not yet enabled on this instance, will come soon.
- Those who are not registered in the CERN DB would need to ask for CRIC local account. Please, send a mail to cric-devs@cernNOSPAMPLEASE.ch with your name, family name and mail address to be used by CRIC to communicate with you.
- As soon as you are logged in, you will be able to see details of any CRIC entity, however in order to edit in order to edit information, one would need to get specific privileges. * As soon as you are authenticated, you will see 'Request privileges' on the top of the page next to your login name. Please, click on it and follow up the request procedure which allows to request global admin privileges, site admin privileges or federation admin privileges. Ask for sites admin privileges for your site. You will be shortly informed that your privileges are enabled. Please re-login.
Editing storage info
- Once you login with appropriate privileges, you should be able to edit information about your site. At the moment we are particularly interested in storage info at your site, namely its implementation, implementation version and SRR URL when it enabled.
- CRIC creates virtual storage service per site/per VO/per media/per implementation. By default it creates 1 disk and 1 tape virtual storage for every VO which is served by a given T1 site. However, if for a given VO there are storage instances for the same media but different implementation (for example EOS and dCache instances for disk storage for ATLAS), CRIC should create two different disk virtual storage instances for this VO. Unfortunately, for the moment, there is no reliable primary source for this kind of information, so it is highly likely that only a single virtual storage will be created by CRIC in such cases. Would be great if you could correct it using CRIC UI and add other storage virtual instances with their implementation , implementation versions for your site and SRR URL when it is enabled. In the future we hope to get this information through SRR (Storage Resource Reporting).
- In the service table view, click on a particular service name
- You get a form with detailed information about service
- Click on the 'Edit' button under the first block of information
- You get another form. Please, correct 'Version' of your DPM implementation. In case Dome is enabled, please provide version number complemented 'with DOME' and provide "Resource Reporting URL" value
- Click on 'Check input data' and save info
Creating a new virtual storage instance in CRIC
- Staring from the entry page: https://wlcg-cric.cern.ch/
- in the block "Site & Services" click 'Create Storage Service'. You get a form to fill in
- Keep service name field empty as the form suggests
- Select your site form the drop down menu
- Service type (SE) should not be touched
- Select Disk or Tape media in the "Architecture" filed drop down menu
- Provide value for implementation (EOS, Castor, Xrootd, dCache, DPM)
- Provide value for implementation version
- You can provide a value in the endpoint field or leave it empty if it does not make sense
- Please, select value for the VO name. As mentioned above, the virtual storage in CRIC is created for a single VO even though several VOs can share the same physical storage service of the site
- Leave 'ACTIVE' object state
- All other attributes are optional, you can leave them empty
Creating a new protocol for a given virtual storage in CRIC
- Currently, even if there is one single protocol shared by several virtual storage instance in CRIC, for each
- To create new protocol for a given virtual storage instance, select corresponding service in the service list and click on the name to get a detailed description of the virtual storage service
- Below the table with the list of protocols, click on the 'Add protocol' button. You will get a form to fill.
- Leave the name of the protocol empty, the system will generate it for you
- "Flavour" and "endpoint" are mandatory attributes, other fields could be empty
Deleting virtual storage instance from CRIC
For the time being , deletion from the UI is not allowed. Change the object state to "Disabled" in order to make it disappear from the listingDeleting protocols from CRIC
The protocol attached to a particular virtual storage can be deleted from the protocol list from the detailed page describing the virtual storage service.DPM service monitoring for EGI.
In order to enable DPM service monitoring for EGI one needs to configure webdav (HTTPS) for the ops VO and register the endpoint on GOCDBRegistration of webdav service endpoint in GocDB
For registering on GOC-DB the webdav service endpoint, follow the HOWTO21 in order to fill in the proper information.
In particular: - register a new service endpoint, separated from the SRM one and with the "production" flag disabled;
- on GOC-DB fill in the webdav URL containing also the VO ops folder. Example
- it corresponds to the value of GLUE2 attribute GLUE2EndpointURL (containing the used port and without the VO folder);
- verify that the webdav url is properly accessible.
- check if the tests are ok (https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_webdav&style=detail )and then switch the production flag to "yes"
Enable gridftp monitoring for ops VO (if you provide such protocol)
- register a new service endpoint, associating the storage element hostname to the service type “globus-GRIDFTP”, with the "production" flag disabled;
- in the “Extension Properties” section of the service endpoint page, fill in the following fields:
- Name: SE_PATH
- Value: /dpm/ui.savba.sk/home/ops #this is an example, set the proper path
- check if the tests are ok (it might take some hours for detecting the new service endpoint) and then switch the production flag to "yes"
Detected problems
Useful links
- SRR presentation at the DOMA meeting
- SRR example for Prague
- Presentation at the WLCG Operations Coordination meeting
- DPM twiki
- DPM 1.10.0 Upgrade Guide
Participants
- Fabrizio Furano (DPM)
- Oliver Keeble (DPM and WLCG Steering Group)
- Dimitrios Christidis (WLCg Storage Space Accounting)
- Julia Andreeva (WLCG Operations Coordination)
First Upgrade Circle
Topic revision: r214 - 2021-09-02 - JuliaAndreeva
LCG Wikis
- ABATBEA
- ACPP
- ADCgroup
- AEGIS
- AfricaMap
- AgileInfrastructure
- ALICE
- AliceEbyE
- AliceSPD
- AliceSSD
- AliceTOF
- AliFemto
- ALPHA
- Altair
- ArdaGrid
- ASACUSA
- AthenaFCalTBAna
- Atlas
- AtlasLBNL
- AXIALPET
- CAE
- CALICE
- CDS
- CENF
- CERNSearch
- CLIC
- Cloud
- CloudServices
- CMS
- Controls
- CTA
- CvmFS
- DB
- DefaultWeb
- DESgroup
- DPHEP
- DM-LHC
- DSSGroup
- EGEE
- EgeePtf
- ELFms
- EMI
- ETICS
- FIOgroup
- FlukaTeam
- Frontier
- Gaudi
- GeneratorServices
- GuidesInfo
- HardwareLabs
- HCC
- HEPIX
- ILCBDSColl
- ILCTPC
- IMWG
- Inspire
- IPv6
- IT
- ItCommTeam
- ITCoord
- ITdeptTechForum
- ITDRP
- ITGT
- ITSDC
- LAr
- LCG
- LCGAAWorkbook
- Leade
- LHCAccess
- LHCAtHome
- LHCb
- LHCgas
- LHCONE
- LHCOPN
- LinuxSupport
- Main
- Medipix
- Messaging
- MPGD
- NA49
- NA61
- NA62
- NTOF
- Openlab
- PDBService
- Persistency
- PESgroup
- Plugins
- PSAccess
- PSBUpgrade
- R2Eproject
- RCTF
- RD42
- RFCond12
- RFLowLevel
- ROXIE
- Sandbox
- SocialActivities
- SPI
- SRMDev
- SSM
- Student
- SuperComputing
- Support
- SwfCatalogue
- TMVA
- TOTEM
- TWiki
- UNOSAT
- Virtualization
- VOBox
- WITCH
- XTCA
 |
|
Copyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors. or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback