EMI Data Client Consolidation F2F Meeting, June. 29, 2011, 09:30-16:30

Participants: Zsolt, Michail (only the first our), Oliver (only the last hour), Jon, David, Zsombor

Where: 28 R-014 @ CERN

Minutes

Goal: we need to have an outline of the plan

Agenda

• Work distribution - who does what?
• Status of clients and libraries
• Status of consolidation ideas
• Create outline of plan
• Table of components clean-up <https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1DataClientEvaluation>
• Create table of affected external components

Word distribution

• Jon will have to lead JRA1 75% so he will have problems working on this, but he will still leading it
• Zsolt says they also don't have much time, maybe a student could be assigned
• David is not working for EMI, but for NDGF
• Michail is not formally part of the group, he is just observing it, is a backup of Zsolt
• Zsombor should be able to work on this
• It depends what we decide related to the plan.

Status of clients and libraries

• Jon tried to create a GFAL plugin, but there was a problem: gfal open command when using SRM will hang, probably because of threadlocks (this is a known bug in GFAL)
• Jon created a big figure trying to map ARC commands to GFAL commands, you cannot map from GFAL to ARC because e.g. ARC doesn't have seek
• Zsolt says that if the RFIA plugin and GFAL links to threaded/non-threaded globus mixed, it could cause problems, it could be solved with reorganizing, and also there is a GFAL2, which a complete redesign of the API, maybe that already fixed this problem
• ARC has an xroot plugin now but it's not in the release yet, nobody tried it
• We should collect a table of test endpoints for different protocols (testbed?) Zsolt has some script which gets SRM endpoints from a BDII (e.g. dteam VO)
• ARC people should join the dteam VO

Status of consolidation ideas

• consolidate the file catalog API: remove lcg utils catalog tools, needs testing
• GFAL plugin for ARC as a proof of concept (already started)
• using ARC libraries in GFAL? C/C++ problem! One candidate could be to use the xroot plugin of ARC - does it have a point? GFAL (POSIX) -> ARC (non-POSIX) -> xroot (POSIX)
• the way to create a directory in ARC is to create a file inside
• ARC has an SRM plugin which doesn't use external libraries
• GFAL cannot do POSIX on LFC, ARC has an LFC plugin which looks up the replicas
• if ARC would use GFAL, these can be thrown out: file, gridftp; not really: lfc (not in GFAL), srm (not in the POSIX-layer of GFAL, but somewhere else)
• lcg util is also an API (C and python), so the SRM stuff should go into this and not in GFAL (?)
• srm part uses gsoap and cgsi-gsoap plugin which is a security component on top of gsoap - could cause problems

(Michail leaves)

Picture 1 of whiteboard

• Picture 1 of whiteboard:
  

• GFAL supports normal FTP as well
• ARC can drop file and gridftp
• ARC has to keep HTTP
• ARC LFC is just a small wrapper around the lfc libraries, no point to drop
• ARC SRM is native (no dependencies), uses HTTPS
• GFAL SRM uses gsoap, so it could be configured to use HTTPS
• GFAL SRM can use space tokens
• if ARC uses GFAL, can it use the SRM parts if SRM is not in the POSIX SRM, but outside of it? where?
• if the SRM would return a HTTP TURL, what would GFAL do with it?
• gsoap will stay until the cgsi stuff is needed, ARC doesn't want gsoap
• what about other platforms? ARC SRM runs on mac, windows, can GFAL do that?
• GFAL SRM could be ported to windows - cgsi? voms libraries?
• the SRM in GFAL is internal not a plugin, it wouldn't be easy to refactor that
• and GFAL SRM still needs the cgsi stuff
• packaging: will we have a gfal package, an libarcdata plugin and an emi-data plugin? which depeneds on which? how wouldn't it be a big mess?

Picture 2 of whiteboard

• Picture 2 of whiteboard:
  

• GFAL has a POSIX interface with plugins (gridftp, file, rfio, dcap)
• GFAL should have http and xrootd in the future
• GFAL has SRM which is not a plugin because it's a different interface
• GFAL has LFC which is a plugin
• lcg util library uses the gfal library for doing higher level operations
• the lcg util cli turns the command line parameters and calls the apropriate lcg util library function
• FTS3 also uses lcg util to move data
• ARC has libarcdata which has the DMC plugins, and ARC CLI uses the libarcdata to move files, and also ARC server side uses libarcdata to stage files
• ARC create a GFAL/POSIX DMC, which maps the ARC DMC interface to GFAL POSIX calls, so ARC can use all the protocols which GFAL provides
• and also ARC would keep http and xrootd along with GFAL DMC
• GFAL currently needs gsoap (development time dependency)
• currently the GFAL plugins need globus library (runtime dependency)
• GFAL should have some support on windows and mac os with at least the file plugin and with the new plugins (http!)
• why not the other way around? GFAL having an ARC plugin? C/C++ and POSIX/non-POSIX

Problems

• from ARC to GFAL: platforms (windows, mac os, all kind of linuxes)
• from GFAL to ARC: C/C++ and POSIX/non-POSIX

Picture 3 of whiteboard

• Picture 3 of whiteboard:
  

• if ARC would be a GFAL plugin, then GFAL can get rid of file and gridftp and lfc and srm but then every POSIX call should be somehow turned into libarcdata methods and then those convert it back to posix calls, which seems not very nice
• what if lcg util (and FTS) would use libarcdata, so we will have a posix library and a "file" library

Picture 4 of whiteboard

• Picture 4 of whiteboard:
  

• FTS and LCG util will use libarcdata, libarcdata use gfal for file transfer
• components can be removed, need to change, new component, benefits
• see picture

(Oliver joins, Zsolt explained the situation to him in the break)

• if the plan should be ready in October, the implementation in December, and then 10 month testing - so EMI2 will contain this big change untested?
• in EMI2 we have to make the new library available - what does it mean? - does this mean that there will be paralel the old and the new stuff (with the same names)
• GFAL has some internal dependencies
• GFAL depends on VOMS - why? delegation?
• GFAL has information system stuff, e.g. incomplete SRM (endpoint coming from BDII), copy and register (find a catalog) (it needs libldap depency) - so with this ARC would gain infosys queries
• the plan should make some suggestions how should we phase out the old library, e.g. new functionality only goes to the new library, please use that
• question about renaming components: we shouldn't care about it right now

Action items

• Zsombor will write an explanation about the plan and create a figure, and also try to collect questions (deadline: end of this week)

-- JonKerrNilsen - 28-Jun-2011