Status of last phases of gLite restructuring

The following text is extracted from the presentations that the JRA1 developers made at the JRA1 All Hands meeting on 27 October 2007 and following modifications.

The detailed status of the Dependency Challenge is available here.

The detaled status of client-server separation is available here.

Description of dependencies of metapackages

Security cluster

• Dependency Challenge (content extracted from components report)
  • Trustmanager/Java-Utils
    • Need to fix the “bcprov-jdk14” vs. “bouncycastle” issue.
      • Old version (1.22) of BC vs. new version (1.3x).
    • For Util-java dependency on test-utils and therefore on globus openssl.
      • Fix in the pipeline.
  • LCAS/LCMAPS
    • Circular dependency in ETICS in LCMAPS solved.
  • glexec
    • Subject to extensive scrutiny: Sites in LCG wary.
      • Code review from UK sys-admins. Suggested fixes made.
      • More valgrind testing.
      • More acceptance from the site people now.
  • VOMS/VOMS-Admin
    • VOMS roadmap to OpenSSL state clear:
      1. Server capable of GSI and SSL (1.8.0).
      2. Clients OpenSSL-only (1.9.0). In addition to GSI ones)
      3. Server OpenSSL-only (2.0.0).
    • Presented at MSWG @ EGEE07.
  • gJAF
    • Developed new test classes to support SAML2-XACML2 protocol.
      • Based on extension of OpenSAML2 and SunXACML OpenSource libraries.
• gLite restructuring
  • All security components have finished dependency challenge.
  • No major dependency problems found.
    • bcprov vs. bouncycastle issue identified (not only security components).
    • gsoap versions rationalized from 4 to 1 version (not only security).
  • Client/Server/Common separation.
    • Mainly applicable to VOMS/VOMS-Admin.
    • VOMS-Admin client/server well separated logically, no RT deps.
  • SLC4 readiness.
    • Trustmanager needs bouncycastle.
    • glexec more testing on CentOS/SL4.
    • LCAS/LCMAPS still underway.

CERN cluster

• gLite Restructuring
  • Rationalise gSoap dependencies
    • Done for all components
  • Remove unnecessary CGSI_gSOAP usage
    • Replace by openssl and GridSite
    • Done for all components it makes sense for
  • Migrate to libxml2
    • 75% of the work done on the development branches
  • General cleanup of build dependencies
    • Mostly done
  • 32 bit / 64 bit SLC4 now ~completed
    • LFC/DPM done and in production + FTS underway
  • Test DM code with VDT 1.6 / Tomcat 5.5 / Java 1.5
    • DPM/LFC done, lcg-util + FTS underway
  • Client / Server split
    • The runtime packages are cleanly separated
    • On the source side there is much overlap: doesn’t make much sense
• Common Logging Format
  • Following minimal model
    • Don’t disturb current production services. The services are critical to the ongoing WLCG production
    • There are lots of scripts / tools using the current logs
    • Close relationship anyway with SA1 / WLCG sys-admins: their comments / suggestions about the logs are usually fed into the next version quite quickly
  • We’ll carefully introduce the security related ones

Italian cluster

• DGAS
  • new dgas.servicecommon includes some parts of dgas.common used only on the server side
    • Less dependencies on the client modules
    • Easier portability of DGAS clients and sensors
  • Restructuring of the sensors
    • Decoupled the packaging of the sensors from the packaging of the generic clients and API
    • Ongoing activity to decouple UR production on the CE and their delivery to the HLR
• WMS/WMSUI
  • Removed deprecated features
    • Checkpointing, partitioning
  • Removed unneeded UI dependencies
    • wmscommon and wmsconfiguration
  • General dependency cleanup
• OLD UI
  • Commands were spread into different components/subsystems:
    • wms-ui.cli-python
      • Python commands to LB server
      • status & logginginfo
    • wms.client
      • C++ commands to WMProxy server
      • Submission, cancellation, getoutput, listmatch,...
  • Libraries were spread into different components:
    • wms-ui.cli-python
      • Python modules
    • wms-ui.wrap-python
      • Swig modules around C++ APIs
• NEW UI:
  • wms-ui.api-python
    • • Contains all python/swig modules/libraries
      • More userfriendly interface (API style)
  • wms-ui.commands
    • • Contains all python (LB) & C++ (WMProxy) commands
  • wms.wmproxy-api-* still under org.glite.wms
    • • Which relation should be between a metapackage and a subsystem?
• SLC4-ia32
  • A WMS almost equivalent to patch #1251 is under test
    • with the official globus gridftp using lcmapsbased authz
    • RPMs produced with ETICS
    • the installation and configuration are still partially manual
  • CREAM is under test
    • on top of BLAH
    • passed acceptance tests
    • moving towards certification (expected in December)
  • DGAS, VOMS and VOMSAdmin builds are available
  • Although now ETICS reports failures
    • Starting with proxy renewal
• SLC4_x86_64
  • Mainly a build exercise
  • CREAM, DGAS, BLAH, VOMS, VOMSAdmin builds are available
  • WMS and WMSUI not yet
    • not much effort yet
    • but some deps are not correct
      • libclassad.a built without fPIC
      • org.glite.security.gsoapplugin uses gsoap2.7.9b
      • Proxy renewal fails
      • ...
• Common Logging Format
  • Almost ready to start applying the recommendations
    1. Review all our logging statements
    2. Modify the code (in the main CVS trunk)
  • Proposed a system based on predefined madros (not included here)

Czech cluster

• Dependency Challenge
  • L&B, JP, gsoap-plugin, proxyrenewal
  • only cosmetic issues identified
    • Build vs. Runtime dependencies etc.
  • all addressed with recent 3.1 configurations
• gLite Restructuring
  • progress tracked at http://egee.cesnet.cz/mediawiki/index.php/LB_and_JP_cleanup
  • mostly internal cleanup
  • encapsulate direct Globus dependencies to security.gss – Done
    • VOMS – to be wraped in security.gss too – TODO
    • Gridsite – can provide standalone GACL library? – TODO
  • move event and jobstat definitions (.T’s) to lb.types – Done
    • standard interaction via stage, no more cross-module copy
    • org.glite.lb becomes container-only
  • jobid and exception – remove dependence on wms-utils
    • org.glite.jobid, mostly Done
  • L&B interface
    • move event and jobstat definition to lb.common – Done
    • required everywhere
    • no need for lb.client-interface anymore
  • extract client-only code from lb.common – In progress
    • includes splitting L&B context to client and server parts
  • lbjp-common subsystem
    • extract mini-libraries reused in L&B and JP – Done
    • trio, database layer, “maildir”, server-bones, common logging . . .
    • switch to use them consistently – In progress
    • generalized interlogger (aka IL-NT) – In progress, longer term
  • unify L&B proxy and server – Done, untested
    • one component providing both functionalities
    • can be deployed as proxy, server, or both at time
    • easier maintenance, better performance in the “both” deployment
  • database layer – TODO
    • store unparsed events only
      • fewer tables, better performance
    • drop unused fields, general cleanup
    • provide migration procedures
    • review transactional code
  • L&B–JP interaction – TODO
    • extract L&B plugin for JP to separate component
      • currently provided by lb.server – too heavy dependence
      • includes whole L&B state machine implementation
    • migrate affected components (lb.server, lb.utils)
  • common logging – TODO
    • only code review was done
  • merge 3.1 branch to HEAD – mostly Done
• SL4 and 64-bit readiness
  • skurut68-[12].cesnet.cz (preview testbed) run whole L&B and are SL4
  • development partially done on 64-bit SuSE (everything works there)
  • no SL4 64-bit tests yet, though

UK cluster

• Dependency challenge
  • Have done the “1 week” things
  • Other items addressed by head code (with refactoring)
• Logging
  • Mostly conforms
• 64 Bit
  • No problems

-- ClaudioGrandi - 07 Dec 2007