WLCG Operations Coordination Minutes, July 2, 2020

Highlights

Agenda

https://indico.cern.ch/event/933764/

Attendance

• local:
• remote:
• apologies:

Operations News

• After a few years of inactivity, we propose to close the Machine / Job Features TF

• The next meeting is planned for Sep 3

Special topics

WLCG Critical Services proposal followup

CERN Grid CA OCSP incident

• After a scheduled intervention on June 24, the CERN Grid CA OCSP service
  became inaccessible from outside CERN (OTG:0057432)
• Requests to the service were dropped by the CERN perimeter firewall
• A CREAM CE will try to check a client certificate's status via OCSP,
  if the existence of such an endpoint is indicated in the certificate details
  • It appears other CE flavors rely on CRLs only and just ignore OCSP services
• Checks of CERN Grid CA certificates were then hanging until a timeout was reached
• The CREAM client code would time out first, thus failing job submissions that
  used CERN Grid CA certificates
• This affected the 4 experiments and, through the SAM tests, sites running CREAM
  • Some A/R recomputations may be needed
• The service was restored about 24h later on June 25
• Some improvements are foreseen to make reoccurrence a lot less probable

• Further details here

SAM migration progress

Middleware News

• Useful Links
  • WLCGBaselineTable
  • MW Issues
  • Storage Deployment
• Baselines/News

Tier 0 News

Tier 1 Feedback

Tier 2 Feedback

Experiments Reports

ALICE

• Mostly business as usual, no major issues

ATLAS

CMS

• Covid-19 compute contributions being returned to experiment use
• main proccessing activities:
  • Run 2 ultra-legacy Monte Carlo
  • Run 2 pre-UL Monte Carlo
• migration to Rucio ongoing
  • production of nanoAOD samples configured for PhEDEx being bumped up to complete more quickl

LHCb

Task Forces and Working Groups

GDPR and WLCG services

• Updated list of services

Accounting TF

Archival Storage WG

Containers WG

CREAM migration TF

Details here

Summary:

• 90 tickets
• 14 done: 7 ARC, 7 HTCondor
• 16 sites plan for ARC, 15 are considering it
• 20 sites plan for HTCondor, 14 are considering it, 8 consider using SIMPLE
• 14 tickets on hold, to be continued in the coming weeks / months
• 7 tickets without reply
  • response times possibly affected by COVID-19 measures

dCache upgrade TF

DPM upgrade TF

StoRM upgrade TF

Information System Evolution TF

IPv6 Validation and Deployment TF

Detailed status here.

Machine/Job Features TF

Monitoring

MW Readiness WG

Network Throughput WG

• perfSONAR infrastructure status - 4.2.4 versions was released - please upgrade
  • Release notes: https://www.perfsonar.net/releasenotes-2020-04-01-4-2-4.html
• 100 Gbps perfSONAR mesh was established with 6 participating sites (TRIUMF, CERN, BNL, KIT, IC, AGLT2, Prague)
  • 100G testing/tuning resources added to WG Twiki (https://twiki.cern.ch/twiki/bin/view/LCG/NetworkTransferMetrics#perfSONAR_100G)
  • Testbed mailing list to join: http://cern.ch/simba3/SelfSubscription.aspx?groupName=wlcg-perfsonar-100g
• New LHCONE mesh established testing from sites to R&E perfSONAR endpoints (on LHCONE)
• OSG/WLCG infrastructure
  • Discussing plan of migration to the push model - direct publishing of results from toolkits to RabbitMQ
  • ESnet (router) traffic feed now available, working on its integration to our pipeline - prototype already working
• WLCG Network Throughput Support Unit: see twiki for summary of recent activities.

Traceability WG

Action list

Specific actions for experiments

Specific actions for sites

AOB