Week of 230109

WLCG Operations Call details

• The connection details for remote participation are provided on this agenda page.

General Information

• The purpose of the meeting is:
  • to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting;
  • to announce or schedule interventions at Tier-1 sites;
  • to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites;
  • to provide important news about the middleware;
  • to communicate any other information considered interesting for WLCG operations.
• The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30.
• The SCOD rota for the next few weeks is at ScodRota
• Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the wlcg-scod list (at cern.ch) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting.

Best practices for scheduled downtimes

• Best practices for scheduled downtimes

Monday

Attendance:

• local:
• remote: David (IN2P3), Xavier (KIT), David (FNAL), Petr (ATLAS), Maarten (ALICE, GGUS), Onno (NL-T1), Miro (Chair, DB), Christoph (CMS), Andrew (TRIUMF), Concezzio (LHCb), Doug (BNL), Darren (NDGF), Nils (Compute), Jose (PIC), Julia (WLCG)

Experiments round table:

• ATLAS reports ( raw view) -
  • ATLAS IAM VOMS expiring certificate INC3345128
    • Maarten asked why SNow ticket was created instead of GGUS? Petr agreed it should have been a GGUS one.
  • job monitoring issues during xmas RQF2202878
  • voms2 user interface issues during xmas INC3337085
    • lcg2-voms was working but users received link to voms2 in the email about AUP expiration

• CMS reports ( raw view) -
  • A good and successful year 2023 to everybody
  • One of the CERN VOMS servers (voms2.cern.ch) found with issues on Jan 2nd: INC:3340668
    • Similar question raised about SNow vs GGUS ticket
  • Otherwise processing went without major issues to good degree unattended during the break

• ALICE
  • Best wishes for 2023 !
  • Lowish to normal activity levels on average in the last 3 weeks.
  • No major issues.
    • GGUS:160008 was opened because the VOMS-Admin service on voms2.cern.ch was failing.

• LHCb reports ( raw view) -
  • all the best for 2023!
  • smooth running during the holiday season
  • PIC: 1024 bits proxies seems now rejected after storage upgrade (https://ggus.eu/index.php?mode=ticket_info&ticket_id=160025#update#7) Restriction should be lifted on the server side to allow for a smooth transition

Sites / Services round table:

• ASGC: NC
• BNL: NTR
• CNAF: NTR
• EGI: NC
• FNAL: NTR
  • Julia requested validation of accounting numbers
  • looking at dCache upgrade - no specific timeline yet
• IN2P3: NTR. Best wishes for 2023!
• JINR: NC
• KISTI: NC
• KIT: NTR
• NDGF: NTR
  • looking at dCache upgrade - no specific timeline yet
• NL-T1: NTR. Best wishes for 2023!
  • Julia requested validation of accounting numbers
• NRC-KI: NC
• OSG: NC
• PIC:
  • Looking at the problem with 1024bits proxies reported by LHCb
• RAL: NTR
• TRIUMF: NTR

• CERN computing services: Temporary drop in batch capacity tomorrow due to power tests at point 8. (Ref. OTG:0074835 )
• CERN storage services: NC
• CERN databases:
  • OTG:0074877 Oracle databases ATONR and ATLR software update (patching) on 10th Jan.
  • OTG:0074773 CMSONR database - Application of latest patches
  • OTG:0074775 CMSR database - Application of latest patches
  • OTG:0074774 CMSONR ADG and CMSARC databases - Application of latest patches
• GGUS: NTR
• Monitoring:
  • Distributed final SiteMon availability/reliability reports for November 2022
  • Doug asked about about the flow of accounting numbers to debug discrepancies between local and WLCG accounting. Julia will send email to clarify
• Middleware: NTR
• Networks: NTR
• Security: NC

AOB:

• Wishing you a joyful a successful 2023!