TWiki
>
LCG Web
>
WLCGCommonComputingReadinessChallenges
>
WLCGOperationsMeetings
>
WLCGOpsMeetingWeek240422
(2024-04-22,
MaartenLitmaath
)
(raw view)
E
dit
A
ttach
P
DF
---+!! Week of 240422 %TOC% ---++ WLCG Operations Call details * The connection details for remote participation are provided on this [[https://indico.cern.ch/event/1188295/][agenda page]]. ---++ General Information * The purpose of the meeting is: * to report significant operational issues (i.e. issues which can or did degrade experiment or site operations) which are ongoing or were resolved after the previous meeting; * to announce or schedule interventions at Tier-1 sites; * to inform about recent or upcoming changes in the experiment activities or systems having a visible impact on sites; * to provide important news about the middleware; * to communicate any other information considered interesting for WLCG operations. * The meeting should run from 15:00 Geneva time until 15:20, exceptionally to 15:30. * The SCOD rota for the next few weeks is at ScodRota * Whenever a particular topic needs to be discussed at the operations meeting requiring information from sites or experiments, it is highly recommended to announce it by email to the =wlcg-scod= list (at =cern.ch=) to allow the SCOD to make sure that the relevant parties have the time to collect the required information, or invite the right people at the meeting. ---++ Best practices for scheduled downtimes * [[BestPracticesForSchedDT][Best practices for scheduled downtimes]] ---++ Monday Attendance: * local: Julia (WLCG), Maarten (WLCG, ALICE, GGUS) * remote: Darren (RAL), Christian (NDGF), Daniele (CNAF), Dave (FNAL), Farrukh (FNAL), Ivan (BNL), Nils (Computing), Panos (Chair, CMS), Peter (ATLAS), Stephan (CMS), Xavier (KIT) Experiments round table: * ATLAS [[AtlasComputing.ADCOperationsWeeklySummaries2024][reports]] ([[AtlasComputing.ADCOperationsWeeklySummaries2024?raw=on][raw view]]) - * NTR * CMS [[CMS.FacOps_WLCGdailyreports][reports]] ([[CMS.FacOps_WLCGdailyreports?raw=on][raw view]]) - * CMS encountered a problem with an ORACLE database (CMS_TRG_R) Saturday morning and needed on-call support from IT, which was not available. We were lucky that Kate was able to fix the problem in hero mode during the LHC interfill period, so CMS did not lose data, but we easily could have if the incident had lasted much longer. * Maarten: Online DBs are actually the only IT services that should offer an on-call piquet. * Stephan/Panos: CMS coordinators are aware of this and will follow up on the necessary communication channels. * significant drop in core usage early last week that came from opportunistic resources * several monitoring outages/interrupts the last week * we started decommissioning of GSIftp/gridFTP protocols with sites * ALICE * after a Grafana upgrade last week, the !SiteMon (SAM) [[https://monit-grafana.cern.ch/d/000000619/wlcg-sitemon-historical-profiles?orgId=20][GUI]] suffered usability regressions * both ALICE and CMS opened tickets about that * in particular, the logs now show a =\r= sequence at the end of each line * the MONIT team are looking into how that can be fixed * LHCb [[LHCb.ProductionOperationsWLCGdailyReports][reports]] ([[LHCb.ProductionOperationsWLCGdailyReports?raw=on][raw view]]) - Sites / Services round table: * BNL: NTR * CNAF: Upgrading HTCondor cluster to HTCondor 23, ~1/2 of cluster is already upgraded with 4 CEs as well * EGI: * FNAL: * !IN2P3: NTR * JINR: * KISTI: * KIT: * HPSS tape backend is in downtime since today 9 a.m. CEST (GOCDB:35274). * Alice was suffering last week from few badly performing data servers in their xrootd cluster. Those were removed from service for further investigations. * Streamlining service endpoint names for all dCache SEs: ATLAS (GGUS:166267), LHCb (GGUS:166268), Belle (GGUS:166269). The changes for CMS were applied by our in-house CMS contacts to the GridKa specific =storage.xml= already. * NCBJ: * NDGF: NTR * NL-T1: * NRC-KI: * OSG: * PIC: * !RAL: NTR * TRIUMF: NTR * CERN computing services: NTR * CERN storage services: * CERN databases: * GGUS: * A new release is planned for Wed this week * [[https://ggus.eu/?mode=release_notes][Release notes]] * A [[https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=34848][downtime]] has been scheduled from 06:00 to 08:00 UTC * Test alarms will be submitted as needed * Monitoring: * Middleware: NTR * Networks: * Security: AOB:
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r16
<
r15
<
r14
<
r13
<
r12
|
B
acklinks
|
V
iew topic
|
WYSIWYG
|
M
ore topic actions
Topic revision: r16 - 2024-04-22
-
MaartenLitmaath
Log In
LCG
LCG Wiki Home
LCG Web Home
Changes
Index
Search
LCG Wikis
LCG Service
Coordination
LCG Grid
Deployment
LCG
Apps Area
Public webs
Public webs
ABATBEA
ACPP
ADCgroup
AEGIS
AfricaMap
AgileInfrastructure
ALICE
AliceEbyE
AliceSPD
AliceSSD
AliceTOF
AliFemto
ALPHA
Altair
ArdaGrid
ASACUSA
AthenaFCalTBAna
Atlas
AtlasLBNL
AXIALPET
CAE
CALICE
CDS
CENF
CERNSearch
CLIC
Cloud
CloudServices
CMS
Controls
CTA
CvmFS
DB
DefaultWeb
DESgroup
DPHEP
DM-LHC
DSSGroup
EGEE
EgeePtf
ELFms
EMI
ETICS
FIOgroup
FlukaTeam
Frontier
Gaudi
GeneratorServices
GuidesInfo
HardwareLabs
HCC
HEPIX
ILCBDSColl
ILCTPC
IMWG
Inspire
IPv6
IT
ItCommTeam
ITCoord
ITdeptTechForum
ITDRP
ITGT
ITSDC
LAr
LCG
LCGAAWorkbook
Leade
LHCAccess
LHCAtHome
LHCb
LHCgas
LHCONE
LHCOPN
LinuxSupport
Main
Medipix
Messaging
MPGD
NA49
NA61
NA62
NTOF
Openlab
PDBService
Persistency
PESgroup
Plugins
PSAccess
PSBUpgrade
R2Eproject
RCTF
RD42
RFCond12
RFLowLevel
ROXIE
Sandbox
SocialActivities
SPI
SRMDev
SSM
Student
SuperComputing
Support
SwfCatalogue
TMVA
TOTEM
TWiki
UNOSAT
Virtualization
VOBox
WITCH
XTCA
Welcome Guest
Login
or
Register
Cern Search
TWiki Search
Google Search
LCG
All webs
Copyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use
Discourse
or
Send feedback