Notes for DAQ on-call training on 10-Feb-2020.

General

• trainees should confirm that they can log into the DAQ cluster as np04daq before the training
• as much as possible, training should take place in the control room and be a "hands on" introduction
• reminder that on-call experts are first-level support, and it's OK to contact others for help

Topics:

• elog - https://pdsp-elog.cern.ch/elisa/display
• Slack channel(s) - https://dunescience.slack.com/#np04-shifterassistant
• intro to documentation on the TWiki - https://twiki.cern.ch/twiki/bin/view/CENF/DUNEProtSPDAQ
• introductory diagram - RC, DAQInterface, BRs, EBs, RM/DFO, InhibitMaster, OnMon; storage servers, data disks, FTS-Light
• log into np04daq and run 'np04-status.sh'
• checking FELIX health - https://twiki.cern.ch/twiki/bin/view/CENF/FelixBasicOp#flxHealth_FELIX_Health_diagnosti
• checking Timing System health - https://twiki.cern.ch/twiki/bin/view/CENF/TimingSystemAdvancedOp#Essential_recovery_guide
• running Run Control (partitions, taking/releasing resources, monitoring plots, force shutdown, sending Timing Delays)
• individual transitions (e.g. config of WIB)
• log files - individual application windows in Run Control (also /log/ application type on each computer, where application type is e.g. boardreader or eventbuilder)
• intro to configurations, including editing: https://twiki.cern.ch/twiki/bin/view/CENF/TextConfigEditorP1
• checking the source of Inhibits - open the InhibitMaster window in Run Control
• "ps" command for finding rogue processes
• finding processes on computers (individual process page in RC, log area in main RC screen, ranks.txt file in run_records area)
• run_records area (/nfs/sw/artdaq/run_records/ run number )
• intro to software areas
• VNC session through lxplus
• slow controls web interface: http://np04-slow-control.web.cern.ch/np04-slow-control

-- KurtBiery - 2020-02-10