Versions

Current version of these instructions is at NanorcRunControl Previous versions:

• For running releases up to fddaq-v4.2.1, as was done prior to Feb 2024 here
• For running a 2.11.1-based release, as was done in coldbox running prior to Fall 2022 here

Run Control

Getting started

Log into np04-srv-024 using the np04daq(!) account.

Please read the 'welcome' message that is printed (if you don't see one, check that you're on the right server and right account). It will give you some instructions like this:

 ---- Welcome to NP04 Run Control Server! ---- 



See https://twiki.cern.ch/twiki/bin/view/CENF/NanorcRunControl for further instructions.


Currently running tmux sessions are:

np02_timing_fanout_2: 1 windows (created Thu Aug 24 10:29:41 2023) [120x29]
np02_vd: 1 windows (created Thu Aug 24 17:37:26 2023) [157x43]
np04_hd: 1 windows (created Thu Aug 24 16:04:53 2023) [147x35]
np04_timing_fanout_0: 1 windows (created Thu Aug 24 10:14:00 2023) [120x29]
np04_timing_fanout_1: 1 windows (created Thu Aug 24 10:03:48 2023) [120x29]

To attach to a session, do:  'tmux a -t <session_name>'


If you are taking a run, please attach to the appropriate session and run from that environment.

If you need to do other other DAQ work,
The DAQ NP04_HD stable area is DAQ_NP04_HD_AREA (/nfs/sw/dunedaq/dunedaq-fddaq-v4.1.1)
The DAQ NP04_HD stable area is DAQ_NP02_VD_AREA (/nfs/sw/dunedaq/dunedaq-fddaq-v4.1.1)
The DAQ NP04_HD dev/integration area is DAQ_NP04_HD_DEV_AREA (/nfs/sw/dunedaq/dunedaq-fddaq-v4.1.0-dev)

Follow the instructions to attach to the relevant tmux session for running. Common tmux sessions and their use are:

Name	Use for ...	Area in use
np04_hd	Stable release running on NP04 (ProtoDUNE-II HD Module 0)	DAQ_NP04_HD_AREA
np02_vd	Stable release running on NP02 (ProtoDUNE-II VD Module 0)	DAQ_NP02_VD_AREA

If you don't see the right one running, it may need to be restarted. Do

tmux new -s np0X_coldbox

(replace with the appropriate session name). After restarting it, you will need to setup the DAQ environment for running:

NOTA BENE: The tmux prefix key for the np04daq account is set to Ctrl-\ (that's control and back slash)

cd ~np04daq/DAQ_NP04_HD_AREA
source setup_for_run.sh

(or appropriate area)

You're ready to start!

Using Run Control

• Choose a global configuration corresponding to the required setup. Configurations are located in:

 ls ~np04daq/DAQ_NP04_HD_AREA/global_configs 

(Note: global_configs should be a soft link to np04daq-configs/global_configs, in that same area.)

• Start run control with

nano04rc --partition-number <partition_number> --timeout 120 global_configs/<config_name>.json <username> <partition_name>

where config_name is one of the configs in `global_configs` that is printed out when you setup for the run. Typical configurations to use are:

Name	Details	Components used
np04_tp.json	NP04, all readout (TPC+DAPHNE), with TPC TPs
np04	NP04, all readout (TPC+DAPHNE)
np04_WIB	NP04, TPC readout on all APAs	NP04_APA1-4
np02_CRPs45_WIB	NP02, TPC readout for CRPs 4 and 5 (bottom detector electronics)	NP02_CRP4, NP02_CRP5
np04_WIB_pulser	NP04, TPC readout on all APAs, WIB pulsers on	NP04_APA1-4
np02_CRPs45_WIB_pulser	NP02, TPC readout for CRPs 4 and 5 (bottom detector electronics), WIB pulsers on	NP02_CRP4, NP02_CRP5

To see the commands available use help

username@np04rc> help

Documented commands (type help <topic>):
========================================
boot              enable_triggers  ls           start_run             terminate
change_rate       exclude          message      start_shell           wait     
change_user       expert_command   pin_threads  status              
conf              include          scrap        stop                
disable_triggers  kinit            shutdown     stop_run            
drain_dataflow    klist            start        stop_trigger_sources

Undocumented commands:
======================
exit  help  quit

• boot will start your applications.
• You can then issue start_run {TEST|PROD} : use TEST if the data is not going to be used or PROD if the data is planned to be suitable for further analysis. Usually you want to run start_run PROD.
• start_run can also take an optional argument for a message to post to the run control, change the default trigger rate, or more. Do help start_run to see details.

Commands produce quite verbose output so that you can see what was sent directly to the applications without digging in the logfiles.

Use status to see what's going on.

• When you're done use stop_run, scrap and exit commands.

Using tmux

To be able to share the same terminal session and pass running instances of nanorc between people, we use tmux. The commands that we use are:

• tmux new -s session_name to create a new named session, this name could be for example np02_coldbox to run the nanorc session for the np02 coldbox inside
• To check which sessions are running, use tmux ls
• To attach to an existing tmux session, use tmux attach -t session_name
• To detach from the session, use tmux detach

Make sure that you are on the same machine that the session was created on and use the correct user (np04daq) to attach to it!

Align timing Scripts for ProtoDUNE - I --

How to align the timing depends on the system that is used, for all the commands you have to be logged in to np04-srv-012 and setup the work area (cd DAQ_COLDBOX_AREA; source setup.sh)

• For the SSPs on top of np02: pdtbutler mst PROD_FANOUT_2 align apply-delay 0 0 0 --force
• For the np02 coldbox: pdtbutler mst PROD_MASTER_VDC align apply-delay 0 0 0 --force
• For the np04 coldbox: pdtbutler mst TERTIARY align apply-delay 0 0 0 --force

Sometimes these commands have to be issued multiple times until the system is correctly aligned

FELIX alignment

For configurations using the FELIX (for example the coldboxes) a felixcardcontroller daq_application, configured and aligns the links. You can monitor their status via the grafana (frontend dashboard):

• For the np02 coldbox, the FELIX is on server np04-srv-030
• For the np04 coldbox, the FELIX is on server np04-srv-028
• For the VST, the FELIX is on server np04-srv-026

You can use the following commands on the hosts if you suspect that something is not right (setup the workarea before to load them, cd DAQ_COLDBOX_AREA; source setup.sh)

• flx-info GBT This shows the alignment for every link, for the np02 you should see 8 YES and for the np04 coldbox 10 YES if the links are correctly aligned
• flx-reset GTH Resets the links, check that they are aligned afterwards. If not, there might be a bigger underlying problem (i.e. the WIBs are off/disconnected)

These commands can be executed during a run! They won't affect data taking conditions.

Switching on and off FEMB, and checking their status Scripts for ProtoDUNE - I

source /cvmfs/dunedaq.opensciencegrid.org/products/setup
source /cvmfs/dunedaq-development.opensciencegrid.org/products/setup
setup protodune_wibsoft v351a -q e19:prof
cd /nfs/sw/wib/WIBSoftwareTrunk/scripts
BUTool.exe -X power_status_FEMB_VD_ColdBox.script # will give you the status
BUTool.exe -X power_on_FEMB_VD_ColdBox.script # turn the FEMB on
BUTool.exe -X power_off_FEMB_VD_ColdBox.script # turn the FEMB off

Troubleshooting

• If your run fails on init and you see response code 503, then unset the proxy: exit from nano04rc and run source ~np04daq/bin/web_proxy.sh -u, then restart
• The conf commands failed and you see a message that tells you that some FIFO queues could not be synced: The WIB timing is not correctly aligned, see the section for "Align timing"
• The run was started but there is no data (and you are not running with the SSPs): Make sure that the FELIX is aligned

Viewing run information

• Open the Grafana server at http://np04-srv-009:3000/
• Select your partition id on the top left.
• The messages shown in top box can be filtered to show for example just ERROR messages
• Lower panels show key quantities like the trigger rate and the data writer which show that data is flowing through the DAQ system.
• From the overview screen one can select subsystems in the top right

• To get information about the latest runs in the run registry, see this page: http://dunedaq-microservices:5005/runregistry/getRunMetaLast/100

• The DQM display can be found here

-- WesleyRobertKetchum - 2024-02-27