-- AlessandroThea - 2022-06-08

Taking runs at EHN1 01/12/22:

Getting started

You need to login to these hosts:

np04-srv-024 - take runs

np04-srv-026, np04-srv-028, np04-srv-029, np04-srv-030 - control the FELIX card, log files for readout:

When logging into np04-srv-024 log in as np04daq i.e.

ssh np04daq@np04-srv-024

Note: documentation uses /nfs/sw/dunedaq/dunedaq-v3.2.1-24-01-23-fwtp as the default environment, but other dunedaq environments can be created (see: https://dune-daq-sw.readthedocs.io/en/latest/packages/daq-buildtools/)

np04-srv-024

When logging into np04-srv-024 you will see a list of tmux sessions at the start, if not run tmux ls. Connect to an existing session which typically ends in fwtpg e.g.

tmux a -t np04-coldbox-fwtpg

to safely exit out of this window run tmux detach

If the configuration doesn't exist make a new one with

tmux new -s np04-<detector type>-fwtpg

Where detector type refers to the readout host i.e. coldbox/APA1 etc. In this window you will be in the run area and the dunedaq environment should be setup. If not run the following:

cd /nfs/sw/dunedaq/dunedaq-v3.2.1-rc-2-fwtp
source setup_for_run.sh

Configurations

A list of all possible configurations are in /nfs/sw/dunedaq/dunedaq-v3.2.1-24-01-23-fwtp/configurations.

Configurations fall into three categories

• daq configurations
• flxcard configurations
• wib configurations

and with a combination of three it is possible to run with the APAs either indicudually or together (referred to as TPC configurations).

The configurations are compiled using the script /nfs/sw/dunedaq/dunedaq-v3.2.1-24-01-23-fwtp/configurations/recreate_all_configurations.sh and lists commands for how to compile each indidual one. It is generally recommended to not run this script all the time unless major changes to configuration generation are made e.g. a new configuration parameter is introduced in the schema or an update to how daqconf parses the consifuration is done.

DAQ configuration:

there are three types of daq configurations made:

• only ADC links : np04_daq.json
• ADC + TP links for threshold 1500 (pulser) : np04_daq_fwtp.json
• ADC + TP links for threshold 20 : np04_daq_fwtp_noise.json

An example of how to create a daq configuration (see recreate_all_configurations.sh for more)

daqconf_multiru_gen --debug -c np04_daq_fwtp.json --hardware-map-file `pwd`/hw_maps/np04_hw_map.txt np04_daq_fwtp_conf

For firmware TP configuration there are some particularly important parameters to pay attention to, see one of the previously mentioned json files to see a full ist.

• emulator : sets fake timestamps for data, should only be true used when testing with pattern files
• use_felix : creates a readout configuration which uses the FELIX card, should always be true
• enable_firmware_tpg : enables creation of TP linkhandlers and dtpcontroller plugin, should be true
• firmware_hit_threshold : ADC threshold in the hitfinder algorithm
• dtp_connections_file : address tables for the hitfinding firmware, should match the firmwre you are testing/running with
• readout_sends_tp_fragments : tp fragments are connected to the MLT i.e. enables the redout of TPs to trigger records, should be true
• enable_tpset_writing : enable writing of TPs that the trigger recieves to file, not really needed to be true if tp fragments are being readout
• output_paths : output path that dataflow writes the hdf5 file to, make sure the directory is readble/writable to the user np04daq and the firectory actualy exists.

FELIX card configuration:

there are two types of flxcard configurations:

• ADC links only np04_flxcard.json
• ADC + tp links np04_flxcard_fwtp.json

An example of how to create a daq configuration (see recreate_all_configurations.sh for more)

felixcardcontrollerconf_gen.py -c np04_flxcard_fwtp.json --hardware-map-file `pwd`/hw_maps/np04_hw_map_APA1.txt np04_flxcard_APA1_fwtp_conf 

all the configuration parameters are important in the flxcard controller but are fairly simple. In general this configuration should not be changed or modified, rather if you want to do things like disable some links, the hardware maps should be changed ad then the configrations should be regenerated (DAQ condifurations as well).

WIB configurations:

There is a WIB configuration for each APA and comibnation of them, but in general are borken down into two categories (take APA1 configs as an example):

• default WIB configuration : np04_wib_APA1.json
• WIB configuration which produces a 32dac calibation pulse : np04_wib_APA1_pulser.json

The pulser configurations are particularly useful for testing as they produce a constant TP flux given the pulser amplitude can be made to be much larger than the baseline noise.

Important configuration parameters are

• pulser : enable pusler
• pulser_dac : pulser amplitude

In principle the pulse frequency can be changed but this is not helpful as the TP flux is much higher than what would be expected in nominal operation.

Hardware maps decide which front end is readout from and contains the link mapping. To disable readout from certain links one would comment out a file in the map (with #) and then regenerate the flxcard configuration and daq configuration. The individual values themselves should not be altered.

If there is an issue with the configurations or hardware maps, please ask on dundaq-integration or check the standard list of configurations located in /nfs/home/np04daq/DAQ_NP04_HD_DEV_AREA/configurations.

Global configurations:

Global configurations are compilations of the three main types in a json file, an example is shown below:

{
    "apparatus_id":"fwtpg_test",
    "np04_wib":"../configurations/np04_wib_APA1_conf",
    "np04_flxcard": "../configurations/np04_flxcard_APA1_conf",
    "np04_daq":"../configurations/np04_daq_APA1_conf"
}

these are kept in /nfs/sw/dunedaq/dunedaq-v3.2.1-rc-2-fwtp/runarea/global_configs and it what you typically run with nano04rc.

Please note the configuration/hardware map depends on the readout host you are working with.

Please also keep all configurations inside the run area, and when modiying file paths, always use absolute paths!!

readout host (np04-srv-026, np04-srv-028, np04-srv-029, np04-srv-030)

dunedaq environment which works: /nfs/sw/dunedaq/dunedaq-v3.2.1-24-01-23-fwtp

and setup using the following:

cd /nfs/sw/dunedaq/dunedaq-v3.2.1-24-01-23-fwtp
source setup.sh

Check the git tag of the firmware is on the Felix card:

dtpbutler.py -v flx-0-p2-hf # SLR 0 card 0
dtpbutler.py -v flx-1-p2-hf # SLR 1 card 0
dtpbutler.py -v flx-2-p2-hf # SLR 0 card 1
dtpbutler.py -v flx-3-p2-hf # SLR 1 card 1

If it is not the correct one or not the one you are expecting, then the card needs to be flashed with the firmware bit file, the server needs to be rebooted, and drivers need to be restarted.

To flash the firmware on the felix card follow the instructions here: Integration Tests Instructions

Main scripts:

• For soft reset of card use configure.sh, note should only be done once at the start.

when configuring the felix card pass 0 to configure card 0 or 1 to card 2 i.e:

./configure.sh 0 # card 0
./configure.sh 1 # card 1

To monitor the link processors in the TPG core run:

dtpbutler.py flx-0-p2-hf link watch -RB # SLR 0 card 0
dtpbutler.py flx-1-p2-hf link watch -RB # SLR 1 card 0
dtpbutler.py flx-2-p2-hf link watch -RB # SLR 0 card 1
dtpbutler.py flx-3-p2-hf link watch -RB # SLR 1 card 1

Take Runs

On np04-srv-024:

nano04rc --timeout 120 <config-name> <username>
boot
conf
start_run TEST

Here the <config-name> can be either a DAQ config , WIB config or both

Check pages on grafana for monitoring and status: http://np04-srv-009:3000/

Make sure the window is np04-coldbox.

When done, on np04-srv-024:

stop_run

and from here you can start another run, or if you need to make changes to code:

scrap
terminate
quit

note for high TP rates the trigger module typcially times out resulting in an error in run control. In this instance just execute the quit command.

Raw Record Data

On np04-srv-024:

nano04rc --timeout 120 np04_coldbox_daq_fwtp_raw_record <username>
boot
conf
start_run TEST

Check pages on grafana for monitoring and status: http://np04-srv-009:3000/

Make sure the window is np04-coldbox.

In nan04rc run the following:

expert_command np04_coldbox_daq_fwtp_reaw_record/np04_coldbox_daq_fwtp_raw_record/runp04srv0280 record-cmd.json

and data will be recorded for 30s. note this data will be recorded to runarea and will be overwritten, so before recording any more data or when you are done move the files to the directory /data1/readout-HD-wib2fw/wib2-v3.1.0-fwtpg/ in np04-srv-001.

When done, on np04-srv-024:

stop_run

and from here, you can start another run, or if you need to make changes to the code:

scrap
terminate
quit

**Note down any significant runs or tests in the elogs and validation spreadsheet:** https://pdsp-elog.cern.ch/elisa/display?logbook=ProtoDUNE-SP, https://docs.google.com/spreadsheets/d/1KIy0UAP5FpjGJt437yfA8sovCHp8LE4Iyuw31HvZwY8/edit#gid=0

Troubleshooting Runs:

To check for issues in the run, the grafana dashboard main page contains a log which shows warnings/errors so if something is not right e.g. TRs aren't being written to file, then check there first. This will name the module which produced the error. Now, stop the run and check the log files. Log files are procuded in the directory /log on the machine which the hosts the particular module. To check which host to look at, refer back to the configuration or the terminal history for the run control. The log files are only identified by the time the run started and the module name so make sure to keep a reference of these. Inspect the log file, typically the last few lines will contain the error message you are looking for (if the system crashes you may need to reproduce the crash a few times to see the error message). Make note of the log file and post on dunedaq-integration on slack and (hopefully) a DAQ expert can assist you.

For the firmware TPG specifically, you first check the activity of the link processors as described in the previous section and make sure data is flowing through the full chain. You can then inspect the frontend page on grafana to see the data stream and error rates for the ADC and TP data. (TP data are links 0_0, 0_5, 1_0, 1_5). This is the first place you can check the effctive TP rate as recieved from the FELIX, which indicates immediate issues with the FELIX or the firmware. If no block rate is seen on the TP links (especially with WIB pulser runs) this can indicate link processor lockup for various reasons, so debugging with the dtpbutler tools is required. If there are high amounts of block errors this may point to an issue with the readout host or FELIX itself i.e. drivers or the machine itself, so if this occurs, check the block errors are reproducable (ideally with and without firmware TPG enabled) to narrow down the issue. Then best to address a FELIX expert or post on dunedaq-integration.

Links to documentation and references:

General DUNEDAQ documentation: https://twiki.cern.ch/twiki/bin/view/CENF/DUNEDAQProto

Run control documentation: https://twiki.cern.ch/twiki/bin/view/CENF/NanorcRunControl

Making dunedaq environments: https://dune-daq-sw.readthedocs.io/en/latest/packages/daq-buildtools/

flxlibs integration tests: https://dune-daq-sw.readthedocs.io/en/latest/packages/flxlibs/Integration-tests/

Run logs and analysis: https://docs.google.com/spreadsheets/d/1vwtVPSAwBSrzUtolyYWtaVipnT8TedUsp2-HN1niq-c/edit?usp=sharing

HD Coldbox tests 12/07/22:

tested new FELIX firmware which is able to produce TPs with the DUNE WIB (WIB2).

1. checked nominal reaodut of ADC was possible

2. raw recorded ADC + firmware TP data

3. checked firmware TPs are written to TP stream files, and Trigger records are produced

Firmare Tested:

WIB2_PACKET_PROTECTION_ral-fpga-forge-01_220623_1635

software tested:

dunedaq-v2.11.1 used for point 1

dunedaq-v3.0.1 plus branches:

https://github.com/DUNE-DAQ/detdataformats/tree/jbrooke/fw_tp_update https://github.com/DUNE-DAQ/fdreadoutlibs/tree/hristova/fw_tp_update

used for points 1, 2 and 3.

recorded data is noise

location of raw recorded data and TRs + TPstreams is /data1/readout-HD-wib2fw on np04-srv-001

Working area for dunedaq-v3.0.1 is /nfs/sw/dunedaq/dunedaq-v3.0.1-flx-fw-test

configuration used:

daqconf_multiru _gen --host-ru np04-srv-028 --host-df np04-srv-001 --host-dfo np04-srv-001 --host-hsi np04-srv-001 --host-trigger np04-srv-001 --op-env np04_coldbox -o /data1/readout-HD-wib2fw --opmon-i mpl cern --ers-impl cern -n 10 -b 260000 -a 2144 --clock-speed-hz 62500000 -f --enable-firmware-tpg --enable-raw-recording --region-id 0 --frontend-type wib2 --thread-pinning-file /nfs/s w/dunedaq/dunedaq-v2.11.0-dev/configurations/thread_pinning_files/cpupin-np04-srv-028.json --hsi-trigger-type-passthrough --enable-dqm --host-dqm np04-srv-001 --dqm-cmap PD2HD --dqm-impl cern -t 0.1 np04_coldbox_daq_100mHz-fwtpg

Notes:

no operational monitoring was possible at the time so log files were used mainly to check the state of the system.

VD coldbox test software enviromnents:

For coldbox tests, software areas are located on nfs mounted storage on the np04 servers:

2.11.1 release including a branch of daqconf to allow firmware tp configuration.

Use this software environment for raw recording of ADC + TP data. configurations should be located in configurations in the home directory of the environment

/nfs/sw/dunedaq/dunedaq-v2.11.1-readout

3.0.0 release candidate containing branches from fdreadoutlibs and daqconf (subject to change pending v3.0.0-5 release candidate). fdreadoutlibs branch necessary for testing self triggering of firmware TPs.

Use this software environment for raw recording of ADC + TP data. configurations should be located in configurations in the home directory of the environment

/nfs/sw/dunedaq/dunedaq-v3.0.0-rc5-readout

Use this software environment for self triggering tests and firmware TP writing (raw recording also possible).

Location of data will be in np04-srv-002 in directory /data0/test-readout-raw-record

3.0.0 software enviroment (TP publishing workarea):

Configuration tested is a 1ADC + 1TP configuration with the real card:

daqconf_multiru_gen -f -e --host-ru localhost --region-id 0 --host-df localhost -o . --number-of-data-producers 1 --enable-firmware-tpg --enable-raw-recording --enable-tpset-writing --trigger-activity-config 'dict(prescale=500)' --trigger-candidate-config 'dict(prescale=20)' --tpg-channel-map ProtoDUNESP1ChannelMap flx-fw-json --debug

daqconf module graphs (TBD)

Run Log at NP-02:

Link to logbook

Test readout of ADC data with firmware v2.1.0 and dunedaq-v2.11.1

Raw record with dunedaq v2.11.1

Test readout/tpset publishing of firmware TPs with dunedaq-v3.0.0 (1st attempt)

Test readout/tpset publishing of firmware TPs with dunedaq-v3.0.0 (2nd attempt)

Longest run with triggering on firmware TPs

Map block rate of TP links from FELIX

Raw recorded Firmware TPs at varying thresholds with v2.1.0 and v2.1.3 firmware

Firmware Run Validation book

Link to validation notebook

Attempt at running readout and TP publishing of firmware TP's with dunedaq-v3.0.0

Run area: /nfs/sw/dunedaq/dunedaq-v3.0.0-rc5-readout

nano04rc configuration ran: /nfs/sw/dunedaq/dunedaq-v3.0.0-rc5-readout/runarea/np02_coldbox_daq_100mHz-fwtpg

DUNEDAQ software environment based on release candidate 5 and the following branches:

daqconf

* batchelor/add_booleans_triggergen

detdataformats

* batchelor/michel_trigger_type

fdreadoutlibs

* hristova/tpset_ehn1_testing_backup

nanorc

* develop

trigger

* batchelor/michel_trigger_3_0_0

triggeralgs

* batchelor/michel_trigger

command for generating configuration:

daqconf_multiru_gen -n 1 -b 200000 -a 4800 -o /data0/test-readout-raw-record/datasets-v3.0.0-rc -f --enable-firmware-tpg --enable-raw-recording --host-ru np04-srv-030 --host-df np04-srv-002 --host-trigger np04-srv-002 --host-hsi np04-srv-002 --region-id 4 --enable-dqm --opmon-impl cern --ers-impl cern --dqm-impl cern --dqm-cmap VD --tpc-region-name-prefix CRP --op-env np02_bde_coldbox --use-hsi-hw --control-hsi-hw --hsi-device-name PROD_MASTER_VDC_CH --hsi-source 1 --ttcm-s1 128 --hsi-re-mask 1 --host-timing np04-srv-012 --control-timing-partition --timing-partition-master-device-name PROD_MASTER_VDC_CH --trigger-rate-hz 0.1 --host-dqm np04-srv-002 --thread-pinning-file /nfs/sw/work_dirs/phrodrig/dunedaq-v2.10.2-testrels/cpu-pin-for-coldbox/sourcecode/readoutlibs/config/cpupins/cpupin-np04-srv-030-realdata.json --trigger-activity-plugin TriggerActivityMakerHorizontalMuonPlugin --trigger-candidate-plugin TriggerCandidateMakerHorizontalMuonPlugin --trigger-activity-config 'dict(trigger_on_n_channels=True, trigger_on_adjacency=False, window_length=10000, n_channels_threshold=40)' np02_coldbox_daq_100mHz-fwtpg

Configuration has a signle datalinkhandler and tp linkhandler on SLR 0.

HfButler commands:

hfButler.py flx-0-p2-hf init
hfButler.py flx-1-p2-hf init
hfButler.py flx-0-p2-hf cr-if --on --drop-empty
hfButler.py flx-1-p2-hf cr-if --on --drop-empty

# set source to gbt i.e. process data from the elinks
hfButler.py flx-0-p2-hf flowmaster --src-sel gbt --sink-sel hits

# set the threshold of the hitfinder
hfButler.py flx-0-p2-hf link hitfinder -t 20

# mask all but channel 0 on each link proc
hfButler.py flx-0-p2-hf link mask enable -c 1-63

hfButler.py flx-0-p2-hf link config -i 4 --dr-on --dpr-mux passthrough --drop-empty

HfButler commands enables link processors for single link, mask channels 63 channels on each TPG pod (4 channels total) and sets the hit threshold to 20 (default value).

Felix configuration commands:

# full reset of card
flx-reset ALL
flx-init
flx-reset GTH
# check connection to front end devices
flx-info POD
flx-info LINK

echo "enabling readout links:"
sleep 1

echo "elink status before setting"
for i in $(seq 0 5); do flx-config -d 0 DECODING_LINK0${i}_EGROUP0_CTRL_EPATH_ENA; done;

for i in 0 5; do flx-config -d 0 set DECODING_LINK0${i}_EGROUP0_CTRL_EPATH_ENA=1; done;


echo "elink status after setting"
for i in $(seq 0 5); do flx-config -d 0 DECODING_LINK0${i}_EGROUP0_CTRL_EPATH_ENA; done;


echo "setting superchunk factor:"
sleep 1

echo "superchunk factor before setting"
for i in $(seq 0 5); do flx-config -d 0 list | grep SUPER_CHUNK_FACTOR_LINK_0${i}; done;

for i in 0 5; do flx-config -d 0 set SUPER_CHUNK_FACTOR_LINK_0${i}=0x0c; done;


echo "superchunk factor after setting"
for i in $(seq 0 5); do flx-config -d 0 list | grep SUPER_CHUNK_FACTOR_LINK_0${i}; done;


echo "card configuration complete!"

Enables elink 0 (data link) and 5 (tp link), sets superchunk factor of each link to 12.

Initial run resulted in readout application crashing with segmentation fault:

2022-Jun-10 18:14:19,704 ERROR [static void ers::ErrorHandler::SignalHandler::action(int, siginfo_t*, void*) at /tmp/root/spack-stage/spack-stage-ers-v1.1.6-kc2qh3mkzji2bjmxkww2q54utrsznz76/spack-src/src/ErrorHandler.cpp:90] Got signal 11 Segmentation fault (invalid memory reference)
   Parameters = 'name=Segmentation fault (invalid memory reference)' 'signum=11' 
   Qualifiers = 'unknown' 
   host = np04-srv-030
   user = np04daq (106200)
   process id = 136116
   thread id = 136127
   process wd = /nfs/sw/dunedaq/dunedaq-v3.0.0-rc5-readout/runarea
   stack trace of the crashing thread:
     #0  /cvmfs/dunedaq-development.opensciencegrid.org/candidate-releases/rc-dunedaq-v3.0.0-5/spack-0.17.1/opt/spack/gcc-8.2.0/readoutmodules-v1.1.1-hjkwjswnqenhvymd7fuj2kpk37y2wrkl/lib64/libreadoutmodules_DataLinkHandler_duneDAQModule.so(dunedaq::fdreadoutlibs::WIBTPHandler::try_sending_tpsets(unsigned long)+0x150) [0x7f2f89350c20]
     #1  /lib64/libpthread.so.0(+0x12ce0) [0x7f2fac169ce0]
     #2  /cvmfs/dunedaq-development.opensciencegrid.org/candidate-releases/rc-dunedaq-v3.0.0-5/spack-0.17.1/opt/spack/gcc-8.2.0/readoutmodules-v1.1.1-hjkwjswnqenhvymd7fuj2kpk37y2wrkl/lib64/libreadoutmodules_DataLinkHandler_duneDAQModule.so(dunedaq::fdreadoutlibs::WIBTPHandler::try_sending_tpsets(unsigned long)+0x150) [0x7f2f89350c20]
     #3  /cvmfs/dunedaq-development.opensciencegrid.org/candidate-releases/rc-dunedaq-v3.0.0-5/spack-0.17.1/opt/spack/gcc-8.2.0/readoutmodules-v1.1.1-hjkwjswnqenhvymd7fuj2kpk37y2wrkl/lib64/libreadoutmodules_DataLinkHandler_duneDAQModule.so(dunedaq::fdreadoutlibs::RAWWIBTriggerPrimitiveProcessor::tp_stitch(dunedaq::detdataformats::wib::RawWIBTp*)+0x186) [0x7f2f89352206]
     #4  /cvmfs/dunedaq-development.opensciencegrid.org/candidate-releases/rc-dunedaq-v3.0.0-5/spack-0.17.1/opt/spack/gcc-8.2.0/readoutmodules-v1.1.1-hjkwjswnqenhvymd7fuj2kpk37y2wrkl/lib64/libreadoutmodules_DataLinkHandler_duneDAQModule.so(dunedaq::fdreadoutlibs::RAWWIBTriggerPrimitiveProcessor::tp_unpack(dunedaq::fdreadoutlibs::types::RAW_WIB_TRIGGERPRIMITIVE_STRUCT*)+0x1a1) [0x7f2f89352cc1]
     #5  /cvmfs/dunedaq-development.opensciencegrid.org/candidate-releases/rc-dunedaq-v3.0.0-5/spack-0.17.1/opt/spack/gcc-8.2.0/readoutmodules-v1.1.1-hjkwjswnqenhvymd7fuj2kpk37y2wrkl/lib64/libreadoutmodules_DataLinkHandler_duneDAQModule.so(dunedaq::readoutlibs::ReadoutModel<dunedaq::fdreadoutlibs::types::RAW_WIB_TRIGGERPRIMITIVE_STRUCT, dunedaq::readoutlibs::DefaultRequestHandlerModel<dunedaq::fdreadoutlibs::types::RAW_WIB_TRIGGERPRIMITIVE_STRUCT, dunedaq::readoutlibs::BinarySearchQueueModel<dunedaq::fdreadoutlibs::types::RAW_WIB_TRIGGERPRIMITIVE_STRUCT> >, dunedaq::readoutlibs::BinarySearchQueueModel<dunedaq::fdreadoutlibs::types::RAW_WIB_TRIGGERPRIMITIVE_STRUCT>, dunedaq::fdreadoutlibs::RAWWIBTriggerPrimitiveProcessor>::run_consume()+0x2c9) [0x7f2f89341169]
     #6  /cvmfs/dunedaq-development.opensciencegrid.org/candidate-releases/rc-dunedaq-v3.0.0-5/spack-0.17.1/opt/spack/gcc-8.2.0/readoutmodules-v1.1.1-hjkwjswnqenhvymd7fuj2kpk37y2wrkl/lib64/libreadoutmodules_DataLinkHandler_duneDAQModule.so(dunedaq::readoutlibs::ReusableThread::thread_worker()+0x66) [0x7f2f8930c3f6]
     #7  /cvmfs/dunedaq.opensciencegrid.org/spack-externals/spack-0.17.1/opt/spack/gcc-8.2.0/gcc-8.2.0-qzyv72qtwh7vtxakynu3vs6iup6hhhug/lib64/libstdc++.so.6(+0xc3c5f) [0x7f2fab745c5f]
     #8  /lib64/libpthread.so.0(+0x81ca) [0x7f2fac15f1ca]
     #9  /lib64/libc.so.6(clone+0x43) [0x7f2faaf59d83]
bash: line 1: 136116 Aborted                 (core dumped) daq_application --name ruflx0 -c rest://localhost:3337 -i influx://opmondb.cern.ch:31002/write?db=influxdb 

Error points to issue in RawWIBTPHandler and RawWIBTriggerPrimitiveProcessor (more debugging needed).

Configuring the FELIX card after a run like this causes server to reboot (np04-srv-030).

Without configuring the card, another run would not cause the readout to crash and nanorc can be stopped without crashing. Raw recording from nanorc only resulted in ADC data being recorded and no firmware TPs.

Tests with low level tools inicates TPs are flowing in Firmware i.e. no lockup of link processor and TPs are reaching the DMA.