2016 TOF EOR reasons

249215 (COSMICS) 08/03: TOF went in MIXED due to a misconfigured TRM.

250766 (COSMICS) 06/04: LV failures in crate 43 consequently to a spike in the HV.

252987 (COSMICS) 30/04: glitch in the LV of crate 11, failure in the DCDCs and in the 2.7V PSs.

253018 (COSMICS) 01/05: TOF in MIXED because all the VME boards in SM12 went in STANDBY (reboot of FEE servers on alitofsm12?).

253073 (COSMICS) 02/05: 3.3V HVMAX in crate 17, PAR executed successfully but then the crate went again in error while resuming the data taking.

253113 (COSMICS) 02/05: 3.3V HVMAX in crate 17, PAR not executed (issue found in the ECS configuration).

253116 (COSMICS) 02/05: 3.3V HVMAX in crate 17, crate turned off and excluded from FSM and DAQ.

254199 (pp, PHYSICS) 15/05: LV failure in crate 6 (3.3V HVMAX?), the TOF didn't move to MIXED because the communication via the OPC was broken after the central DCS intervention on 12/05. Communication restored restarting the OPC servers and client.

254200 (pp, PHYSICS) 15/05: the DDL 1286 was excluded from the DAQ after the crash of run 254199, but clearly the DDL 1287 failed the SOR since the corresponding crate right was OFF (but crate 06 was still READY in the DCS because the OPC was down).

254201 (pp, PHYSICS) 15/05: the DDL 1286 was excluded from the DAQ after the crash of run 254199, but clearly the DDL 1287 failed the SOR since the corresponding crate right was OFF (but crate 06 was still READY in the DCS because the OPC was down).

254202 (pp, PHYSICS) 15/05: the DDL 1287 was excluded from the DAQ, but the DDL 1286 was included back by mistake (crate 06 was still READY in the DCS).

254822 (COSMICS) 23/05: we try to reinclude the crate 17 in the readout (it stayed completely ON but excluded from the DAQ during the whole weekend without any problem), but after few hours we get 3.3V HVMAX and some TRMs deconfigured in crate 16 (TOF MIXED, PAR not triggered).

255350 (pp, PHYSICS) 01/06: 3.3V HVMAX in crate 29, recovered with a power cycle. PAR not executed because the CAEN declared the crate OFF and only 5 second later the HVMAX.

255442 (pp, PHYSICS) 02/06: 3.3V HVMAX in crate 37, recovered with a power cycle. PAR not executed because the CAEN declared the crate OFF and only 5 second later the HVMAX.

255541 (pp, PHYSICS) 04/06: CTTM in ERROR.

255577 (pp, PHYSICS) 05/06: problems with DDL crate 04.

255578 (pp, PHYSICS) 05/06: CDH error crate 04.

255579 (pp, PHYSICS) 05/06: CDH error crate 04.

255580 (pp, PHYSICS) 05/06: CDH error crate 04.

255581 (pp, PHYSICS) 05/06: CDH error crate 04, fixed with a reset and reconfiguration.

255583 (pp, PHYSICS) 05/06: 3.3V HVMAX in crate 35, crate excluded.

256219 (pp, PHYSICS) 12/06: all LV channels in crate 37 went in ERROR (HVMAX).

256356 (pp, PHYSICS) 14/06: LV failure in crate 04, not recoverable. DDL 04 and 05 excluded.

256361 (pp, PHYSICS) 14/06: TOF moved in ERROR by mistake by TOF oncall (while trying to recover the LV of crate 04).

256418 (pp, PHYSICS) 16/06: 3.3V HVMAX in crate 06, in the DCS crate 06 in MIXED and crate 07 in OFF... off course no PAR executed.

256565 (pp, PHYSICS) 18/06: HV moved by mistake by the TOF oncall.

256589 (pp, PHYSICS) 18/06: crate 27 Unplugged.

257082 (pp, PHYSICS) 27/06: CTTM in ERROR.

257892 (pp, PHYSICS) 13/07: CDH error crate 28.

257957 (pp, PHYSICS) 14/07: crate 02 Unplugged.

258047 (pp, PHYSICS) 15/07: CTTM in ERROR.

258455 (pp, PHYSICS) 24/07: TRM 6 crate 10 recovered by dim server while running (it should never happen...).

258887 (pp, PHYSICS) 01/08: CTTM failed SOR ack (but CTTM excluded from FSM...).

259307 (pp, PHYSICS) 07/08: LV failure in crate 11 (3.3V 0V Prot).

259606 (COSMICS) 12/08: DRM 51 broken (slot in FAIL).

260355 (pp, PHYSICS) 27/08: run not started, DRM 48 in ERROR.

260490 (pp, PHYSICS) 29/08: crate 28 Unplugged, no way to recover it with 48V Srv power cycle.

263652 (pp, PHYSICS) 15/10: HV module SM11Mod1Pos OFF, TOF moved to MIXED.

264164 (pp, PHYSICS) 23/10: DDL 54 stuck.

264165 (pp, PHYSICS) 23/10: run doesn't start (CDH error equipment 1334).

264166 (pp, PHYSICS) 23/10: run doesn't start (CDH error equipment 1334, same as run 264165).

264167 (pp, PHYSICS) 23/10: run doesn't start (CDH error equipment 1334, same as run 264165 and 264166). Fixed by restarting the vme server.

265059 (COSMICS) 05/11: run stopped by TOF oncall mistake trying to debug a readout problem on crate 03 (TOF excluded from the readout by still triggering).

265334 (p-Pb 5TeV, PHYSICS) 11/11: 3.3V HVMAX in crate 52, recovered with a power cycle. PAR not executed because the CAEN declared the crate OFF and only 5 second later the HVMAX.

-- AndreaAlici - 2016-05-19