.. raw:: html

    <style> .green {color:#228B22; font-weight:bold; font-size:16px} </style>

.. role:: green

.. raw:: html

    <style> .red {color:#aa0060; font-weight:bold; font-size:16px} </style>

.. role:: red

.. raw:: html

    <style> .yellow {color:#FEE128; font-weight:bold; font-size:16px} </style>

.. role:: yellow

Troubleshooting
===============

This chapter presents a list of the most common problems encountered in running the DSSC detector.

What to do when
---------------

Here is a list of what to check during detector operation to ensure everything is working fine:

:green:`Detector status: up and running (sending data)`

.. table:: Detector safety issues
    :widths: 20 20 60

    +---------------------------+--------------------------+-------------------------------------------------------+
    | Scene                     | Issue                    | Action                                                |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`DSSC_Power_Status` | Channel :red:`red`       | :red:`Call DET OCD immediately.`                      |
    |                           | (ERROR)                  | Switch OFF power with All Off button in the MAIN scene|
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`DSSC_Power_Status` | Channel :yellow:`yellow` | :red:`Call DET OCD immediately.` MPOD has lost        | 
    |                           | (UNKNOWN)                | connection with Karabo and can no longer be controlled| 
    |                           |                          | This is however not an emergency situation, but the   |
    |                           |                          | detector cannot be switched OFF as control is         |
    |                           |                          | disconnected.                                         |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`DSSC_HV_Status`    | Channel :red:`red`       | :red:`Call DET OCD immediately.`                      |
    |                           | (ERROR)                  | Switch OFF power with All Off button in the MAIN scene|
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`DSSC_HV_Status`    | Channel :yellow:`yellow` | :red:`Call DET OCD immediately.` MPOD has lost        |
    |                           | (UNKNOWN)                | connection with Karabo and cannot be controlled       |
    |                           |                          | anymore. This is however not an emergency situation,  |
    |                           |                          | but the detector cannot be switched OFF as control is |
    |                           |                          | disconnected.                                         |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`SIB`               | :code:`Crate En.` lamps  | :red:`Call DET OCD immediately.` If you cannot reach  |
    |                           | turn from green to blue  | an expert, switch OFF power with All OFF button in    |
    |                           | or red during operation  | the MAIN scene.                                       |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`SIB`               | ASIC_T status flickering | Flickering from time to time is normal if the         |
    |                           | :red:`red`               | flickering channel goes back to normal at some point  |
    |                           | :red:`(value 2)` in one  |                                                       |
    |                           | or more quadrants        |                                                       |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`SIB`               | ASIC_T status stably     | Open the :code:`SIB_ASICTempChecks` scene and check   |
    |                           | :red:`red (value 2)` in  | the values. During operation at -20 degC, they should |
    |                           | one or more quadrants    | be at -12 degC (within 2 degC). The threshold is at   |
    |                           |                          | -10 degC. If some value is obviously wrong (e.g.,     |
    |                           |                          | -50 degC) do not do anything but note down the ASIC   |
    |                           |                          | and quadrant number in elog. If some value is between |
    |                           |                          | -10 degC and 10 degC, call DET OCD during the day,    |
    |                           |                          | and note down and monitor during the night. If the    |
    |                           |                          | value seems to increase, call DET OCD. If some value  |
    |                           |                          | is above zero degC, call DET OCD.                     |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`SIB`               | ASIC_T status stably     | Check that the quadrant where the value is 3 is       |
    |                           | :red:`red (value 3)` in  | sending data. If data are not sent, this value is     |
    |                           | one or more quadrants    | normally 3. It should go back to zero when data are   |
    |                           |                          | sent.                                                 |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`SIB`               | IOB T Status :red:`red`  | Check that ASICs are ON in the DSSC MAIN panel. If    |
    |                           | :red:`(status 3)` in one | ASICS are OFF, a status 3 there is normal. If ASICS   |
    |                           | or more quadrants        | are ON, call DET OCD.                                 |
    +---------------------------+--------------------------+-------------------------------------------------------+
    | :code:`SIB`               | IOB T Status :red:`red`  | :red:`Call DET OCD immediately.`                      |
    |                           | :red:`(status 2)` in one | Switch OFF power with ALL OFF button in the MAIN scene|
    |                           | or more quadrants        |                                                       |
    +---------------------------+--------------------------+-------------------------------------------------------+

List of NaN in the :code:`SIB_ASICTempScene`:

For the following ASICs, it is normal to have a NaN readout in the :code:`SIB_ASICTempScene`:

Q1, Q2 and Q3: No ASIC should have a NaN readout.

Q4: ASIC 9 and 38

.. table:: Data taking issues

    +---------------------------+---------------------------+------------------------------------------------------+
    | Scene                     | Issue                     | Action                                               |
    +---------------------------+---------------------------+------------------------------------------------------+
    | :code:`DSSC_Main`/        | PPT devices in error state| Try shutting down the corresponding                  |
    | PPT/ASICs Configuration   |                           | :code:`SCS_CDIDET_DSSC(SQS_NSQ_DSSC)/FPGA/PPT_Qi` devices and      |
    | panels                    |                           | instantiate them again. If this does not work, call  |
    |                           |                           | DET OCD.                                             |
    +---------------------------+---------------------------+------------------------------------------------------+
    | :code:`DSSC_Main`/        | Output rate is zero for   | Check if data are sent from detector. Press start    |
    | PPT/ASICs Configuration   | all quadrants             | data sending.                                        |
    | panels                    |                           |                                                      |
    +---------------------------+---------------------------+------------------------------------------------------+
    | :code:`DSSC_Main`/        | Output is zero for one    | Open the Init Qi scene of the quadrant not sending   |   
    | PPT/ASICs Configuration   | quadrant                  | the data. Press Init System and then RUN XFEL mode.  |
    | panels                    |                           | If this does not work, call DET OCD.                 |
    +---------------------------+---------------------------+------------------------------------------------------+
    | :code:`DSSC_Main`/        | Start PPT devices button  | Open the Init Qi scenes of the single quadrants and  |
    | PPT/ASICs Configuration   | has no effect (no change  | press Init System in each of them. Then go back to   | 
    | panels                    | of currents as in         | the PPT/ASICs Configuration panels and when the      | 
    |                           | :numref:`config`          | currents are OK (see :numref:`config`), press Init   |
    |                           |                           | PPT devices there. If this does not work, call DET   |
    |                           |                           | OCD.                                                 |
    +---------------------------+---------------------------+------------------------------------------------------+
    | :code:`PREVIEW`           | :code:`PREVIEW` not       | See :numref:`Online`                                 |
    |                           | updating                  |                                                      |
    +---------------------------+---------------------------+------------------------------------------------------+
    | :code:`PREVIEW_CORRECTED` | :code:`PREVIEW` not       | See :numref:`Online`                                 |
    |                           | updating                  |                                                      |
    +---------------------------+---------------------------+------------------------------------------------------+

.. _Online:

Online preview not updating
---------------------------

If the online preview is not updating anymore, there might be different problems.

**Check that the detector is correctly sending data.** Go to the individual :code:`SCS_CDIDET_DSSC/FPGA/PPT_Qi` devices and in the configuration editor, look for 'SFP output rate'. This should be different from zero, normally a number like 4197 Mbit/s, but it also depends on the VETO configuration. If the number is zero for one of the quadrants, then the quadrant is not sending data. Check the currents, in case press on Stop acquisition and then on Init system.

You can also **check the network traffic to the DAQ nodes** via the remcom interface. Type `https://remcom.desy.de/ <https://remcom.desy.de/>`_ on a network browser, log in as :code:`SCSDAQ` or as :code:`SQSDAQ`. Click on 'SCSDAQ(SQSDAQ)' and then on 'DAQ data aggregators network traffic'. Select the required data sources (all the :code:`scs(sqs)-br-sys-daq-XX`, where :code:`XX` goes from 00 to 15). Then click on 'Run Job Now' on the top right side of the screen. The system will give you back the amount of (possible) failures.

**Check that DSSC-XTDF is included in the DAQ data sources** in the SCS(SQS) run control. If this is not the case, the DSSC data are not sent to the DAQ and therefore cannot be displayed in the online preview.

If all this looks OK, you might need to **restart the calibration pipeline.** The procedure is as follows:
    - Stop acquisition on all the scences 'Init_Qi'.
    - On the scene :code:`Subprojects` :math:`\rightarrow` :code:`DSSC_ONLINE_CAL` :math:`\rightarrow` :code:`MANAGER`, push the button 'Restart Servers', and when the process is finished, push 'Restart Pipeline'.
    - Check on the :code:`PREVIEW` scene that all the modules are displaying data correctly.

.. figure:: Figures/fig1_chapter6.png
    :width: 100%

Frames getting IDs up to 810
----------------------------

The problem was solved switching on and off the quadrants. A soft reset of the FPGA on the PPT did not help.

Restarting the DAQ
------------------

:code:`SCS(SQS)_DAQ_DSSC`: in case the DAQ seems not working, restart the servers under this device (:code:`scs(sqs)_daq_0` ... :code:`scs(sqs)_daq_15`). After that, one should restart also online pipeline preview. Follow the procedure below.

Restart DAQ servers (e.g. online preview not updating some module, after restarting online preview)

    1. Stop sending data from the detector **(very important)**. Use the :code:`Init_Qi` scenes.
    #. From `https://remcom.desy.de/ <https://remcom.desy.de/>`_ , go to reset DAQ.
    #. Select all nodes (xtdf data nodes 0-15).
    #. Select NONE of slow data nodes.
    #. Then, press run job now :math:`\rightarrow` Aggregators are then killed.
    #. Then, go to subproject :code:`SCS(SQS)_DAQ_MACRO`.
    #. Choose macro :code:`SCS(SQS)_AUTO_INSTANTIATE`.
    #. Run the macro by pressing the green button, macro comes out in the line below.
    #. In the macro, press start.
    #. Run controller servers and database devices need to be reinstaintiated by hand (:code:`pythonServers/scs(sqs)_daq_rmdc_1`, :code:`cppServers/scs(sqs)-daq_rc_1`, :code:`cppServers/scs(sqs)_daq_conf_1`).
    #. At the end, restart the online preview pipeline (scene :code:`Manager` in :code:`DSSC_ONLINE_CAL`, see manual above).
    #. Select proper proposal and data sources