Restarting Online Calibration

Preface

Restarting the online calibration devices should happen as part of normal shift start-up and during trouble shooting. Please read the following sections first, in case you are not sure if a restart is actually needed:

General Detector DAQ Troubleshooting

Before Restart

Since the restart process involves many device which act on input in an event driven (push) fashion, it is best to assure that no new data is input into the pipeline during the restart process. Either:

  • stop the detector from sending data. DAQ can stay in monitoring mode
  • if you do not want to stop the detector, switch the DAQ to ignore mode

Restart via GUI

Follow this procedure in case you are on-site or otherwise have stable and fast access to a Karabo-GUI of the respective instrument.

  1. Follow the instructions in Initializing a Karabo Environment for an Instrument to open a Karabo-GUI, then load the master project of the respective detector. The will be a sub-project for calibration, which contains a MANAGER scene.
  2. In this scene click on the Restart servers button, as shown in the screenshot. This will restart all servers involved in online calibration. It will take about 1 minute and will output the message “All servers restarted!” in the log window right above the button.
  3. After restarting all servers, click Start pipeline to actually start the pipeline again. This might take a few minutes, especially until all constants have been loaded. The state color field will turn to green and the general state will turn to active when it is done, as shown in the screenshot.
../../_images/cal_online_restart_gui.png

Fig. 2 Restarting servers and re-initializing calibration pipeline for MHz detectors.

Remote restart via command line

Follow this procedure in case you are giving remote support and do not have stable and fast access to a Karabo-GUI.

  1. The restart needs to happen with the control network, please follow the instructions given at Logging onto exflgateway

  2. You will also need to set-up a Karabo environment for the respective interface: Initializing a Karabo Environment for an Instrument

  3. Open an iKarabo console:

    ikarabo

  4. Get the restart device and restart the device servers:

    d = getDevice("<DEVICE_ID>")
d.restart()
waitUntil(lambda: d.state == State.ACTIVE)

    where <DEVICE_ID> is the following for the different instruments:

    Instrument <DEVICE_ID>
    SPB SPB_DET_AGIPD1M-1/CAL/RESTART
    MID MID_DET_AGIPD1M-1/CAL/RESTART
    FXE FXE_DET_LPD1M-1/CAL/RESTART
    SCS SCS_DET_DSSC1M-1/CAL/RESTART

    Note

    The SQS and HED instrument currently do not use MHz detectors and thus all calibration devices are started from the Karabo project.

    The last command will only return after the pipeline is initialized again.

  5. Now re-init the calibration pipeline

    d = getDevice("<DEVICE_ID>")
d.start_pipeline()
waitUntil(lambda: d.state == State.ACTIVE)

    where <DEVICE_ID> is the following for the different instruments:

    Instrument <DEVICE_ID>
    SPB SPB_DET_AGIPD1M-1/CAL/INIT
    MID MID_DET_AGIPD1M-1/CAL/INIT
    FXE FXE_DET_LPD1M-1/CAL/INIT
    SCS SCS_DET_DSSC1M-1/CAL/INIT

    The last command will only return after the pipeline is initialized again.

Note

If for any reasons one or the other of the above needed devices are not running, ask the instrument to instantiate them via the Karabo-GUI. They are in the online calibration subproject. You can also call CAS OCD.

After restart - push initial data

Regardless of how you restarted the pipeline, after the restart you should first leave, or instruct the instrument operator to leave the filter pulse expression and the output every nth trains at the default value. They should then have the detector push data.

The pipeline is fully initialized if all components show green state colors on the MANAGER scene. This may take up to about 30 seconds - the detector can continuously produce data during this time.

After all indicators are green, the filter expressions may be set to lower values.

Restarting only parts of the pipeline

If only individual components of the pipeline have crashed or are otherwise disfunctional, you can attempt to restart only these, instead of restarting everything. This might be considerably faster than a full restart.

  1. Stop the detector from producing data

  2. Shutdown those devices which need a restart in case they are still alive

  3. Make sure the pipeline is in an idle state: either click Reset in the MANAGER scene, or on the command line, find the calibration manager device and then use its reset method:

    d = getDevice("<DEVICE_ID>")
d.resedt()
waitUntil(lambda: d.state == State.PASSIVE)

    where <DEVICE_ID> is the following for the different instruments:

    Instrument <DEVICE_ID>
    SPB SPB_DET_AGIPD1M-1/CAL/CAL_MANAGER
    MID MID_DET_AGIPD1M-1/CAL/CAL_MANAGER
    FXE FXE_DET_LPD1M-1/CAL/CAL_MANAGER
    SCS SCS_DET_DSSC1M-1/CAL/CAL_MANAGER

  4. Now start the pipeline using either the Start pipeline button on the MANAGER scene, or via the command line

    d = getDevice("<DEVICE_ID>")
d.start_pipeline()
waitUntil(lambda: d.state == State.ACTIVE)

    As with a full restart the overall state should turn to ACTIVE once the restart is done.