Welcome to AGIPD@MID User Manual’s documentation!

Contents:

General

The AGIPD system is located in the MID hutch. More information about the detector itself you can find in this paper from February 2024:Operational experience with Adaptive Gain Integrating Pixel Detectors at European XFEL

Recording data with the agipd detector requires the surrounding infrastructure:

  • a vacuum system to pump the detector vessel
  • a cooling systems for the electronic boards outside the vacuum as well as the FEMs in the vessel.
  • power supplies
  • interlock system to ensure the detector is only cooled when the vessel is pumped and only powered up completely when it is cold.
  • DAQ system
  • Online preview and
  • Clock and Control system
_images/agipdinfrastructure.jpg

Overview of the infrastructure needed for the AGIPD-1M system.

This manual will not cover the infrastructure completely, please also refer to corresponding manuals for more information if necessary (the following list makes no claim to being complete!):

The detector should only be operated by instructed personnel. Users should not operate the detector directly at all or just do very basic things like starting and ending a run. The control system for the detector is still under development, but it allows the operator to control and monitor the most critical parameters (temperatures, pressures, voltages currents, etc..) and do the steps necessary for standard operation with some exceptions (e.g. Huber chiller for cooling of the wings).

The interlock kicks in at a pressure of 1 :math: * 10^-3 mbar. This means that the detector HV will be switched off and the detector will be warmed up.

ELOG

There is an electronic logbook for the MID-AGIPD detector: https://in.xfel.eu/elog/1+MPix+AGIPD+Detector+for+MID/

Everything done with the detector should be documented there. If you do not have access, but will use the detector, please ask Jola or Natascha for access rights.

Naming convention

The AGIPD detector has two separate hemispheres, which are in principle controlled independently. Hemisphere 1 consists of Quadrants 1 and 2 and hemisphere 2 of Quadrants 3 and 4 with modules QXm1 -QXm4 for each quadrant (see Fig. 2). Figure Fig. 2 shows the naming convention used for AGIPD detector. This convention should be used in all control systems to identify geometrical position of the detector single modules.

_images/agipd_naming.png

The naming convention for the AGIPD-1M system. Coordinate Z is the beam direction.

Note

Some control names which come from the “copy” of AGIPD Tango names are still in the different notation. It is mainly for control devices for microcontrollers (i.e. MC1 and MC2)

Below is the name conversion of the AGIPD modules from “CFEL-like”: m1-m8 to “XFEL-like”: QiMi

  • wing 1 (h1)

    m1 –> Q2M4

    m2 –> Q2M3

    m3 –> Q2M2

    m4 –> Q2M1

    m5 –> Q1M4

    m6 –> Q1M3

    m7 –> Q1M2

    m8 –> Q1M1

  • wing 2 (h2)

    m1 –> Q4M1

    m2 –> Q4M2

    m3 –> Q4M3

    m4 –> Q4M4

    m5 –> Q3M1

    m6 –> Q3M2

    m7 –> Q3M3

    m8 –> Q3M4

Figure Fig. 3 shows the preview of the 16 modules including the xfel notation of the modules and the corresponding daq nodes and HV channels.

_images/agipd_modules.png

Preview of the modules, including the xfel notation and daq nodes/ HV channels.

Rules for AGIPD detector operation at MID

General

  • The general rules for MID also apply to detector operators.
  • Users are not allowed to operate the detector, users are all persons who have not received a proper training on detector operation
  • Any intervention shall be reduced to the absolute minimum not to endanger data taking and the quality of scientific data during the beam time
  • Any modifications of detector configuration or/and operating parameters is not allowed without prior consultation of one of the following persons: run coordinator, Roman or Jola

Technical (for AGIPD operators on shift)

  • Information about the configuration will be given to Run coordinator. An example can be seen in Fig. 4.
_images/agipdconfigexample_mid.png

Example of how the the configuration table might look like. The given Data sources are mandatory for processing the raw data later on. Always check the latest version in the hutch!

..note ::
For MID the Trigger delay might vary with the distance of the detector from the sample and should be adjusted & confirmed with a timing scan if the detector is moved during the beam time.
  • Dark runs for high, medium and low gains (~50s per gain) should be taken by the operator:
    • at least twice during the shift: at the beginning and at the end of shift
    • in case detector was power cycled
    • there is any indication that performance of the detector changed or could change
  • Before power cycle the detector, please make sure that it is really needed:
    • check if DAQ is OK
    • check if control systems are OK
  • After power cycling the detector, please run detector in monitoring mode (do not store the data) at least for 20 min. It takes time before the temperatures on detector are stabilized.

At the beginning of the shift

Refer to the section Starting the AGIPD Detector at MID for details and explanation

  • Locate the printout with the configuration and operating scenarios (Fig. 4).
  • Check the Huber chiller on the roof of the Hutch: is it working, is the setpoint at the value that is displayed on the printout?
  • Check, whether the Julabo chiller shows warning 40
  • Start Karabo Gui if not open.
  • Check the vacuum pressure (in the agipd_overview_scene): is the pressure value below what is displayed as acceptable pressure printout?.
  • Check the state of the Julabo chiller (in the agipd_overview_scene): is the Setpoint of the Julabo chiller at the correct value (again the printout)?
  • Does the state of DAQ and preview look ok (no errors, crosses, red indicators)?
  • Are Electronics and Asics of the detector powered? If not: power up.
  • Check preview for the state of the modules (compare to the printout above the screens, Fig. 3 )-> Is it Ok?
  • Power High Voltage
  • Take dark runs (use the dark data procedure in the main overview scene)

At the end of the shift

  • Take dark data (one run for each gain setting)
  • Stop data taking
  • Power down HV of the detector
  • Check, whether the Julabo chiller shows warning 40

Starting the AGIPD Detector at MID

This is a guideline on how to start up the AGIPD detector at MID.

Prerequisites/First steps

Prerequisite for powering the detector is sufficiently low temperature of the detector head and electronics in the wings and therefore low pressure in the detector vessel. In order to preview and record the data the DAQ and calibration pipeline have to be in working state. In the following subsections the prerequisites for starting AGIPD and monitoring/recording data are described in more detail and what to do if they are not met (yet).

Karabo GUI

Karabo GUI should be running. If this is not the case start the Karabo GUI by opening a terminal and typing

$ ./karabo-gui

Then click on the connect button and login as admin.

AGIPD project in Karabo

The AGIPD project should be opened in the GUI. If that is not the case, load the project by clicking on the second icon from the left in the project panel (lower left of the GUI window). In the MID domain select the Agipd-master project and open it.

The main scene that you will need is the agipd_overview scene in the project Agipd_master, which can be seen in figoverview.

_images/agipd_overview_mid_03_2024.png

The agipd_overview scene includes subscenes for control as well as for monitoring. In later subsections there will be more information on how to use these. For normal user operation the main control scenes are:

  • AGIPD Cooling Control: automatic cooling procedure for in-vacuum cooling and monitoring of the chiller status
  • AGIPD Power Control: automatic power procedures and monitoring of the interlock status for the power and is described in more detail in the section Automatic power procedure
  • AGIPD Combined Control for configuration of the detector for data taking and feedback from the MFPGAs. It also shows the status of the data aggregators for the DAQ.
  • AGIPD Dark Runs: automatic procedure to take a full set of dark data.
  • Ext. housing control: where the status of the Huber chiller for cooling of AGIPDs wings is displayed.
  • Agipd Monitoring, where the temperature and vacuum status is monitored, as well as the status of the mpod devices.
  • Copy of MID Run Controller with the main components from the run controller that are needed for data taking
  • Subscene on the lower right corner where one can find links to the online preview scenes for the raw and corrected preview.

Note

Currently the vacuum system is not yet completely integrated in Karabo, therefore the devices for the valves are displayed in red as disabled even though they are running. The pressures gauges, however, are read out via Karabo and their indicators should be green. Usually they show a pressure in the \(10^{-6}\) mbar range. The Status of the Turbopump is displayed (and the icon should be green) but the speed will be displayed as zero, even if the pump is running.

Additionally there are scenes for C&C, Timing scans and for Pulse Capacitor data:

  • AGIPD C&C and Veto Control: for configuration of the Clock and Control and Veto settings, contains also a link to the Clock and Control Scene
  • Timing Scan: automated procedure to take a run, during which the trigger delay is changed.
  • Pulsed Capacitor: automated procedure to take a set of Pulsed capacitor data for a set of parameters.

Usually the configurations that are tested and controlled with these scenes are set before user operation, unless given other information just check that the Trigger delay in the clock and control configuration subscene is set according to the latest version of Fig. 4, that will be hanging in the hutch.

AGIPD Combined Control scene

This scene displays the status of the MC, MFPGAs and FPGAs of the detector see (Fig. 6) and is the interface to configure this detector. How to configure is described in the section Configure the Detector and check the modules.

_images/agipd_combined_control2020.png

AGIPD combined control: the rectangle indicators show the state of the devices for the fpgas of the detector, while the square indicators show the status of the DAQ channels.

AGIPD power Control scene

This scene is used to power the detector with the automatic power procedure (as described in Section automatic Power procedure).

In addition the scene provides information about the power and interlock status of the detector:

  • Under the log window, you can find information about the Interlock state and about the conditions necessary to power up.
  • The Indicators for ELECTRONICS on, HV on and ASICs on show, whether the correscponding parts are actually switched on (with ELECTRONICS on containing MC,ADC,AUX,FPGA and MFPGA).
If each mpod channel is either Interlocked or off the indicator All off will be green.

Pressing Check Power Status will prompt the procedure to check which mpod channels are powered and display the corresponding power status in the log window. Note that the Message Only ELECTRONICS on is also displayed if the Asics are on as well, please check the indicators that are updated when the check is performed.

Note

Checking the power status can result in an error state of the Power procedure if there is a short time failure of the communication between the mpod feedback and Karabo. As first step one should try to reset the error and recheck the power status before starting to troubleshoot.

This scene also provides a link to manual power procedure scene (for the manual power procedure check the section manual Power procedure) and an Emergency OFF button

Note

Pressing Emergency OFF will power down all mpod channels at the same time, however the High Voltage channels will still ramp down in the normal speed of 1 V/s

Huber Unichiller

The Huber Unichiller cools the part of the electronics of AGIPD that are located in the wings. In the corresponcing subscene (see (figoverview) ) you can check, that the chiller is on, the device is active and at the current temperature.

_images/huber_wing_chiller.jpg

The Huber Unichiller for cooling of the electronics outside vacuum in the wings of AGIPD.

The temperature should be set to the value that is displayed in the hutch (in the updated version of Fig. 4).

The housing temperatures should not increase above 35°C, at 40°C the power for the ASICS will be interlocked.

_images/Huber_display.jpg

The Huber chiller display.

Detector temperature

While the wings are cooled by the Huber Unichiller, the cooling blocks and FEMS are cooled by a Julabo chiller that is located above the hutch and should ideally be running continuously.

Check the detector temperatures and cooling status of the julabo chiller in the scene agipd_overview under AGIPD Cooling Control.

In the beginning of a shift usually the detector should be in the system state COLD already, in this case the Current temp. is around -32°.

Check “AGIPD Cooling Control” part in the agipd_overview scene:

  • The state of the device should be active, if not press reset.
  • status window below: Temperature within precision.
  • System state: COLD
  • The indicators for pressure in tolerable range, chiller ok and chiller state should be green.
  • Current temp. is at about -32°C.
  • Hardware region: NORMAL
  • Final Setpoint: -32°C
  • status: chiller is cooling or chiller is at SetPoint

If the detector still has to be cooled down:

  • The state of the device should be active, if not press reset.
  • Check that the indicators for pressure in tolerable range, chiller ok and chiller state are green. If chiller state is red the Julabo chiller is of and has to be switched on, see below.
  • Press Cool
  • Final Setpoint should change to -32°C.
  • system_state will change to unknown (orange)
  • Current temp might first increase by a one or two degree, then start to decrease

Note

The procedure might end in unknown state with the message that the ramping did not finish in time. Check that the temperature is at -32°C and reset the cooling procedure device.

If the Julabo still has to be switched on:

  • go to the Rackroom above the control room (if you are DET you might need to ask a colleague from MID for direction and to let you in)
  • check whether the red level indicator of the chiller is at least as high as the mark
  • first switch on the main switch of the Julabo chiller
  • wait a moment, then switch on the control box on top of the chiller, it will display “OFF”
  • go to the control room and navigate to the chiller device
  • check, whether any warnings appear. In case of warning 21 first wait, in most cases it will just disappear after a few minutes. If not try to restart again.

Note

During the cooling procedure the setpoint will be ramped down with 1°C per minute, the scene will only show the final setpoint. One can monitor the progress of the procedure by observing the Current temp which shows the bath temperature of the Julabo chiller.

  • check “AGIPD Temperature and Vacuum Monitoring” subscene:
    • External housing temperature should be below 35°C for both halves of the detector
    • Cooling blocks temperature: should be around -25°C in case Asics ar ON and - 27 C in case Asics are OFF
    • FEM(LTCC) Temperatures: should be in range of -5 to 16°C when ASIC power is ON, in case ASIC power is off it will show temperatures below -25 C. please note , that LTCC temp. can be readout only if microcontroler and vacuum boards are powered ON. It means, if LV power is OFF you do not have access to the LTCC temp. readout and ‘nans’ are displayed in the scene.

Vacuum pressure

Verify the pressure in the “AGIPD Temperature and Vacuum Monitoring” part of the agipd_overview scene. Along with temperature this part depicts the status of the vacuum components, that are relevant for the vacuum in the AGIPD vessel:

  • pressure sensors at the vacuum vessel (P_FR_det and PG2_det) with the measured pressure
  • (currently disabled) Turbopump attached to the Vessel (Turbo_12), with the status and rotation speed
  • (currently disabled) two valves and and two pressure gauges the pre-vacuum system

Pressure P_FR_det should be in the \(10^{-6}\) mbar range. .. At \(1*10^{-3}\) mbar the interlock will be triggered, the voltages will be ramped down and the middlelayer device of the chiller will start a rapid warm up procedure. .. The turbopump should be started, the prevacuum pressure is usually below 0.1 mbar, all symbols for the vacuum components should be green. .. Indications of trouble with the vacuum system might be:

The interlock will trip at a pressure of \(1*10^{-3}\) mbar, the voltages will be ramped down and the detector will be warmed up. In case there is a vacuum problem that cannot be resolved quickly by MID one can already power down the detector and start warming up. In order to warm it up fast, go to the chiller device MID_EXP_CHL/DCTRL/CHILLER and set Chiller Temperature Setpoint to 20°.

DAQ

In the subscene Copy of MID Run Controller one can choose the proposal (according to the printout in the hutch), change run types and samples and start and stop runs. All indicators in the scene should be green (ready to take data), the big indicator corresponds to the state of the run controller, AGIPD1MCTRL/0 and AGIPD1MCTRL/1 correspond to the slow data of AGIPD. AGIPD1MCTRL/0 contains the configuration and voltages, AGIPD1MCTRL/1 contains temperatures and timing information. Make sure that these slow data sources are included, they are mandatory for data processing. If the DAQ is not monitoring yet, always press ‘apply configuration’ first, then ‘monitoring’. If symbols indicate errors (red) or errors appear when trying to apply the configuration contact ITDM (phone numbers can be found in the hutch).

Start up prodecure for the detector

The detector is powered via MPOD channels that have to be switched on in a defined order. There are the following ways to power the detector that will be described in the corresponding subsections in more detail:

  • with the automatic procedure (in one step or three)
  • manually, where each step has to be done by the operator. This procedure allows to power only parts of the detector if necessary and can always be used if the automatic procedure gets stuck.

Note

Currently the fully automatic procedure is recomended.

For each procedure it is recomended to monitor the status of the MPOD channels. To do this open the power supply (MPOD) feedback display for the hemispheres H1 and H2 and the high voltage HV (see Fig. 9) in a browser window:

H1: http://mid-exp-agipd1m-mcps-h1-lv.desy.de/

H2: http://mid-exp-agipd1m-mcps-h2-lv.desy.de/

HV: http://mid-exp-agipd1m-mcps-hv.desy.de/

On the gui pc there are also bookmarks to the links.

_images/agipd-power_23.02.png

The MPOD feedback displays in browser windows, the upper two windows are for the LV channels of H1 and H2 respectively, the third is for the HV channels. This screenshot is from 23.02.19. Check the Elog for the most recent screenshot!

For each procedure it is also recomended to make sure that all FPGA devices are instantiated and without error. For this check the status indicators of the devices in the subscene Agipd combined control.

Configure the Detector

The detector can already be configured once the Electronics are on.

In the subscene AGIPD combined Control the following parameters should be set:

  • Pattern template: pattern_standard - this is the current standard pattern

  • Pattern type:

    • Xray: for taking Xray data
    • Dark HG: used to take darks for the high gain stage, essentially the same as the Xray pattern type
    • Dark MG: used to take darks, while the detector is forced to the medium gain stage
    • Dark LG: used to take darks, while the detector is forced to the low gain stage
    • PC: only for pulsed capacitor data (which is usually rather taken with the automatic procedure)
    • CS: only for current source
    • Gain Mode:
    • ADAPTIVE: normal operation mode, gain will be adapted
    • FIXED_HG: fixed in high gain
    • FIXED_MG: fixed in medium gain
    • FIXED_LG: fixed in low gain

Note

the Parameters Pattern type and Gain Mode are not independent. Fixed gain modes are to be used with Pattern type Xray. The controls will only display those options of the Gain mode, that are applicable with the current Pattern type.

  • Gain Settings:

    • 0 for standard operation
    • 1 for operation at very low intensities (0.5 photons/pixel!)
  • Integration Time (in clocks):
    • 12 is the standard integration time and the minimum. Integration times of more than 12 have to be agreed with the detector experts.

  • # Pulses: sets the number of memory cells. In theory this can be set to any value up to 352, but there are only specific values for which calibration data exists (64, 202, 352). Please check the print out in the hutch or confirm with the run coordinator

  • Repetition rate: should be set according to the pulse rate of the FEL, but again not for all parameters calibration data might be available.

  • Aquisition time: can be set up to ~ 60000s

Note

When the repetition rate of 0.5 MHz is set the detector will sent the maximum of 338 memory cells, if Mem cells are set to more than that.

To configure the detector:
  • Go to the AGIPD combined Control part of the agipd_overview scene.
  • Check if the the log windows of the MFPGAs of Hemisphere 1 and 2 are updating. When the detector is not sending data they should repeat ‘Null Stop’ with a time stamp that should increase every second.
  • Choose the parameters you want to apply (see description above)
  • Make sure to hit enter, then click Configure Detector. In the log windows of the MFPGAs the message “loaded bunch structure file” will appear briefly before it displays “null stop” again.
  • Click on the Start sending button underneath. The button should become inactive and the (m)fpga devices will display the state “Aquiring”. In the log windows of the MFPGAs (that previously said ‘Null Stop’ ) the messages ‘Running’ and ‘Stopped at [some number]’ should appear alternately.

Check in the online preview (see previous section), whether all of the modules appear and if the modules are in good shape. For comparison a picture of the current expected status (an updated version of Fig. 3) is hanging above the monitors of the gui pc. If there are more hot regions on the modules than in the printout, or there are modules missing from the preview go to the troubleshooting section.

Manual power procedure (not recomended, if you can use the automatic procedure instead)

This procedure can be used in case the automatic power procedure fails. Open the manual_power scene, the link can be found at the bottom of the AGIPD Power Control Check that all interlock devices are available, i.e. the indicators are either violet or green. Orange or red indicate problems on the PLC/Beckhoff level and you should contact MID.

The powering sequence

For the powering sequence the Manual Power scene is used. It is also described in bullet points in the lower part of the scene.

./images/screenshots/manual_power_scene_03_2024.png

With the field ‘Tag filter’ of the scene one can choose which channels should be powered up. Once the tag is applied and one clicks on ‘On’ all the channels will power up, that have the corresponding tag assigned to them.

You can combine tags with “|”, this stands for ‘and’. So in case of the tag “mc|efan” channels with tag mc (microcontroller on channel 901) and with tag efan (ventilators on channel 900) will be powered on each half of the detector. When adding a tag behind a comma, you can select for that tag, for example ‘mc,h1’ will only switch on the channel 901 for the microcontroller on the wing H1, while the one on H2 will stay off. In the following the necessary tags for the power procedure are given for each step.

  1. Power efans and microcontroller (MC). First, make sure the interlock is armed (green). If it is blue, arm the MC interlock by clicking on ‘arm’:

    'MC and EFAN' interlock => arm

    Then, in the right tag filter field write mc|efan (the | stands for &) and acknowledge with enter. Then click ‘On’ in ‘Switch Channels’:

    Tag filter = `mc|efan`   Switch channels = press On

    Monitor the browser windows for the low voltage of H1 and H2, the channels 900 and 901 should power up on both.

  2. Wait until the microcontroller fields on the right of the scene will become online showing NORMAL:

    MC1,2: State='NORMAL'==> AUTOMATIC=1 for micro-Controller 1,2

    Set AUTOMATIC to 1 for both MCs. A current rise in the channels U900 an U901 should be observed. Wait until both display OPMODE=3. One will take longer than the other:

    MC1,2: OPMODE=3 ==> continue

  3. Now, arm all blue interlocks (MFPGA, FPGA, ADC, AUX) by clicking on them:

    'MFPGA' interlock => arm
'FPGA' interlock => arm
'ADC' interlock => arm
'AUX' interlock => arm

    All interlocks except ASIC and BIAS should now be in a green state. All channels below U500 and U0 - U4 should go from state ‘interlocked’ to ‘off’.

  4. Power the master FPGA by setting mfpga in the tag filter field and then switching it on:

    Tag filter = `mfpga`   Switch channels = press On

    Channel U904 is powered.

  5. Power the ADCs and AUX channels by setting ‘adc|aux’ in the tag filter field and clicking on:

    Tag filter = `adc|aux`   Switch channels = press On

    Channels U500-U607 (adcs) and U800 - U807 (aux) are powered. The status of the voltages in the two big MPOD browser windows should be symmetric. If it is not, try repeating this step, if the problem remains, call experts.

  6. Tell the MCs that the AUX channels are on, by setting the two fields to 1. Now wait until OPMODE=7 on both MCs:

    AUX=1 for micro-Controller 1,2
MC1,2: OPMODE=7 ==> continue

  7. Switch the FPGAs on by setting ‘fpga’ in tag filter and clicking on. Wait 1 minute until ADCs have been tuned. During the waiting time you can continue with steps 8 and 9 though:

    Tag filter = `fpga`   Switch channels = press On

    Channels U700-U703 and U0 - U4 are powered.

  8. Configure the detector

  9. Arm the ASIC interlock, so it is in the green state and the state in the browser goes to ‘off’ for channels U100 - U407:

    'ASIC' interlock => arm

  10. Power the ASICs by setting the tag filter to asic and clicking on:

    Tag filter = `asic`   Switch channels = press On

    Channels U100 - U407 are powered. If it is not symmetric (see above), redo. Push data immediately by pressing ‘send data’ and check the online preview as described above. If everything looks ok, you can directly proceed to the next step.

  11. Arm the HV interlock so that it is green. In addition, clear the interlocks by clicking on the button:

    'BIAS' interlock => arm
clear interlock

  12. Power the HV , by changing the tag filter to bias and clicking on ‘on’.

    Tag filter = bias Switch channels = press On

    Wait for the channels to fully ramp up and change to status ‘ON’ before proceeding to take data

Dark data

At the start of the shift as well as when the detector had to be powercycled a set of dark frames should be recorded. Confirm with the MID staff that the detector is in the dark (eg shutter is closed or there is no beam).

One can take a set of dark data either manually or with the automatic procedure. The automatic procedure is recommended, unless there are any issues (for example, when it gets stuck repeatedly) In the following, both procedures are described. In both cases the recorded data has to be checked, this is described in the section Data check.

Automatic procedure to take dark data

Go to the subscene AGIPD Dark Runs, click on Update Scenarios and options, this will ensure that the scenarios and Run and sample types that are available in the current Proposal will be displayed in the drop down menus and you can choose correctly. If you stick with the Proposal and there are no changes to the Proposal or Scenarios you just have to do this once.

  • Choose the correct values for Gain Setting, # Pulses, and Repetition rate, for the Aquisition time 50s is sufficient for darks.
  • Choose the correct Run and sample types for each run (the names of these will depend on the proposal, if it is not obvious consult the run coordinator).
  • Press the button Take dark runs
  • Check the data as described in the corresponding section Data check

Note

In case you cancel the procedure in mid-run please make sure that the the run is also stopped. Otherwise a lot of empty files will be produced.

Manual procedure to take dark data

In order to record data, use the part Copy of MID Run Controller in the agipd_overview scene.

Choose the values for pattern template, # pulses, repetition rate and Gain Setting. For the Aquisition time 50s is sufficient for darks.

Take one run these parameters and for each of the three pattern types Dark HG, Dark MG and Dark LG.

The Run Type and Sample have to be set for each run individually, for example sample type ‘no sample’ and Run Type ‘Calibration - Dark HG’ for the high gain dark run. To change Run Type and Sample, choose the correct one for each from the drop down menu of the “Copy of the MID Run Controller” and hit enter.

To take a run:

  • Set the DAQ in monitoring (if it is not already) state by hitting apply configuration (check that the configure indicator is green) and monitor data (which should also turn the Monitor indicator green).
  • Choose the correct Parameters and press Configure Detector and check in the mfpga log window below, the message “bunch structure loaded” should appear in both
  • click Start Sending in AGIPD Combined Control to send data and start the Run by clicking on Start Run in the Copy of MID Run Controller
  • When the detector stopped sending data (indicated by the Stop Sending button being greyed out and the fpga devices stop showing “Aquiring”) Press Stop Run in the Copy of MID Run Controller.
  • Check that the data is in the folder: Open a terminal (or use the open one, it is usually there) and navigate to the runfolder. Go to the run folder on the online cluster ( , path: ‘/gpfs/exfel/exp/MID/2018[..]/p[proposelnumber]/raw/r[runnumber]’). h5 files should be produced for each module (‘RAW-R[runnumber]-AGIPD[modulenumber-s00[filenumber].hf’) as well as a files for slow data AGIPD1MCTRL00 and (‘RAW-R[runnumber]-AGIPD1MCTRL00-s00[filenumber].hf’, which contains the configuration and voltages and ‘RAW-R[runnumber]-AGIPD1MCTRL01-s00[filenumber].hf’, which contains temperatures and timing information). If there are modules for which only empty files produced, there might be a problem with the daq or these modules might not be sending data, please go to the Troubleshooting section for further investigation of the problem.

As soon as the Run is started in the proposal folder a folder with the run number is generated. Make sure that there are files for each module and that they have similar sizes. Every 250 trains a new set of h5 files is produced. If the run is started but the detector does not start immediately the first files of the modules can be empty, also if the run is not stopped soon after the detector stops sending data empty files for the modules are produced. In order to minimize the amount of empty data and if it is not required to record a specific number of trains one can also set the number of trains to up to ~60000, start the detector and then start and stop the runs, while the detector keeps sending data.

Data check (in case the lit frame finder is NOT used)

How to check the data in the h5 files:
  • First run the command ‘load module xray’, then open the file you want to inspect with ‘hdfview filename.h5’ The hdf viewer will open in a new window.
  • In the left window, which displays the file structure, navigate to: INSTRUMENT -> MID_DET_AGIPD1M-1-> [modulenumber]:xtdf->data’
  • right click on data and choose ‘open as’, in the window that opens choose for ‘Display as’ ‘Image’, set the Palette to rainbow. Set the dimensions to: Height=3, Width=2, Depth=0 (see also Fig. 11)
  • click ok and observe the image. The values should be around 8000.
  • go to the file structure again and navigate to: INSTRUMENT -> MID_DET_AGIPD1M-1-> [modulenumber]:xtdf->cellid’ and display the table. The cellids should count up to the cell number in the name of the scenario that is used. If there are jumps in the cellID number, or if the cell ID number 370 appears, the configuration has not been correctly applied - the run is bad and must be taken again
  • navigate to: ‘pulseid’, if the scenario for 1MHz is used the pulse id should be 0,4,8…, if the scenario for 0.5MHz is used the pulse ids should be 0,8,16…
_images/Screenshot_h5view.png

The hdf5 viewer

Timing Scan (to be updated, contact one of the experts if a new timing scan is necessary)

The Performance of the AGIPD detector is very sensitive to timing. To confirm or find the correct delay for the detector a timing scan has to be carried out and the resulting data has to be investigated. For this a run is recorded that includes data with a series of delays. From the curve of the mean signal in dependency to the delay the optimum value for the delay can be obtained. The value of the delay is given in the number of AGIPD clocks, 24 clocks amount to 220ns

There is a macro for the timing scan, which will change the delay for the given amounts of steps and step sizes and take one run to collect data while the delay increases. However, in order for the scan to be useful some preconditions have to be met. In case there is no other information given these are:

  1. Sample: preferably a copper target that produces flat field data, if this is not available a Lithium Titanate or another sample that produces rings (which is less convenient to analise)
  2. Run type “Calibration – Timing Scan”, Sample: “Cu”/“LiTi2O3“
  3. It has to be determined which is the suspected ideal delay (by calculation or “experimentally”) and verified roughly by observation of the preview. Check the elog for previous scans under similar conditions in the hutch and consult with the instrument what changes have been made since the last scan. During the initial search it is recommended to use rather more attenuation.
  4. A clear signal is needed: the intensity should be high, but in the ROI that will be analized later no gain switchig should occur.
  5. A stable beam and XGM data have to be available: since the mean signal will fluctuate with the intensity of the incoming beam the scan might be not useful if the beam is fluctuating too strong. In any case the XGM data for the beam intensity should be available in the data set to allow for correction.
  6. Pulse pattern from the machine: At least one pulse of X-rays is needed, the ideal pulse patterns contains X-rays in every 8th pulse to cover each memory cell row (while reducing the amount of radiation on the detector to what is necessary

In the AGIPD_overview scene is a sub scene for the timing scan.

Before executing the scan choose the parameters:

  1. The start delay will be the delay of the first trains that are recorded.
  2. Step size should be one.
  3. Aquisition time per step can be 3 to 5s
  4. The Number of steps should be high enough that it contains all delays for which signal is observed plus additional ~ 5 delay steps before and after the signal (+/- 16 around the value, that is suspected to be the correct one). Please verify this by observing the preview during the scan.

In principle not a full set of modules is necessary for the timing scan, so in case data aggregators stop working the scan does not have to be cancelled as long as there is still data from modules on which there is a good signal. This is required for at least one module per detector half.

To make a judgement on the correct value for the delay, corrections should be applied given by the dark runs (offset correction) and by the XGM data to correct for instabilities of the beam. The region of interest, that is used for the analysis should contain a signal that is well above the noise (> 500 ADU) but still low enough that no gain switch occurs . The XGM-corrected mean signal level in this region of interest is plotted against the trigger delay for different memory cells.

The image shows an example for a timing scan:

_images/timing_scan_example.PNG

Example of timing scan result for one module and with signal in one memory cell

The ideal delay value is the one before the drop of the signal. In a second step it can be verified that the gain in the regions where gain switching occurs is not negative (this would be an indication that the value for the delay is too high).

The evaluation of the delay should be done for both halves of the detector!

For the evaluation a notebook is provided by DA, it can be found on Gitlab: <https://git.xfel.eu/gitlab/dataAnalysis/calibration-services#how-to-use-at-xfel> The full and most recent instructions for installation and use of the notebooks are also available in the readme: https://git.xfel.eu/gitlab/dataAnalysis/calibration-services/blob/dev/README.md

During beamtime it is convenient to run the notebooks on the online cluster to avoid having to wait for migration to the maxwell cluster

  • Get the repository in your workspace on the online cluster: user@machine git clone https://git.xfel.eu/gitlab/dataAnalysis/calibration-services.git Notebooks are available in the folder calibration-services/notebooks
  • Running a jupyter server on the online cluster via FastX: Open a terminal and type the following commands
    • user@max-display: ssh exflgateway
    • ssh exflonc<NN> #NN = 12 for eg.
    • module load exfel calibration-services
    • jupyter-notebook - -port 8008 - -no-browser This will start a jupyter server on online cluster and a link will be provided that you can copy.
    • Now open another terminal and do the port forwarding: user@max-display: ssh -L 8008:localhost:8008 exflgateway -t ssh -L 8008:localhost:8008 exflonc<NN>
    • Open your regular browser and paste the link that you copied before the last step. Provided you have cloned the repository as shown in first step, the calibration-services folder will be visible in the jupyter notebook.
  • From the hutch control room computers: Open a terminal and type the following commands
    • ssh exflonc<NN>
    • module load exfel calibration-services
    • jupyter-notebook - -port 8008 - -no-browser
    • Open another terminal and do the port forwarding
    • ssh -L 8008:localhost:8008 exflonc<NN>
    • Open your regular browser and paste the link that you copied before the last step.

Stopping the AGIPD Detector at MID

This is a guideline on how to stop and power down the AGIPD detector at MID.

Data taking should be stopped for doing this procedure.

Switching off sequence

Depending on what are the next plans with the detector, not all these steps are necessary. Please check with the run coordinator from MID (for example whether warming up of the detector is necessary).

Powering down the detector

The detector can be powered down with the automatic procedure or manually.

Powering down the detector with the automatic procedure

  • One Step procedure:

    • Go to the AGIPD power Control part of the agipd_overview scene and press Detector Shutdown
    • When the procedure is finished, the indicator for`All off` should be green and all channels in the browser feedback (Fig. 9) either ‘Off’ or ‘Interlocked’

  • Procedure in three steps:

    • Go to the AGIPD Power part of the scnene agipd_overview and press the button ‘HV Off’. When the procedure is finished the indicator for HV on should be blue and all HV channels in the browser feedback should be off.
    • Press ASICS off, the Asics on indicator should turn blue, and the channels U100 - U407 off
    • press Elecronics Off, when this procedure is finished,`Electronics on` should be blue, All off should be green and all channels should be off or interlocked

Note

Emergency off: in case the power procedure results in an error state and the normal control buttons are not available the Emergency off button can be pressed, which will prompt all power channels (low and high voltage) to power down. In case this includes the HV channels this will still take ~ 5 minutes to ramp down. When all channels in the browser feedback ar off or interlocked the device can be resetted. Another Check of the power status should result in the indicator for “All off” to be green.

Powering down the detector manually

In case the automatic procedures do not work the detector can always be switched off manually: There is a guideline in the lower right part of the manual power scene regarding how to power down the detector. This scene is also used for the powering down.

  1. Using the manual power scene switch off the high voltage:

    Tag filter = bias    Switch channels = press Off
wait for ramping

  2. Using the manual power scene switch off ASIC power:

    Tag filter = asic   Switch channels = press Off

  3. Using the manual power scene switch off FPGA:

    Tag filter = fpga   Switch channels = press Off

  4. Using the manual power scene switch off the remaining powers:

    Tag filter = adc|aux|mfpga|mc|efan   Switch channels = press Off

A mix of both methods is also possible, e.g. switching off the HV and asics manually (step 1. and 2. from the manual procedure) and using the automatic procedure to switch off the Electronics.

Switching off Huber Unichiller

The Huber chiller can be switched off by pressing the ‘off’ button on the ‘external housing cooling control’ subscene of the Agipd overview scene

Note

Usually it is not necessary to switch off the Unichiller at the end of the shift, unless this is necessary for maintenance.

Warm up detector

This only needs to be done when the detector will not be operated for a long time and/or if maintenance needs to be done which requires access to the detector. Please check with thr run coordinator if warming up the detector is necessary.

To warm the detector
  • make sure the detector is powered down.
  • go to the subscene AGIPD Cooling Control.
  • press heat in the subscene, the Final Setpoint should change to +20 degC and the Current temp. should slowly increase by approximately 1 degC per minute.
  • when the procedure is finished the system state is warm.

Troubleshooting

This section mentions known problems and how to react to them

Cooling issues

Cooling procedure does not start

  • Check state of the Cooling procedure, if it is ‘unknown” try to reset
  • Check whether cooling is allowed, if it is interlocked go to Interlock issues
  • Check the status of the Julabo chiller (is “Chiller ok” green in the cooling control subscene). In case of an issue check the Julabo chiller in the rack room and follow the manual for the Julabo

Detector does not cool down

If one or more of the cooling Blocks do not get cooled down but get stuck at around 0 degC, the reason might be that ice was formed due to water in the cooling circuit. Please inform users, it might be necessary to heat up the circuit to +45 degC to get rid of the water.

Power issues

Channels do not power up (or down)

If the channels do not power up (or down) even when using the manual procedure:

  • Go to AGIPD_POWER subproject and check the status of the corresponding HV or LV device ( MID_EXP_AGIPD1M/PSC//*)

    • if the device is shut down: reinstantiate

    • if the device is in error: try to reset the error or shutdown and reinstantiate the device

    • if the device is instantiated and without error:

      • navigate to Channels (last in the list) and unfold the channels to check whether the information for “Tags” is there (as can be seen in Fig. 13, check for the first channel and the last, where voltage should be applied)
      • if there is no tag information try to reinstantiate the device

If none of this works call DOC/Controls OCD (the power xml configuration file is not acccessible)

_images/Screenshot_tags_.png

Channel 901 tripped, ‘sense voltage too high’, power of the detector is down.

Channel 901 is the channel for the microcontroller, if this trips the wing that is affected will power down to the point where only 900 is on, 901 displays ‘sense voltage too high’ in red, channels 902 and 903 will be off, all other will be interlocked.

If this happens, power cycle the detector:

  1. Switch off by clicking ‘Emergency off’ in the subscene AGIPD Power.

  2. Check the MPOD feedback in the browser, make sure that all channels are either ‘off’ or ‘interlocked’.

    If not: press ‘Emergency off’ again.

  3. Power the detector as described in the corresponding section.

ASIC power channel trips

It can happen that one or two channels for ASIC power trips. It was observed on channel U302 for wing 2.

Do not panic! It is not necessary to power down the whole detector!

  1. Stop detector ( in case is sending data)
  2. Check in the corresponding configuration table ( displayed in the scenes power_lv_h1 or power_lv_h2) - the “tags” (i.e. q,m..) for this channel
  3. Go to the manual_power scene to power down just the asic of the affected module. (For example channel U302 for half 2 has the following tags “asic,h2,q4,m1,v2”)
  4. For the module QxMy you should put in the Tag field “asic,qx,my” then click ‘OFF’
  5. power up the module
  6. Start detector to properly initialise the ASICs
  7. Check if the image quality in the online preview still looks fine.

Start up issues

Failure of Automatic power procedure

Check how far the procedure got and which channels are already powered

  • If no channels are powered, or if you observe any tripped channels go to Power issue
  • You can try to go on with the manual power procedure
  • Or give the automatic procedure another try:
    • Reinstantiate the power procedure device
    • Press ‘Emergency Off’ and verify that all channels are off or interlocked in the mpod browser feedback
    • follow the automatic power procedure

Detector is not sending data

  • Check, whether the fpga devices show the state ‘Aquiring’ in the combined control scene. If not or only half of them are in the ‘Aquiring’ state, as a very first check stop sending data, configure again (check that you choose operation parameters that are known to work) and wait a second before pressing the ‘start sending’ buttons.
  • Check with Rundeck whether it is actually the detector that is not sending data, see Verify, that detector is not sending data
  • After confirming that the issue is on the detector side further troubleshooting depends on whether a complete half of the detector is affected or single modules.

Modules are not sending data

If at least one module from each half of the detector is sending data, the possibility that the reason is the configuration or the c&c signal is already ruled out:

  • try to power cycling the system.

H1 and/or H2 are not sending data

  • Reconfigure the detector (check that you choose operation parameters that are known to work) to verify that there is no issue with the configuration, then try sending data again
  • Check the MFPGA log windows for unusual feedback:
  • If all else fails: try power cycling
MFPGA feedback is not updating

One of the log windows in the AGIPD combined control subscene (that say ‘Null Stop’or ‘Stopped at…’ with a timestamp) are not updating each second.

First check that the detector is not in the process of applying the configuration or number of trains (the updates will stop temporarily during this process). If you are sure that this is not the case:

  1. shutdown and re-instantiate FPGA devices from cppSPB/AGIPD server
  2. Press ‘initialize Asics’ in the combined control scene
  3. Configure detector, wait till state of the composite is Active.

If it does not help, please power cycle the system

Detector keeps “Running” when stopped

The update window of the MFPGA (can be only one wing or both) keeps updating after data starts to be sent but only updates “Running” instead of changing to “Stopped at [some number]” now and then.

  1. Stop the data taking.
  2. If the MFPGA of the affected half will not go back to “Null Stop” but keep on updating “Running”, power down the asics in the manual_power scene by setting the Tag to ‘asic’, hitting enter and then clicking on ‘off’.
  3. When the asics are off, try to resubmit the configuration (possibly twice).
    • Stop the affected half by navigating to the device in the GUI and press “Stop”
    • If that works and the detector starts updating ‘Null Stop’, try again to push data and to stop again
    • If the detector behaves normalpower up asics again and check the preview.

If resubmitting the configuration does not work power cycling is necessary.

MFPGA feedback only updates “Stopped at 66” and no data is send

If one of the MFPGA log window keeps displaying “Stopped at 66” and no data is sent from the detector:

Stop the detector and check whether the display changes:

  • If “Null Stop” is displayed: try to reconfigure and start again.
  • If the log window keeps on displaying “Stopped at 66” and stops, when “start” is pressed, there might be an issue with the clock and control system:
    • Check, whether the c&c cables are connected
    • Confirm with AE that there is no issue with the C & C crate

If no issue can be identified, try power cycling.

Verify with Rundeck, that the detector is not sending data

In order to check, whether the fault is already on detector side one can use the tool Rundeck.

  1. Find the link ( remcom.xfel.eu/project/SPBDAQ ) to the login page of the tool bookmarked in the upper bar of the browser of the gui pc (look for ‘Rundeck login’).
The username to login is SPBDAQ, the name and password are saved.
  1. Make sure that the detector is sending data and start a new job for ‘DAQ Data Aggregators Network Traffic’ for this select all the nodes that belong to module data and click ‘Run Job Now’.
The nodes spb-br-sys-daq-00 to -15 belong to the 16 modules, the allocation of the nodes to the modules is shown in Figure Fig. 3 in the introduction (and also hangs out in the hutch). The nodes spb-br-sys-daq-da01 and 02 belong to the slow data.
  1. Observe while the job runs: all nodes will be listed, and disappear one after another from he list. If nodes stay exceptionally long, this is already an indication that there is a problem with the data from the corresponding module.
  2. When the Job is finished go to ‘report’. Check the reports from the suspicious nodes whether packages were sent.

Possible results:

  • If all module sent data packages to the correct IP the problem is in all likelyhood on the daq side, please consult ITDM.
  • If no data is coming from a complete half of the detector (or both) go on troubleshooting at H1 and/or H2 are not sending data
  • If there is one or more modules that stay in the list and the report says ‘0 packages captured’ but there is still data from some module of each half, try to power cycle the system.

Issues with data taking procedures

Dark Data procedure goes to error state or gets stuck

If starting the Dark Data procedure results in an error of the procedure or the procedure gets stuck, please check that the conditions for the procedure are met:

  • Please stop the detector before starting the procedure
  • Do the Run and Sample Types exist in the proposal that was set?
  • Where Scenarios and Options updated in case the proposal was changed?
  • Make sure that the DAQ_Controller state is not in error or unknown, if necessary re-instantiate the device

Reset the Dark Data procedure with the reset button on the scene, check that the conditions above are met and try again. In case it is not possible to resolve the issues one can take the darks manually as described in “Manual procedure to take dark data”

Issues with applying configurations

Combined Control and/or MFPGA devices are in error

If the MFPGA devices and composite went into error state (this can for example happen if the configure button gets clicked accidentally twice in quick succession) go to subproject AGIPD_CONTROL and “reset error” in the MFPGA devices (FPGA/MASTER_H1 and FPGA/MASTER_H2), then try to configure again. If the FPGA_COMP (subproject AGIPD_CONTROL_MDL) goes into error again upon trying to reconfigure: reset the device and try initializing the asics again ( with the respective buttons on the scene) then reconfiguring should work again. In some cases resetting the error on FPGA_COMP has to be repeated.

Image Performance Issues

Online preview is not updating

If the online preview does not update, there is a variety of possible reasons, and several things can be checked:

  • The detector does not send data - this can be checked with Rundeck, see the last section
  • Make sure DAQ is monitoring (indicators next to Monitor and Configuration in the Copy of SPB Run Controller are green - if not click apply configuration and Monitor
  • The detector is not configured correctly. If the Detector was not configured correctly the online preview might not be able to use the data that arrives from the detector - Check, whether the configuration scenario is one of those displayed in the hutch as up to date, reapply the configuration and set the number of trains, then try again to push data.
  • Issues with the DAQ that affect the preview - You can cross check by taking data and checking whether image data for the modules has sufficient size to make sure to rule out there is a problem with the DAQ

If all of this issues are cleared please refer to the online calibration troubleshooting guide for trouble shooting and a description for restarting the calibration pipeline

Corrected image performance is not as expected

In case any features are observed on the corrected preview please verify that they are observed on the raw preview as well. For reference what the raw preview is expected to look like check the printout in the hutch.

  • If the raw preview looks as expected:
    • check that the parameters for the calibration constants are correct for the current operation scenario
    • refer to the online calibration troubleshooting guide

Mosaic like image appears on the preview

If the detector is sending data and the preview of looks like to the one in figure Fig. 14 there was an issue with applying configurations. This can be resolved by power cycling the detector.

_images/mosaicagipdmid.png

Preview in case of misconfiguration

Interlock issues

In case the interlock is triggered first verify the status of the of the vacuum and cooling system (in the AGIPD overview scene):

  • check the vacuum system, if it is not ok:
    • make sure the detector is warming up
    • switch OFF the detector power
    • restore vacuum as soon as possible
  • check temperatures of cooling blocks and status of Julabo chiller, if they are not ok:
    • check the Julabo Chiller in the rack room and follow the Julabo manual
  • check temperatures of external housing and the status of the K3 chiller in the hutch, if they are not ok:
    • check the K3 chiller in the hutch and follow the K3 manual

Warning

If the detector was accidentally vented (>1 mbar): confirm with DET before cooling down again and powering.

Interlocks cannot be armed

When the interlocks cannot be armed even though all prerequisites are met, please go to the agipd_interlock_control subproject, shutdown the device servers under ‘mdlSPB/AGIPD’ and instantiate them again.

MC live signal tripped

In case of trip of MC live signal the microcontroller and efans will still be on all other channels will be in interlocked state. If there is no issue with with vacuum or cooling (see above): power cycle the detector. It will be necessary to either power down manually or with the ‘Emergency off’ button.

Interlock triggered by pressure spike

In case Detector was warmed up due to interlock trip caused by a short(!) pressure spike (i.e. due to injection of sample):

Warning

This only applies if the pressure did not rise above 1 mbar and the vacuum was restored immediately. Otherwise this has to be treated as interlock issue

  • If the detector was warmed up above -5 degC (i.e. temp. of cooling blocks > -5degC): switch OFF the detector power
  • Make sure that the vacuum condition are fine, i.e. the state in SPB_IRU_CHL/DCTRL/CHILLER device is not INTERLOCKED.
  • Press reset in the automatic procedure.
  • Try to cool down the detector with the automatic procedure. For any issues that occur while cooling down refer to the section cooling issues

Warning

Please do NOT try to cool down the detector if the state of the chiller is INTERLOCKED.

Grafana Panel for the DOC shifters

People who work on the DOC shift will notice an issue or start an investigation on the AGIPD monitoring Grafana Panel ( https://ctrend.xfel.eu/d/wfNl2EYGz/doc-agipd-monitoring?orgId=1&refresh=5s ) and go on from there. For Documentation about the panel also refer to https://redmine.xfel.eu//projects/data-operation-center/wiki/AGIPD_Monitoting , this link is also found on top of the Grafana Panel.

Please also take note of the OCD guidelines in this manual: https://rtd.xfel.eu/docs/agipd-manual-mid/en/latest/OCD_guidelines.html

Figure Fig. 15 shows the AGIPD Monitoring Panel of Grafana (without trendlines) and in Fig. 16 it is depicted which panel corresponds to which indicator in the main overview scene of AGIPD.

_images/grafanapanelspb.png

DOC AGIPD Monitoring Grafana Panel

_images/grafanatokarabomid.png

How to find the information in Grafana in the main scene of AGIPD

Grafana alerts

SPB chiller alert
  • Will go red if the bath temperature deviates more than 2 degree from the nominal value of -32°
  • to check in the AGIPD main Scene in Karabo: Subscene ‘In-Vacuum Cooling Control’ and ‘AGIPD Monitoring’ subscene in case of issues with the pressure in the vacuum vessel.
  • Possibly helpful troubleshooting sections: Cooling Issues and Interlock Issues
SPB AGIPD chiller alert
  • will go red if the bath temperature deviates more than 2 degree from the nominal value of -32°
  • to check in the AGIPD main Scene in Karabo: Subscene ‘In-Vacuum Cooling Control’ and ‘AGIPD Monitoring’ subscene in case of issues with the pressure in the vacuum vessel.
  • Possibly helpful troubleshooting sections: Cooling Issues and Interlock Issues
SPB AGIPD Vacuum alert
  • will go red if the pressure rises above 10^-5 mbar
  • to check in the main scene: subscene ‘AGIPD Monitoring’ shows the the pressure in the Vessel (P_FR_det) as well as the states of other components of the vacuum system which might give a hint for the reason, in any case communication to the instrument is necessary
  • Possibly helpful troubleshootimg sections: Interlock Issues
SPB AGIPD Power alert
  • will go red if the automatic power procedure is in error state
  • to check in the main scene: in subscene ‘AGIPD Power Control’, as first step the automatic procedure device can be reset and the status checked with the ‘Check Power Status’ button.
  • Possibly helpful troubleshootimg sections: Start up Issues
SPB AGIPD INTERLOCK Overview alert
  • will go red if any of the interlock devices that correspond to the different electronic parts of AGIPD prevent them from being powered. Since for different electronic parts different conditions have to be met there could be a range or reasons
  • to check in the main scene: in subscene ‘AGIPD Power Control’ check under ‘Interlock summary’ which part of the AGIPD is interlocked, the indicators next to it (Pressure, Temps etc) give an indication for the reason, as usual check the ‘AGIPD Monitoring’ subscene, specifically the ‘Information on uControllers’.
  • Possibly helpful troubleshootimg sections: Interlock Issues and Power Issues

OCD Guidelines.

This section mentions known problems and how to react to them

1. SEVERE EVENT (i.e. power cut, vacuum failure, leak of Si-oil or water)

In general, the detector interlock system will try to bring the detector to a state which is safe for the detector. Nevertheless there are situations (i.e. power cut), in which the interlock system is also affected and, depending on the type of failure, some of the safety actions cannot be automatically executed.

  • 1st step: Assess the situation, check if the detector is in the safe condition:
  • Detector is powered OFF
  • Detector is warmed / is warming up to +20 deg C
  • no Si-oil or water leak is observed
  • 2nd step: if detector is not in a safe condition, depending on the type of failure, try to minimize the risk of the detector damage:
  • Si-oil leak observed: power down detector and switch OFF the Julabo chiller in the hutch (front panel, main power switch, see user Manual for more details)
  • Vacuum failure (system is accidentally vented > 1mbar ) => check if detector is warming up, it takes ~ 15 min till the temp reach +20degC, try to bring the vacuum system up as soon as possible.
  • 3rd step: Call DET OCD

Warning

Do not try to cool down and start up detector before consultation with DET expert.

2. OPERATIONAL ISSUES - AGIPD

These issues can frequently be solved by procedures given in the manuals. OCD will only take on these problems after the prescribed troubleshooting has been performed. Therefore please follow the procedure below: * 1st step: Use trouble shooting guide to solve the problem

  • Not seeing any image updates in online preview:
  • check if there are no ERRORs in the run controller DAQ states
  • Is the DAQ in monitoring state and are the data sources selected?
  • Is the detector sending data ( use Rundeck script from ITDM)
  • Detector is not sending data:
  • Verify DAQ status
  • follow up the trouble shooting section in manual 1
  • Corrected image performance is not as expected:
  • verify image performance in raw online preview, if not ok follow up performance trouble shooting section in the manual
  • if raw image looks ok:
  • check that the parameters in the “operating conditions’ table are correct for the current operation scenario (Calibration Manager Scene)
  • follow up online preview trouble shooting 3
  • No updates in online previews
  • see online preview trouble shooting 3
  • Online previews are slow
  • see online preview trouble shooting 3
  • Calibration constants in online preview cannot be loaded
  • see online preview trouble shooting 3
  • Karabo bridge not sending data, but online preview updating
  • call Control OCD
  • DAQ crashed
  • call ITDM OCD
  • Karabo servers are down or other control problems
  • call Control OCD
  • Detector interlocks were triggered:
  • check vacuum system status, if not ok:
  • check that the detector is warming up
  • power down detector
  • fix the problem of the vacuum system
  • if the detector pressure was > 1mbar, call DET OCD to verify if the detector can be again cooled down and powered up.
  • check if the temperatures of cooling blocks and external housing are in the usual range. If not:
  • heck Julabo chiller in Rack Room, if not ok follow up Julabo manual
  • Check K3 chiller hutch rooftop, if not ok follow up K3 chiller manual
  • follow up trouble shooting section in the manual
  • Issues with startup of the detector => follow up trouble shooting section in the manual
  • Detector cannot be cooled down
  • check Julabo chiller in RR, if not ok follow up Julabo manual
  • Check K3 chiller on the hutch rooftop, if not ok follow up K3 chiller manual
  • follow up trouble shooting section in the manual

Warning

do NOT try to cool the detector if chiller is in INTERLOCKED state.

  • 2nd step: If the trouble shooting fails, call DET OCD
If you do call DET OCD, please be prepared to answer the following questions:
  • what was done before the failure occurred
  • have you tried any recovery procedures
  • have other OCD lines been called
  • have you verified the problem is due to the detector, and not of a different cause. If so, how?
  1. https://rtd.xfel.eu/docs/agipd-manual-mid/en/latest/introduction.html
  2. https://rtd.xfel.eu/docs/detector-support/en/latest/diagnosis/generic/gen_daq_troubleshooting.html
  3. https://rtd.xfel.eu/docs/detector-support/en/latest/diagnosis/generic/cal_online_troubleshooting.html