Revisions

Version	Date	Description	Author
0.1	01/10/2018	Initial revision with instructions relative to the setup in lab e0.606	Astrid
1.0 beta	05/12/2018	Beta version intended for use at SCS	Michele
1.0 beta2	06/12/2018	Added start- and end-of-run checklists	Michele
1.0 beta3	07/12/2018	Added current Karabo screenshots and revisions section	Michele
1.1	14/02/2019	Added information on new temperature management	Michele
2.0 beta	08/04/2019	Major review with refactoring, corrections, updates.	Michele, Kiana

Introduction

This manual provides information about the FastCCD X-ray camera and its setup. First the hardware setup is described followed by instructions how to operate the system with Karabo, some additional information and a troubleshooting section.

The latest approved version of this document can always be found on in.xfel.eu/readthedocs.

Hardware Setup

Overview

FastCCD is a 2-D imaging detector. It can be run in Full-Frame mode (FF, 2 Megapixel, up to 60 Hz) and in Frame-Store mode (FS, 1 Megapixel, up to 120 Hz). Currently, the camera is operated always in FF mode.

The detector has been developed at Lawrence Berkeley National Laboratory in the U.S. and has been delivered with a DAQ rack.

As the intensity of the European XFEL beam on the detector may be very high, a central hole for the beam to pass through is required. This hole, 1.8-mm in diameter, was laser-drilled after the sensor was fabricated, bonded and tested.

The main components of the system are an in-vacuum camera head shown in Fig. 3.1, a camera interface electronics board, and an Advanced Telecommunications Computing Architecture (ATCA) based readout and data processing subsystem.

More information on the performance of the detector can be found in arXiv:1612.03605 [physics.ins-det]

The FastCCD detector head.

Property	Value
Pixel size	30 μm × 30 μm
Sensor thickness	200 μm
Dynamic range	\(10^3\) above 0.5 keV
Sensor size	1920 × 960 pixels
Photon energy range	0.25-6 keV
Beam hole size	1.8 mm
Quantum efficiency	\(\geq94\%\) for Eγ >1 keV
Speed	60 fps (FF), 120 fps (FS)

The Detector

The primary components of the detector include: the camera head with all in-vacuum modules, a vacuum-to-air interface, the backend readout electronics, and auxiliary components like temperature controller, chiller, and power supply.

Readout and Control Electronics

The FastCCD electronics is hosted in a mobile rack that contains:

• an ATCA crate with:

  • the Camera Interface Node (see Fig. 3.6 and Fig. 3.7)
  • RAID unit
  • a control pc
  • an integrated network switch

• a Keithley digital Multimeter (see Fig. 3.8)

• a Lakeshore temperature control unit (see Fig. 3.9)

• an uninterruptible power supply (see Fig. 3.10)

• an Acopian power supply for the camera electronics with a web controlled switch (see Fig. 3.11)

• a Keysight power supply for the fiber optics transceivers (see Fig. 3.12)

The Camera Interface Node (CIN) module is the one where cyan fiber optics cables are connected.

The black lever on the right of the module (1) can slide upward to power cycle the CIN. The handle underneath (2) will pop out during this process and needs to be pushed in to be able to restart the module’s power.

The Keithley multimeter unit.

The Lakeshore temperature control unit.

The uninterruptible power supply.

The Acopian power supply. The LEDs on the right side must all be ON (green) when this power supply is ON.

The Keysight power supply for the fiber optics transceivers.

The Camera Interface Node

The Camera Interface Node (CIN) is an ATCA module connected to the camera head via 6 fiber optics channels (see Fig. 3.6 and Fig. 3.7). It provides the control signals for the camera, and reads out and de-scrambles the acquired frames.

The CIN has a number of LEDs on its front panel that can be used to infer the status of the device.

CIN LEDs legend. Some of the unused LEDs are used for debuggiong purposes.

Cooling System

The detector requires cooling during operation to prevent overheating.

Chiller Setup

The CRYOMECH CP830 cryo-chiller is located near the electronics rack. Follow these instructions to turn it on (if you encounter problems, refer to the troubleshooting section):

• make sure that the power is connected and the interlock allows the chiller operation. If this is the case, the red LED on the chiller’s front panel (under interlock connector) is on (see Fig. 3.14 and Fig. 3.15).

• make sure the cooling water valve is open, the valves are on the wall, under one of the network patch panels, look for a green valve marked FastCCD:

  • open the green valve marked FastCCD.
  • turn the red valve to the right of the green valve anti-clockwise all the way.

• close the circuit breaker (up position is closed) on the chiller front panel.

• press the large green button on the front panel, the button will lighten up.

• the red LED on the back panel (see Fig. 3.16) should turn on and the chiller should make a regular (~1Hz) loud ‘chick’ noise. If it does not, it means that the compressor is not working (check the troubleshooting section).

• the FastCCD temperature should start decreasing after ~1/2 hour.

• allow for ~3h for the temperature to go down to -28 degC.

The front panel of the FastCCD chiller located in the SCS hutch.

If the power to the chiller is connected and the interlock allows the chiller operation, the red LED on the chiller’s front panel (under interlock connector) is on.

The back panel of the FastCCD chiller. The red LED should be on if the compressor is ON and you should hear the chiller’s distinguishable chick noise.

Danger

Do not turn the detector off and leave the chiller on unless you have set the Lakeshore according to Section 6.2.

Chiller Shutdown

Follow these instructions to turn the chiller off:

• press the large green button on the front panel.
• the chiller should fall silent and the green light goes off.
• Open the circuit breaker by lowering the handle on the chiller front panel.

The chiller can also be turned off from the Interlock scene. However, in this case, there is no feedback other than the temperature starting to rise.

Lakeshore Temperature Controller

The temperature of the camera is kept stable by two heating resistors (Ohmite 5 W each) installed next to the CCD plane. A PT-100 temperature sensor is installed on the same support.

An additional 100-W heating cartridge is installed near the tip of the copper rod, close to the camera (see Fig. 3.1). A PT-100 is also installed there. This second heater is required to counterbalance the cooling power provided by the Chiller.

The resistors are controlled by a Lakeshore 336 unit, which powers them in a closed feedback loop with a PT-100 in order to reach a specified temperature.

Note

The Lakeshore 336 has several heater settings 0: Off, 1: Low, 2: Medium, 3: High. You should only ever use the High setting (3).

This means that there are three sources of heat in the system:

1. FastCCD itself (0 W when off, ~20 W when on but unbiased, ~30 W when the CCD is biased and the detector is acquiring at 10 Hz)
2. the CCD heater (0-10 W)
3. the rod heater (1-100 W)

Depending on the camera status and desired CCD temperature, the rod heater needs to be set following the table in Section 6.2.

Note

For safety reasons, the Lakeshore unit will turn off all heating loops if the CCD temperature or the rod temperature exceed 20 degC.

The Lakeshore also provides an alarm signal that will disable the camera power supplies if the CCD temperature is above 0 degC, and disable the chiller if the CCD temperature falls below -60 degC.

Interlock

A hardware interlock box is connected to the Lakeshore relay output (see Fig. 3.19). It is designed to cut the power to the two power supplies (camera head and fiber optics transceivers) in case the camera head’s temperature exceeds 0 degC.

A second Lakeshore relay output will cut the chiller power if the camera temperature falls below -60 degC.

The interlock box can be found at the back of the FastCCD electronics rack in the SCS hutch. This box is connected to the Lakeshore relay output.

Controlling the FastCCD Detector

Control Software

To operate the camera, the user manual should be followed at all times. If you are unsure on what to do, contact an expert first.

Opening Scenes

The karabo project related to the FastCCD.

The FastCCD Karabo project (SCS_CDIDET_FCCD2M_DET, see Fig. 4.1) contains the following scenes:

Control with all that is required for the routine operation of the detector, see Fig. 4.2. The instructions on how to turn on the camera are found in Section 4.2 and in the following.

Bias: This scene is required to check that the correct voltage settings have been received every time the “Send Bias file” button is pressed, see Fig. 4.3.

Interlock: This scene has all the information on the interlock conditions and the sensors used by them, see Fig. 4.5. Every time something is in an error or interlocked state or does not seem to respond to the commands sent via karabo, check this scene to find out if any alarm is on.

Temperature: This scene (see Fig. 4.6) can be used to monitor the temperature sensors in the system, as well as setting the temperatures and heater ranges. Channel A is always the CCD temperature and channel B is usually the temperature of the cold rod. The position of the temperature sensors is shown in the Interlock scene.

Status: This scene (Fig. 4.7) presents the setpoints and readback values for the voltages and currents of different parts of the system. Check this scene to find out whether or not the detector is ON and configured.

The Control scene.

The Bias scene with all read-back values matching the reference.

The Bias scene, with the values that have not been correctly transmitted emphasized.

The Interlock scene.

The Temperature scene.

The Status scene.

Configuration

The camera needs 4 configuration files:

1. Firmware file: is uploaded during the auto-start procedure.
2. Timing file: Defines how the camera operates and the picture size. It is also updated during the auto start procedure.
3. Bias file: Contains the bias value settings.
4. fCRIC file: Configures the camera readout electronics.

Bias and fCRIC files should be sent each time the camera main power is cycled (turned off and on again). Other values are related to the exposure and trigger settings, like the exposure mode, the exposure time and the number of images.

Camera Setup

Follow the instructions below to get the detector ready for acquisition.

Make sure that the chiller is running, the Lakeshore is on, karabo is connected to it, and is reading the temperatures. The CCD temperature should be below 0 degC but above -30 degC (see Section 6.2). The pressure on the FastCCD part of the beam-line should be below 1e-3 mbar.

After each step, wait for the device to come out of the CHANGING state. The status of the device gives some feedback and information about the ongoing or finished task.

Safety Checks

For safety reason, some buttons (Camera On, Main Power On, Bias On) will be disabled if the temperature/pressure is outside the specified conditions.

	Max. Temp (degC)	Min. Temp (degC)	Max Pressure (mbar)
Camera On	0	-30	1E-4
Main Power On	0	-30	1E-4
Bias On	-10	-30	1E-4

Power On Camera

1. Press “Auto start Camera”:

   This will call the auto start procedure defined in the dependency. Once this has finished, exposure related settings that are set in the GUI are sent to the camera. Wait for the state to go back to ACTIVE before the next step (it should take ~40s).

2. Press “Main Power Enable”

   Then wait for the state to go back to ON. Note that nothing will be printed in the status field.

3. Quickly adjust the heater settings

   As soon as the main power is on, you have to change the rod heater setpoint (see Section 6.2). Make sure that the heater range is set to 3.

4. Click “Reconnect” to configure the detector to send data to the DAQ device server.

Prepare the Camera for Data Acquisition

Once the camera is on and the CCD temperature is stable to at least 0.1 degC, follow these instructions to start taking data:

1. Make sure the trigger mode is set to Internal.

2. Press “Send Bias File”. The bias file has to be sent each time the camera is power cycled and before turning the bias on.

3. Check the bias uploaded values:

   After the bias values have been sent to the camera, a read-back is performed and the values are automatically compared to the expected values for the given bias file. ALWAYS check manually: open the Bias scene and check that the read-back values match the reference values (within a few %). If some values have not been transferred properly, send the Bias file again.

   Danger

   If you operate the camera with the wrong bias voltage settings, you may damage the CCD.

4. Make sure that you are not sending any trigger

   Check that the Trigger mode is set to internal.

5. Press “Send fCRIC File”. The fCRIC file has to be sent each time the camera is power cycled and before turning the bias on.

   Danger

   the camera should not receive any trigger while sending the fCRIC configuration file or the data acquired will be corrupted in a subtle way. This is not trivial to catch or recover from.

6. Press “Turn Camera Bias and Clocks On”

   and confirm that you want to perform the operation and wait for the Clock and Bias indicators to become green.

   Note

   If the temperature is already below -30 degC, the electronics might be too cold to react so the system will refuse to turn the bias voltage on. In that case, let the system warm up and try again.

7. Change the heaters set-points

   As soon as you have turned the bias on, you have to adjust the rod heater setpoint (see Section 6.2). Ensure that the heater range is set to 3. Usually you want to cool down the camera to the nominal data-taking temperature.

8. Wait for the camera to cool down to the desired temperature before you take data.

9. Configure the exposure and timing settings

   See the next section for details.

10. Take the dark runs

    Make sure the gate valves are closed (check the Interlock scene). You need a 1 minute-long run (~500 trains) for each gain setting.

Exposure Settings

Trigger Modes

The camera can operate with an external trigger. An internal “software” trigger is also available but it is only used for test purposes.

To use the trigger provided by the accelerator set trigger mode to “External_1”. The camera will start receiving triggers immediately. To turn off the triggers set the trigger mode to “Internal”

If you want to add a delay to the external trigger, the instrument can control the delay of the FastCCD trigger. In alternative you can use the Delay ToE field.

Exposure

The exposure time, cycle time and delay ToE are expressed in ms. Three exposure modes are available:

Single

Acquires a single frame and stops.

Multi-mode

Acquires the number of frames specified in Num of exposures and stops.

Continuous

Keeps acquiring frames as long as triggers arrive.

The cycle time is the period of the software trigger. Cycle time needs to be longer than exposure time plus readout time (about 40 ms)

Note

If Cycle time is shorter than exposure time plus readout time, the detector behavior will be unpredictable.

Gain Settings

The default detector configuration is auto-gain (gain=0): each pixel starts with gain 8, if the readout value saturates the preamps, the gain is switched to 2 and then to 1. The gain can also be set to a specific value for all pixels from the Control scene.

Recording Data

Once the camera is ready, open the SCS_RUN_CONTROL scene (see Fig. 4.8) from the SCS_DAQ_RUN_MGMT subproject. Make sure you have the right proposal number and that SCS_FCCD2M_DETECTOR is selected. If not

1. press “Ignore data”.
2. retrieve the right proposal.
3. add SCS_FCCD2M_DETECTOR to the list of sources and click on the green check mark on the top left corner of the karabo scene.
4. press “Push to DAQ”.
5. press “Apply configuration”.
6. press “start monitoring data”.

The Run Control scene.

Now you are ready to start acquisition. Start sending data from the detector (switch to “External_1” trigger) and start the run.

Routine Operation

Safe Operation of the Detector

Follow the following guidelines to safely operate the detector.

1. The Xray beam should never touch the CCD. Immediately reduce the intensity if the readout is saturated to the point that the charge bleeds into nearby pixels.
2. Do not forget to adjust temperature set points at the beginning or end of a shift, or in general when the status of the detector is changed (see Section 6.2).
3. Make sure that the bias voltages are configured before turning on the bias voltage. The bias configuration is lost when the detector’s main power is turned off. If you are unsure if the configuration has been sent, just send it again.
4. Check the read-back values against the reference values in the Bias scene every time you send the bias file.
5. Make sure that the detector is not receiving any trigger when you send the fCRIC configuration file.
6. DO NOT leave the camera with bias on at the end of the shift.

Power Monitoring

The main power values for the camera and the fiber optics are constantly monitored.

The values displayed in the power status are evaluated to match certain expectation. If the deviation is too large, an error message is written to the log. Usually the voltages are quite stable but the currents drawn do change when turning the bias voltage on and off. Sporadic errors are of no concern but if you get frequent error messages, contact an expert.

Recover after Error

If any kind of error occurred that could not be handled automatically, the device will go into the ERROR state. If the error is due to a timeout the state will be UNKNOWN.

Pressing the “Reset” button in the Control scene will bring the device into the OFF state and power down the detector. If the camera cannot be powered down correctly (for instance because one of the power supplies is disabled by the interlock), the device may go in ERROR/UNKNOWN status again.

After a reset, you will have to start the power up procedure again. You will have to warm up the CCD above -30 degC (and below 0 degC) in order to do that.

Shutting Down the Detector

To stop the camera, press “Turn camera bias and clock off”, wait for the “Bias and clock off” message in the status. Put the detector in the “end of shift” configuration (see Section 6.2).

If you want to completely turn off the camera, press “Camera auto power down” and then power off the chiller and set the CCD heater set-point according to Section 6.2.

Danger

Do not turn off the heaters even if you have turned the chiller off, wait for the detector to reach room temperature. Otherwise FastCCD will quickly cool to below -70 degC.

FastCCD Data Acquisition

Online Preview

Raw Preview

The DAQ panel in the Control scene contains a small preview of the raw data acquired by the detector. The gain bits are not processed in this preview, so Medium and Low gain pixels have a very large, arbitrary offset with respect to High gain pixels.

In general you should set the limits of the z axis (color scale) of this preview to 3400 - 4300 ADU (ADC Units).

Corrected Preview

This preview (Fig. 5.2) shows the offset corrected images based on the most recent calibration constants (from dark run) for particular gain and temperature settings of the detector. It is therefore a processed form of the raw data preview.

The calibrated preview is part of the Control Scene (see Fig. 4.2). To update this preview to use the most recent dark runs:

1. Process the dark runs (see Section 5.2.1) with high, medium and low gain settings, which adds the most recent calibration constants to the Calibration Database.
2. On the Calibrated Preview part of the Control scene, you will see a table with 8 inputs. Change the settings in the table to match the current detector configuration and hit Return twice to commit the changes. Make sure the table does not have a blue edge as that indicates the change is not saved in Karabo. If you see the blue edge around the table, press Return again.
3. Click on Reset button and wait for the status to become PASSIVE.
4. Click on Init button to initialize the change and wait for the status to become ACTIVE.
5. At this point, the output of “Offset is from” field should change to the time the last dark runs were acquired. If it does not make sure that the condition table matches the condition in which the dark runs were acquired.
6. Even though the “Offset is from” field is updated, the preview is still based on the previous calibration constants. This is a known bug. You need to press Reset and Init one more time in the correct order to actually update the calibrated preview with the most recent calibration constants.

When a recent set of calibration constants has been loaded, the z-axis (color scale) of the image preview (and that of the histograms) should be centered around 0 for the high gain, a reasonable default is min:-100 and max:100 ADU.

If the beam intensity is too high, the preamplifiers of the CCD camera may be saturated. If that is the case, you will see dark red lines (or a dark red region) on the calibrated preview. In this case, ask the SCS beam-line scientists to reduce the intensity.

The calibrated preview.

Preview Histograms

There are two online histograms in the histogram scene in the SCS_CDIDET_FFCD2M_CAL subproject (see Fig. 5.3): both based on calibrated preview described above. In the first one only the offset correction is applied while the second one has the relative gain correction. Due to a known bug if you are using the Low or Medium gain, you will need to adjust the range according to the calibrated preview and ranges.

The online histograms.

Offline Data

Processing Dark Runs

After taking a dark run for each gain setting, log into Metadata Catalog (https://in.xfel.eu/metadata). Click on Proposals link on the top right hand side, then on the current proposal number (if you cannot find the current proposal make sure the SCS beam-line scientists have given you the permission to access it). Open the Runs page; there you should see the run numbers corresponding to the dark runs you just took. It might take a few minutes for the runs to appear on the list (you will need to reload the page manually).

If the dark runs were OK, change the Data Assessment to Good (if you cannot do this step, you do not have the right permissions for this proposal and need to ask a member of the SCS group) and migrate the data run to Maxwell cluster.

To inject the new dark runs, please follow the following steps:

1. Wait a few minutes for the data has been migrated to Maxwell.
2. Connect to Maxwell: ssh username@max-exfl
3. load anaconda: module load anaconda/3
4. For each gain/run, execute this command replacing cycle, proposal and run numbers accordingly (you need to have access to the proposal for this step to work): python /gpfs/exfel/data/scratch/haufs/request_darks.py --instrument SCS --cycle ### --proposal ### --run ###
5. You can launch multiple instances of the script simultaneously, or cancel it (with Ctrl+C) after it reports initial feedback (DONE, sometimes truncated to ONE). In case you leave it running, it will report feedback on the process.

Once the dark runs are analyzed and the calibration constants are injected into the Calibration Database, you can setup the calibrated preview (see Section 5.1.2) or reprocess acquired data.

Reprocessing Acquired Data

The instructions for the web service based calibration via Metadata Catalog are found at https://in.xfel.eu/readthedocs/docs/detector-documentation/en/doc-lpd_non_lin/calibration/offline_calibration.html.

After the process is over you will find in the proc/ folder of your proposal two files for each raw file:

1. those starting with CORR contain offset and relative gain corrected data in the “pixels” array (see Figure Fig. 5.4). These files process quite quickly.
2. those starting with CORR-D contain additionally common-mode (“pixels”) and charge split corrected data in the “pixels_classified” array, as well as a “pattern array”, which contains event multiplicity.

Accessing the Raw Data

Configuring hdfviewer to show the raw image data.

If you need to access the raw data, follow these steps:

1. Make sure the SCS beam-line instrument staff have given you the permission to the current proposal. If not, ask them to add your username to the proposal under study.
2. Open a terminal and connect to Maxwell cluster (ssh username@max-exfl)
3. Go to the directory where the current raw data are being stored. An example is: /gpfs/exfel/exp/SCS/201930/p900074/raw/r0100, where SCS is the instrument, 201930 the proposal cycle, p00074 the proposal number, and r0100 is the run number 100.
4. For a specific run number, there should be a few files in that directory. All of them look like RAW-R0100-DA##-S000##.h5, where DA## refers to the data aggregators and DA05 is for FastCCD and S##### refers to the file sequence. If the DAQ is setup correctly and the connection between DAQ device and data aggregator is established correctly, you should see one or more RAW-R0100-DA05-S#####.h5 files in the directory.
5. To take a look at what is inside these raw data files, on the terminal type: module load x-ray.
6. Open the raw data file structure with this command: hdfview RAW-R0100-DA05-S00001.h5
7. Browse to the following array to see the raw image data: /INSTRUMENT/SCS_CDIDET_FCCD2M/DAQ/FCCD:daqOutput/data/image/pixel (see Fig. 5.4)

To ensure data is being properly written on disk make sure that the “pixel” is non empty and non 0.

To visualize the data right click on pixel, select Open As, select the Image as display type then, in the Dimension box, select Height: dim 1, Width: dim 2, Depth: dim 0.

Operation Procedures

Standard Shift Operations

Start of Shift

Opening Karabo

If karabo is already open, skip to the next section. The Karabo project to control FastCCD is SCS_CDIDET_FCCD2M_DET (see Section 4.1). Load it and then open the following scenes:

• Control
• Bias
• Interlock
• Temperature
• Status

Elog

The FastCCD Elog can be found it at in.xfel.eu/elog/FastCCD. Use it to report everything concerning detector commissioning and operation, especially also errors and issues.

At the beginning of your shift start a new elog entry. Make the following checklist table and fill out the required information as shown:

Task	Result	Notes
Screenshot	OK?	Taken before turning the bias on
Initial Check (device status? Main power on? Bias off? Temperature?)	OK?
Turn bias on	OK?
Adjust heaters’ set points to cool down the camera	OK?
Operation Gain	High, Medium, Low or Auto
Integration Time (Exposure Time) in ms	value (usually 1 ms)
Take dark runs once temperature is stabilized	OK?

Powering Up the Detector

If the camera is already on, skip to the next section. Follow these instructions to turn on the chiller and the detector.

1. turn the chiller on as described in Section 3.4.1
2. Make sure that the Lakeshore Set Points are set according to Section 6.2 (camera off configuration)
3. wait for the CCD temperature to fall below 0 degC.
4. if the “Ext power supply” indicator is off, turn it on (Interlock scene)
5. the karabo control device will be in ERROR state because it could not talk to the power supply; press the “Reset” button in the Control scene
6. power up the camera and the main power supply and adjust the temperature control (see Section 4.2)

Main Power Is On

If the camera is on, you should open the Control scene and:

1. make sure that the Control Device status is ON, that the camera is on and the bias voltages and clocks are off

2. check that the CCD heater and the Rod heater are in the “end of shift” configuration (see Section 6.2)

3. configure the detector and turn the bias voltage on (see Section 4.2.3 for details and warnings)

   1. make sure that the detector is not receiving triggers
   2. send the bias file and check the uploaded values
   3. send the fCRIC file
   4. send a few triggers to check that you see the typical fCRIC noise pattern
   5. turn triggers off and turn bias and clock on
   6. change the heaters setting to the “data taking” configuration (see Section 6.2) and wait for the CCD temperature to stabilize

#. Once the temperature is stable, take a dark run of 500 trains for each gain setting. Repeat this step if you change the temperature set-point of the camera or if the detector had to be power cycled. The run types corresponding to the four gains should be available in the Run Controller. If they are not present, go to the Metadata Catalog in.xfel.eu/metadata/proposals , and select the current proposal and add them one by one as follows:

1. Dark High Gain
2. Dark Medium Gain
3. Dark Low Gain
4. Dark Auto Gain

If you want to process the dark run so that it can be used by the online preview, see Section 5.2.1.

End of Shift

Submit the elog entry of the shift.

Turn the bias off and put the detector in the “end of shift” configuration (see Section 6.2).

Detailed instructions to turn the detector off are found in Section 4.4.

Temperature Stabilization

Depending on the camera status and desired CCD temperature, the rod heater needs to be set according to the following table. Make sure that the range of both heaters is 3 (High). For additional details see Section 3.5.

FastCCD status	CCD bias voltage	CCD Heater setpoint (degC)	Rod heater setpoint (degC)	Notes
Off	Off	-28	-40
Main power on	Off	-28	-50	Leave FastCCD in this state at the “end of the shift” or if there is no one in the control room
Main power on	On	-40	-80	These are the nominal “data taking” conditions for the user run

It takes ~1h to switch form the “end of the shift” to the “data taking”. Change the heaters set-points as soon as the camera status changes.

Danger

Do not leave FastCCD with the bias on if there is no one in the control room.

Note

The Bias voltage can only be turned on if T_CCD > -30 degC. Do not cool down the camera below that threshold if the bias voltage is off.

Safety Measures

For instructions on how to safely operate the detector and the significant factors in doing so, see Section 4.3.1.

The temperature related part of the interlock system is provided by the Lakeshore temperature controller; Karabo is not aware of it and may show the power supplies or the chiller as ON even though they have been switched off by the Lakeshore.

The pressure part of the interlock is handled by a Beckoff PLC.

Input	Condition	Action	Notes
Lakeshore 336 Alarm	T_CCD > 0C	Turns off Keysight and Acopian power supply	Turns off camera and fiber optics power supplies
Lakeshore 336 Alarm	T_CCD < -60C	Turns off CRYOMECH	Turns off chiller
Beckoff	P > 1E-3 mbar	Turns off Keysight and Acopian power supply and CRYOMECH	Turns off chiller, camera, and fiber optics power supplies [1]
Beckoff	Turbo pump failure	Turns off Keysight and Acopian power supply and CRYOMECH	Turns off chiller, camera, and fiber optics power supplies

[1]	Bake at 35 degC for 1h before operating again.

Troubleshooting

Camera Control

The Detector Control Device Is in ERROR State

Make sure you have read Section 4.3.3.

The most likely cause of an ERROR state is that the “Exception in slot Fiber status update”. This happens when the detector power supply is completely off and the karabo device is complaining that it cannot check its status.

This can happen when one of the following is true:

• the power is disabled via karabo (power enable “on/off” button on the Interlock scene)

• the interlock has disabled it because one of the following conditions is true

  • the pressure in the vessel is too high
  • a vacuum pump has failed
  • the Lakeshore high temperature alarm is on

You can check these in the Interlock scene. You can also check by looking at the interlock box in the FastCCD crate (if one light is off then the power is cut off). Finally you can see if the screen of the Keysight power supply on the FastCCD rack is completely black.

To solve this, remove the error condition:

• make sure that the FastCCD pressure is below 1e-3 mbar
• make sure that the temperature is below 0 degC and reset the alarm on the Lakeshore. To do that you can use the button in the Interlock scene or press the “alarm” button on the front panel of the Lakeshore itself and answer “yes”
• press the power enable button on the Control scene

Then press the “reset” button on the Control scene to force the device to try and connect to the power supply again.

Cannot Turn the Camera On

The Camera On and Main Power On buttons are greyed out if the CCD temperature or vessel pressure are outside the allowed range (see Section 4.2.1).

If the device is in ERROR check that if the Lakeshore alarm is on (see Section 7.3.3). The power supply is automatically turned off if the temperature or pressure are outside the allowed range. Again check the Interlock scene to make sure you are in the correct configuration and that you have acknowledged the Lakeshore alarms.

Repeated FAILURE Uploading Bias File

Danger

If you operate the camera with the wrong bias voltage settings, you may damage the CCD.

If all of the readout values for the Bias voltages differ from their reference values check if the camera Acopian power supply is physically turned off. Check the voltage provided by the power supply using the Status scene. Check that the button is pressed and that all LEDs are on. The power supply is located at the bottom of the electronics crate and has several red LEDs on the front panel.

Cannot Turn the Bias Voltage On

The bias voltage cannot be turned on if the CCD temperature or vessel pressure is outside the allowed range (see Section 4.2.1). Check the Interlock scene to make sure you are in the correct configuration.

Excessive Current Draw by Detector

The currents drawn by the detector head can spike when turning the bias voltage on and off. Sporadic errors are of no concern but if you get frequent messages it may be a sign that the beam intensity on the CCD is too high. Reduce the signal intensity and contact an expert if the issue does not go away.

Data Acquisition

Raw Data Preview Is Not Being Updated

Here is a list of possible causes and solutions.

The frames are all very similar

Check if the Frame Number is being updated. If it is, then maybe the frames are just all very similar. Adjust the color scale and zoom into the preview picture.

The external trigger selection has not been confirmed

Once you switch to External_1 trigger mode you need to hit RETURN, the DAQ status should then change to ACQUIRING.

The Karabo DAQ device is not connected to the detector

press the “Reconnect” button in the DAQ section of the Control scene.

The camera is off or not properly configured

Check the Status messages and make sure the camera is on and that it is properly configured. If you are unsure, turn it off and follow the start procedure again.

Raw Data Preview Has Some Odd Features

Here is a list of possible causes and solutions.

The raw data is a test pattern

If, after resetting the color scale, the raw preview looks like Fig. 7.3, it means that the Fiber Optics test pattern is on; you can turn the FO test pattern off using the “FO pattern off” button in the FCCD control device. If the preview looks like figures Fig. 7.1 or Fig. 7.2, it means that the fCRICs are not configured correctly; turn the triggers off and send the fCRIC configuration file again.

There is a big blob in the middle of the preview

The Alignment Laser (ALas) may be hitting the CCD. Check if the instrument scientists are using the laser and if it is supposed to go through the CCD.

The fCRICS test pattern visible in the raw preview. You will see this if the fCRICs have not been configured correctly.

The fCRICS test pattern may appear like this if you have fixed the z-axis color scale centered around the offset (in this case 3400-4300).

The raw preview will look like this if the Fiber Optics test pattern is on.

Electronic Noise Visible But No X-Ray Signal

If the CCD temperature is already below -30 degC when you turn the bias voltage on, the electronics might be too cold to react. If it happens, turn the bias off, let the system warm up and try to turn the bias on again.

Problems with Saved Data

Cannot Connect to the Maxwell Cluster

If you are trying to connect from one of the desktops in the control room you will have to login to exflgateway first. If you are outside the DESY/XFEL network (for instance, your laptop is connected to the WiFi network) you will have to go trough bastion.desy.de).

No or Corrupted Data Are Being Written to File

You are using the software trigger

You are using the software trigger. Do not use the “start/stop triggering exposures” button, rather make sure the software trigger is off and set the trigger to “external_1”. See also Section 4.2.4.1.

Connection between DAQ device and Data Aggregator lost

This may happen if the FastCCD DAQ device is restarted when the DAQ is not in ignore mode. You can ask ITDM on-call to verify the status of the TCP connection between SCS_CDIDET_FCCD2M/DAQ/FCCD (on exflcon145) and its Data Aggregator.

To solve the problem:

1. Switch the trigger mode to Internal

2. Go in ignore mode

3. Ask ITDM to

   1. Shut down the Data Aggregator (for FCCD it should be SCS_DAQ_DATA/DA/5).
   2. Shut down the FCCD DAQ device (SCS_CDIDET_FCCD2M/DAQ/FCCD).
   3. Shut down the cppServer/scs_fccd_daq server and all its children.
   4. The cppServer/scs_fccd_daq server should restart itself almost immediately.
   5. Wait for at least 90 seconds after the server restarted. Then restart the Data Aggregator.
   6. Restart the FCCD DAQ device.

4. Push data to DAQ:

   1. make sure the proposal number is correct, if not, change it and retrieve the proposal.
   2. Apply Configuration.
   3. Monitor Data.
   4. Change trigger mode to External_1.

5. Take a test run, if you still have issues, ask ITDM to check the connection again.

Hardware

The most common issues and solutions are listed here. For more information on troubleshooting an issue with the chiller, refer to the troubleshooting section of the chiller manual

Chiller Does Not Turn On

Here is a list of possible causes and solutions.

The chiller has no power

Connect the power cable to the outlet. Check that the circuit breaker on the front panel is up (closed).

The chiller is interlocked

If the red LED on the chiller front panel is off, then the chiller is interlocked (see also Section 3.5.1). Check the Interlock scene in karabo to find out what is causing the issue:

CCD temperature below treshold

See Section 7.3.3

Pressure over threshold or turbo pump error

Wait for pressure to fall below 1e-3 mbar.

Chiller disabled in karabo

Enable the chiller in the Interlock scene in karabo.

The chiller has overheated

this usually happens if the cooling water was off. Follow these instructions to recover:

1. make sure the cooling water is flowing (touch the pipes to feel the water temperature).
2. wait ~15 minutes.
3. press the over-temperature reset switch at the bottom of the fron panel.
4. try to turn on the chiller again.

The Chiller Head Is Not Making a Regular (~1Hz) Loud ‘Chick’ Noise

Here is a list of possible causes and solutions.

the power cable from the motor to the head is not connected

check that it is connected on both sides.

the pins in the chiller connector have receded

Get a qualified electrician to disconnect the power plug, open the chiller and push the pins back in place.

CCD or Rod Temperature Does Not Rise

This can happen if:

the heater range is set to anything other than 3

Change the heater range to 3.

The Lakeshore alarm is on

See the next session.

The Lakeshore Alarm Is On

The Lakeshore 336 continuously monitors the CCD temperature and will go in an Alarm state if the temperature goes above 0 degC or below -60 degC. In the first case, it will disable the camera power supply; and in the second case it will disable the chiller.

The Lakeshore will keep the relevant equipment disabled until the condition that caused the alarm is removed (the temperature has to go back into the allowed range) and the alarm is acknowledged. You can do the latter from the Interlock scene or manually from the front panel of the Lakeshore.

Additional Safety Measures

The Lakeshore will automatically disable a heater if the temperature measured by the corresponding sensor is above ~20 degC. This leaves you enough room to use the Lakeshore to quickly warm up the detector if needed but avoid going above room temperature.