Available Notebooks¶

The following notebooks are currently integrated into European XFEL Offline Calibration tool chain.

AGIPD¶

AGIPD Correction ¶

Offline Correction for AGIPD Detector

AGIPD Characterize Dark Images ¶

This notebook analyzes a set of dark images taken with AGIPD detector to deduce detector offsets (pedestal), noise, bad-pixel maps and thresholds. All four types of constants are evaluated per-pixel and per-memory cell. Data for the detector’s three gain stages needs to be present and separated into separate runs.

The evaluated calibration constants are stored locally and injected in the calibration database.

Characterize AGIPD Pulse Capacitor Data ¶

The following code characterizes AGIPD gain via data take with the pulse capacitor source (PCS). The PCS allows scanning through the high and medium gains of AGIPD, by subsequently increasing the number of charge pulses from on-ASIC capacitor, thus increasing the charge a pixel sees in a given integration time.

Because induced charge does not originate from X-rays on the sensor, the gains evaluated here will later need to be rescaled with gains deduced from X-ray data.

PCS data is organized into multiple runs, as the on-ASIC current source cannot supply all pixels of a given module with charge at the same time. Hence, only certain pixel rows will have seen charge for a given image. These rows then first need to be combined into single module images again. This script uses new style of merging, which does not support old data format.

We then use a K-means clustering algorithm to identify components in the resulting per-pixel data series, matching to three general regions:

High gain slope
Transition region, where gain switching occurs
Medium gain slope.

The same regions are present in the gain-bit data and are used to deduce the switching threshold.

The resulting slopes are then fitted with a linear function and a combination of a linear and exponential decay function to determine the relative gains of the pixels with respect to the module. Additionally, we deduce masks for bad pixels form the data.

Gain Characterization ¶

This notebook is used to produce AGIPD Flat-Field constants.

Histogramming of AGIPD FF data ¶

No description

TODO: Add description for this notebook

Combine Constants ¶

This notebook combines constants from various evaluations

Dark image analysis, yielding offset and noise
Flat field analysis, yielding X-ray gain
Pulse capacitor analysis, yielding medium gain stage slopes and thresholds
Charge injection analysis, yielding low gain stage slopes and thresholds into a single set of calibration constants.

These constants do not include offset and noise as they need to be reevaluated more frequently.

Additionally, a bad pixel mask for all gain stages is deduced from the input. The mask contains dedicated entries for all pixels and memory cells as well as all three gains stages.

DSSC¶

DSSC Offline Correction ¶

Offline Correction for DSSC Detector

DSSC Characterize Dark Images ¶

The following code analyzes a set of dark images taken with the DSSC detector to deduce detector offsets and noise. Data for the detector is presented in one run and don’t acquire multiple gain stages.

The notebook explicitly does what pyDetLib provide in its offset calculation method for streaming data.

EPIX100¶

ePix100 Data Correction ¶

The following notebook provides data correction of images acquired with the ePix100 detector.

The sequence of correction applied are: Offset –> Common Mode Noise –> Relative Gain –> Charge Sharing –> Absolute Gain.

Offset, common mode and gain corrected data is saved to /data/image/pixels in the CORR files.

If pattern classification is applied (charge sharing correction), this data will be saved to /data/image/pixels_classified, while the corresponding patterns will be saved to /data/image/patterns in the CORR files.

ePix100 Dark Characterization ¶

The following notebook provides dark image analysis and calibration constants of the ePix100 detector.

Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.

Noise and bad pixels maps are calculated independently for each of the 4 ASICs of ePix100, since their noise behavior can be significantly different.

Common mode correction can be applied to increase sensitivity to noise related bad pixels. Common mode correction is achieved by subtracting the median of all pixels that are read out at the same time along a row/column. This is done in an iterative process, by which a new bad pixels map is calculated and used to mask data as the common mode values across the rows/columns is updated.

Resulting maps are saved as HDF5 files for a later use and injected to calibration DB.

references:¶

epix100 documentation

GOTTHARD2¶

Gotthard2 Offline Correction ¶

Offline Correction for Gotthard2 Detector.

This notebook is able to correct 25um and 50um GH2 detectors using the same correction steps: - Convert 12bit raw data into 10bit, offset subtraction, then multiply with gain constant.

Correction	constants	boolean to enable/disable
12bit to 10bit	`LUTGotthard2`
Offset	`OffsetGotthard2`	`offset_correction`
Relative gain	`RelativeGainGotthard2` + `BadPixelsFFGotthard2`	`gain_correction`

Beside the corrected data, a mask is stored using the badpixels constant of the same parameter conditions and time. - BadPixelsDarkGotthard2 - BadPixelsFFGotthard2, if relative gain correction is requested.

The correction is done per sequence file. If all selected sequence files have no images to correct the notebook will fail. The same result would be reached in case the needed dark calibration constants were not retrieved for all modules and offset_correction is True. In case one of the gain constants were not retrieved gain_correction is switched to False and gain correction is disabled.

Gotthard2 Dark Image Characterization ¶

The following is a processing for the dark constants (Offset, Noise, and BadPixelsDark) maps using dark images taken with Gotthard2 detector (GH2 50um or 25um). All constants are evaluated per strip, per pulse, and per memory cell. The maps are calculated for each gain stage that is acquired in 3 separate runs.

The three maps can be injected to the database and/or stored locally.

JUNGFRAU¶

Jungfrau Offline Correction ¶

Offline Correction for Jungfrau detector. For single and burst mode and for detector operated in adaptive or fixed gain.

This notebook expects 4 different calibration constants. Offset, BadPixelsDark, RelativeGain, and BadPixelsFF. The raw data is by default offset subtracted then divided by the gain parameter. Both bad pixels parameters are stored as a masked array data/mask along with the corrected data (same data path as RAW data/adc).

The notebook accounts for the raw gain values (0[00], 1[01], 3[11]) during correction. But the raw gain values are stored exactly the same in the corrected files.

Save reduced ROIs This notebook supports writing reduce ROI data into the corrected files. For example FXE uses a spectrometer which sends a spectrum onto a set region of the detector. This can be selected and reduced to 1D array, along with a comparable background region. This reduced data can be read rather than the full images.
Correct strixel sensors. This notebook is able to offline correct strixel JUNGFRAU as well. Using a cython function a the sensor is decoded and pixels are reordered for offline correction. Module details:
- Only 4 ASICs instead of 8
- The pixels on the ASIC are still 75 um x 75 um
- The sensor instead has rectangular pixels: 25 um x 225 um, i.e. 1/3 of the pixel pitch along the x-axis and three times along the y-axis (to maintain the total number of pixel unchanged);
- The readout however is not 'aware' of these changes, so it treats it like a 'normal' JUNGFRAU module, hence the output must be re-shuffled (in the row and column dimensions) in order for the image to make sense.

Jungfrau Dark Image Characterization ¶

Analyzes Jungfrau dark image data to deduce offset, noise and bad pixel maps from three dark runs of each gain. For data of single or burst mode and detector operated in adaptive or fixed gain.

Offset and Noise calibration parameters are computed using mean and standard deviation across pixels per memory cells and gain, respectively.

BadPixelsDark calibration parameter consists of pixels with wrong gain values, empty cell images (cells with 0 pixel values), pixels for offset and noise maps evaluated with values above bad pixel threshold sigmas, and infinite values in offset or noise maps.

Jungfrau Dark Summary ¶

Summary for process dark constants and a comparison with previously injected constants with the same conditions.

References:¶

Jungfrau documentation

LPD¶

LPD Offline Correction ¶

Offline correction for LPD detector data.

This notebook applies Offset correction then gain correction by default on LPD1M. Additionally, bad-pixels are masked and stored in a mask array along with the corrected data.

The notebook expects 6 different constants. Offset, RelativeGain, FFMap, GainAmpMap and two badpixel maps (BadPixelsDark and BadPixelsFF)

Offset –> RelativeGain * FFMap * GainAmpMap.

LPD Offset, Noise and Dead Pixels Characterization ¶

This notebook process dark images to derive offset, noise and bad-pixel maps. All three types of constants are evaluated per-pixel and per-memory cell.

The notebook will correctly handle veto settings, but note that if you veto cells you will not be able to use these offsets for runs with different veto settings - vetoed cells will have zero offset.

The evaluated calibration constants are stored locally and injected in the calibration database.

LPD Radial X-ray Gain Evaluation ¶

Taking proper flat field for LPD can be difficult, as air scattering will always be present. Additionally, the large detector mandates a large distance to the source, in order to reduce \(1/r\) effects.

Because of this a radial evaluation method is used, which assumes that pixels are the same radial distance \(r\) should on average have the same signal \(S(r)\).

Injecting calibration constant data to the database ¶

Reading h5files of calibration constants to inject them to the database. Used for LPD

LPD Gain Characterization (Charge Injection)¶

The following code characterizes the gain of the LPD detector from charge injection data, i.e. data with charge injected into the amplifiers, bypassing the sensor. The data needs to fulfil the following requirements:

Each file should represent one scan point for one muddle, defined by detector gain setting and charge injections setting
Settings need to overlap at least one point for two neighboring gain ranges
100 samples or more per pixel and memory cell should be present for each setting.

The data is then analyzed by calculating the per-pixel, per memory cell mean of the samples for each setting. These means are then normalized to the median peak position of all means of the first module. Overlapping settings in neighboring gain ranges are used to deduce the slopes of the different gains with respect to the high gain setting.

PNCCD¶

pnCCD Data Correction ¶

The following notebook provides offset, common mode, relative gain, split events and pattern classification corrections of images acquired with the pnCCD. This notebook does not yet correct for charge transfer inefficiency.

Offset –> Common Mode –> Gain Correction –> Split pattern classification

The notebook stores the offset corrected data at /data/pixels, the common mode corrected data at /data/pixels_cm, the gain constant used is stored at /data/gain, the pattern classification has two arrays stored /data/pixels_classified and /data/patterns, and the bad pixels are masked and stored at data/mask.

pnCCD Dark Characterization ¶

This notebook provides dark image analysis of the pnCCD detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.

On the first iteration, the offset and noise maps are generated. Initial bad pixels map is obtained based on the offset and initial noise maps. Edge pixels are also added to the bad pixels map.

On the second iteration, the noise map is corrected for common mode. A second bad pixel map is generated based on the offset map and offset-and-common-mode-corrected noise map. Then, the hole in the center of the CCD is added to the second bad pixel map.

On the third and final iteration, the pixels that show up on the above-mentioned bad pixels map are masked. Possible events due to cosmic rays are found and masked. The data are then again offset and common mode corrected and a new final noise and bad pixels maps are generated.

These latter resulting maps together with the offset map are saved as HDF5 files to a local path for a later use. These dark constants are not automatically sent to the database.

pnCCD Gain Characterization ¶

The following notebook provides gain characterization for the pnCCD. It relies on EuXFEL offline corrected data. Prior to running this notebook, the expected applied corrections are:

Offset correction
Common mode correction
Split pattern classification

pnCCD references:¶

pnCCD Manual

REMI¶

Transformation parameters ¶

No description

TODO: Add description for this notebook

EPIX10K¶

Detector notebooks are outdated.

EPIX10K notebooks are outdated as the detector hasn't been used for years.

ePIX10K Data Correction ¶

Offset correction of images acquired with the ePix10K detector.

ePix10K Dark Characterization ¶

Dark image analysis of the ePix10K detector.

Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Resulting maps are saved as HDF5 files for a latter use and injected to the calibration DB.

FASTCCD¶

Detector notebooks are outdated.

FastCCD notebooks are outdated as the detector hasn't been used for years.

FastCCD Data Correction ¶

The following notebook provides correction of images acquired with the FastCCD.

FastCCD Dark Characterization ¶

The following notebook provides dark image analysis of the FastCCD detector.

Dark characterization evaluates offset and noise of the FastCCD detector, corrects the noise for Common Mode (CM), and defines bad pixels relative to offset and CM corrected noise. Bad pixels are then excluded and CM corrected noise is recalculated excluding the bad pixels. Resulting offset and CM corrected noise maps, as well as the bad pixel map are sent to the calibration database.

GENERIC¶

Constants from DB to HDF5 ¶

Currently, available instances are LPD1M1 and AGIPD1M1

Overall modules darks summary ¶

Create a summary chapter for the processed dark calibration constants and a comparison with previous injected constants. Only for AGIPDs, DSSC1M, and LPD1M

TUTORIAL¶

Tutorial Calculation¶

A small example how to adapt a notebook to run with the offline calibration package “pycalibation”.

The first cell contains all parameters that should be exposed to the command line.

To run this notebook with different input parameters in parallel by submitting multiple SLURM jobs, for example for various random seed we can do the following:

xfel-calibrate TUTORIAL TEST –random-seed 1,2,3,4

or

xfel-calibrate TUTORIAL TEST –random-seed 1-5

will produce 4 processing jobs