Available Notebooks¶
The following notebooks are currently integrated into the European XFEL Offline Calibration tool chain.
AGIPD¶
Combine Constants¶
Author: S. Hauf, Version: 0.1
This notebook combines constants from various evaluations
- dark image analysis, yielding offset and noise
- flat field analysis, yielding X-ray gain
- pulse capacitor analysis, yielding medium gain stage slopes and thresholds
- charge injection analysis, yielding low gain stage slopes and thresholds
into a single set of calibration constants. These constants do not include offset and noise as they need to be reevaluated more frequently.
Additionally, a bad pixel mask for all gain stages is deduced from the input. The mask contains dedicated entries for all pixels and memory cells as well as all three gains stages.
To invoke this notebook and display help use:
xfel-calibrate AGIPD COMBINE --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--modules str [str ...]]
--out-folder str [--local-output | --no-local-output]
[--db-output | --no-db-output] [--bias-voltage int]
[--cal-db-interface str] [--mem-cells int]
[--instrument str] [--photon-energy float]
[--offset-store str] [--gain-store str]
[--pc-store-base str] [--ci-store str]
[--high-res-badpix-3d | --no-high-res-badpix-3d]
[--db-input | --no-db-input]
[--deviation-threshold float]
[--max-dev-high-gain float] [--max-dev-med-gain float]
[--threshold-bounds-high-med int [int ...]]
[--thresholds-offset-sigma float]
[--thresholds-offset-hard int [int ...]]
[--thresholds-noise-sigma float]
[--thresholds-noise-hard int [int ...]]
[--thresholds-xraygain-hard-high float [float ...]]
[--thresholds-xraygain-hard-med float [float ...]]
[--thresholds-xraygain-hard-low float [float ...]]
[--no-flat-fields | --no-no-flat-fields]
AGIPD COMBINE
# Combine Constants #
,
Author: S. Hauf, Version: 0.1,
,
This notebook combines constants from various evaluations,
,
* dark image analysis, yielding offset and noise,
* flat field analysis, yielding X-ray gain,
* pulse capacitor analysis, yielding medium gain stage slopes and thresholds,
* charge injection analysis, yielding low gain stage slopes and thresholds,
,
into a single set of calibration constants. These constants do not include
offset and noise as they need to be reevaluated more frequently.,
,
Additionally, a bad pixel mask for all gain stages is deduced from the input.
The mask contains dedicated entries for all pixels and memory cells as well
as all three gains stages.
positional arguments:
AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
COMBINE Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--modules str [str ...]
modules to work on, range allowed. Default: [-1]
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--bias-voltage int detector bias voltage. Default: 300
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:5005
--mem-cells int number of memory cells used. Default: 64
--instrument str Default: SPB
--photon-energy float
the photon energy in keV. Default: 9.2
--offset-store str for file-based access. Default: /gpfs/exfel/exp/SPB/20
1830/p900019/usr/calibration0618/dark0.5MHz/agipd_offs
et_store_r0852_r0853_r0854.h5
--gain-store str for file-based access. Default: /gpfs/exfel/exp/SPB/20
1830/p900019/usr/calibration0618/FF/agipd_gain_store_r
0820_modules_CHANID.h5
--pc-store-base str for file-based access, use CHANID to indicate module
ranges. Default: /gpfs/exfel/exp/SPB/201831/p900039/us
r/AGIPD/PC/Veto_1MHZ/agipd_pc_store_38_39_40_41_42_43_
44_60_CHANID_CHANID.h5
--ci-store str for file based access. Default: /gpfs/exfel/data/scrat
ch/haufs/gain_Data/AGIPD_ci_data_74.h5
--high-res-badpix-3d set this to True if you need high-resolution 3d bad
pixel plots. Runtime: ~ 1h. Default: False
--no-high-res-badpix-3d
Opposite of --high-res-badpix-3d (default: False)
--db-input retreive data from calibration database, setting
offset-store will overwrite this. Default: False
--no-db-input Opposite of --db-input (default: False)
--deviation-threshold float
peaks with an absolute location deviation larger than
the medium are are considere bad. Default: 0.75
--max-dev-high-gain float
Default: 0.2
--max-dev-med-gain float
Default: 0.5
--threshold-bounds-high-med int [int ...]
Default: [100, 8100]
--thresholds-offset-sigma float
Default: 5.0
--thresholds-offset-hard int [int ...]
Default: [4000, 9000]
--thresholds-noise-sigma float
Default: 10.0
--thresholds-noise-hard int [int ...]
Default: [1, 20]
--thresholds-xraygain-hard-high float [float ...]
Default: [0.75, 1.25]
--thresholds-xraygain-hard-med float [float ...]
Default: [0.001, 0.25]
--thresholds-xraygain-hard-low float [float ...]
Default: [0.0001, 0.025]
--no-flat-fields Default: False
--no-no-flat-fields Opposite of --no-flat-fields (default: False)
required arguments:
--out-folder str path to output to, required. Default: None
AGIPD Offline Correction¶
Author: European XFEL Detector Group, Version: 2.0
Offline Calibration for the AGIPD Detector
To invoke this notebook and display help use:
xfel-calibrate AGIPD CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
[--sequences str [str ...]] [--modules str [str ...]]
[--train-ids str [str ...]] --run int [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-template str]
[--path-template str] [--instrument-source-template str]
[--index-source-template str]
[--ctrl-source-template str] [--karabo-id-control str]
[--slopes-ff-from-files str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--creation-date-offset str] [--mem-cells int]
[--bias-voltage int] [--acq-rate float]
[--gain-setting int] [--gain-mode int]
[--overwrite | --no-overwrite]
[--max-pulses int [int ...]] [--mem-cells-db int]
[--integration-time int] [--blc-noise-threshold int]
[--cm-dark-fraction float]
[--cm-dark-range float [float ...]] [--cm-n-itr int]
[--hg-hard-threshold int] [--mg-hard-threshold int]
[--noisy-adc-threshold float] [--ff-gain float]
[--photon-energy float]
[--only-offset | --no-only-offset]
[--rel-gain | --no-rel-gain]
[--xray-gain | --no-xray-gain]
[--blc-noise | --no-blc-noise]
[--blc-stripes | --no-blc-stripes]
[--blc-hmatch | --no-blc-hmatch]
[--match-asics | --no-match-asics]
[--adjust-mg-baseline | --no-adjust-mg-baseline]
[--zero-nans | --no-zero-nans]
[--zero-orange | --no-zero-orange]
[--blc-set-min | --no-blc-set-min]
[--corr-asic-diag | --no-corr-asic-diag]
[--force-hg-if-below | --no-force-hg-if-below]
[--force-mg-if-below | --no-force-mg-if-below]
[--mask-noisy-adc | --no-mask-noisy-adc]
[--common-mode | --no-common-mode]
[--melt-snow | --no-melt-snow]
[--mask-zero-std | --no-mask-zero-std]
[--low-medium-gap | --no-low-medium-gap]
[--round-photons | --no-round-photons]
[--use-ppu-device str] [--ppu-train-offset int]
[--use-litframe-finder str] [--litframe-device-id str]
[--energy-threshold int] [--use-xgm-device str]
[--recast-image-data str]
[--compress-fields str [str ...]]
[--skip-plots | --no-skip-plots] [--cell-id-preview int]
[--chunk-size int] [--n-cores-correct int]
[--n-cores-files int] [--sequences-per-node int]
[--max-nodes int] [--max-tasks-per-worker int]
AGIPD CORRECT
# AGIPD Offline Correction #
,
Author: European XFEL Detector Group, Version: 2.0,
,
Offline Calibration for the AGIPD Detector
positional arguments:
AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequences str [str ...]
sequences to correct, set to -1 for all, range
allowed. Default: [-1]
--modules str [str ...]
modules to correct, set to -1 for all, range allowed.
Default: [-1]
--train-ids str [str ...]
train IDs to correct, set to -1 for all, range
allowed. Default: [-1]
--karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1
--karabo-da str [str ...]
a list of data aggregators names, Default [-1] for
selecting all data aggregators. Default: ['-1']
--receiver-template str
inset for receiver devices. Default: {}CH0
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{:05d}.h5
--instrument-source-template str
path in the HDF5 file to images. Default:
{}/DET/{}:xtdf
--index-source-template str
path in the HDF5 file to images. Default:
INDEX/{}/DET/{}:xtdf/
--ctrl-source-template str
path to control information. Default: {}/MDL/FPGA_COMP
--karabo-id-control str
karabo-id for control device. Default:
MID_EXP_AGIPD1M1
--slopes-ff-from-files str
Path to locally stored SlopesFF and BadPixelsFF
constants, loaded in precorrection notebook. Default:
--use-dir-creation-date
use the creation data of the input dir for database
queries. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8015#8045
--cal-db-timeout int in milliseconds. Default: 30000
--creation-date-offset str
add an offset to creation date, e.g. to get different
constants. Default: 00:00:00
--mem-cells int Number of memory cells used, set to 0 to automatically
infer. Default: 0
--bias-voltage int bias voltage, set to 0 to use stored value in slow
data.. Default: 0
--acq-rate float the detector acquisition rate, use 0 to try to auto-
determine. Default: 0.0
--gain-setting int the gain setting, use -1 to use value stored in slow
data.. Default: -1
--gain-mode int gain mode (0: adaptive, 1-3 fixed high/med/low, -1:
read from CONTROL data). Default: -1
--overwrite set to True if existing data should be overwritten.
Default: True
--no-overwrite Opposite of --overwrite (default: True)
--max-pulses int [int ...]
range list [st, end, step] of memory cell indices to
be processed within a train. 3 allowed maximum list
input elements.. Default: [0, 352, 1]
--mem-cells-db int set to a value different than 0 to use this value for
DB queries. Default: 0
--integration-time int
integration time, negative values for auto-detection..
Default: -1
--blc-noise-threshold int
above this mean signal intensity now baseline
correction via noise is attempted. Default: 5000
--cm-dark-fraction float
threshold for fraction of empty pixels to consider
module enough dark to perform CM correction. Default:
0.66
--cm-dark-range float [float ...]
range for signal value ADU for pixel to be consider as
a dark pixel. Default: [-50.0, 30]
--cm-n-itr int number of iterations for common mode correction.
Default: 4
--hg-hard-threshold int
threshold to force medium gain offset subtracted pixel
to high gain. Default: 1000
--mg-hard-threshold int
threshold to force medium gain offset subtracted pixel
from low to medium gain. Default: 1000
--noisy-adc-threshold float
threshold to mask complete adc. Default: 0.25
--ff-gain float conversion gain for absolute FlatField constants,
while applying xray_gain. Default: 7.2
--photon-energy float
photon energy in keV, non-positive value for XGM
autodetection. Default: -1.0
--only-offset Apply only Offset correction. if False, Offset is
applied by Default. if True, Offset is only applied..
Default: False
--no-only-offset Opposite of --only-offset (default: False)
--rel-gain do relative gain correction based on PC data. Default:
False
--no-rel-gain Opposite of --rel-gain (default: False)
--xray-gain do relative gain correction based on xray data.
Default: False
--no-xray-gain Opposite of --xray-gain (default: False)
--blc-noise if set, baseline correction via noise peak location is
attempted. Default: False
--no-blc-noise Opposite of --blc-noise (default: False)
--blc-stripes if set, baseline corrected via stripes. Default: False
--no-blc-stripes Opposite of --blc-stripes (default: False)
--blc-hmatch if set, base line correction via histogram matching is
attempted. Default: False
--no-blc-hmatch Opposite of --blc-hmatch (default: False)
--match-asics if set, inner ASIC borders are matched to the same
signal level. Default: False
--no-match-asics Opposite of --match-asics (default: False)
--adjust-mg-baseline adjust medium gain baseline to match highest high gain
value. Default: False
--no-adjust-mg-baseline
Opposite of --adjust-mg-baseline (default: False)
--zero-nans set NaN values in corrected data to 0. Default: False
--no-zero-nans Opposite of --zero-nans (default: False)
--zero-orange set to 0 very negative and very large values in
corrected data. Default: False
--no-zero-orange Opposite of --zero-orange (default: False)
--blc-set-min Shift to 0 negative medium gain pixels after offset
corr. Default: False
--no-blc-set-min Opposite of --blc-set-min (default: False)
--corr-asic-diag if set, diagonal drop offs on ASICs are corrected.
Default: False
--no-corr-asic-diag Opposite of --corr-asic-diag (default: False)
--force-hg-if-below set high gain if mg offset subtracted value is below
hg_hard_threshold. Default: False
--no-force-hg-if-below
Opposite of --force-hg-if-below (default: False)
--force-mg-if-below set medium gain if mg offset subtracted value is below
mg_hard_threshold. Default: False
--no-force-mg-if-below
Opposite of --force-mg-if-below (default: False)
--mask-noisy-adc Mask entire ADC if they are noise above a relative
threshold. Default: False
--no-mask-noisy-adc Opposite of --mask-noisy-adc (default: False)
--common-mode Common mode correction. Default: False
--no-common-mode Opposite of --common-mode (default: False)
--melt-snow Identify (and optionally interpolate) 'snowy' pixels.
Default: False
--no-melt-snow Opposite of --melt-snow (default: False)
--mask-zero-std Mask pixels with zero standard deviation across train.
Default: False
--no-mask-zero-std Opposite of --mask-zero-std (default: False)
--low-medium-gap 5 sigma separation in thresholding between low and
medium gain. Default: False
--no-low-medium-gap Opposite of --low-medium-gap (default: False)
--round-photons Round to absolute number of photons, only use with
gain corrections. Default: False
--no-round-photons Opposite of --round-photons (default: False)
--use-ppu-device str Device ID for a pulse picker device to only process
picked trains, empty string to disable. Default:
--ppu-train-offset int
When using the pulse picker, offset between the PPU's
sequence start and actually picked train. Default: 0
--use-litframe-finder str
Process only illuminated frames: 'off' - disable,
'device' - use online device data, 'offline' - use
offline algorithm, 'auto' - choose online/offline
source automatically (default). Default: off
--litframe-device-id str
Device ID for a lit frame finder device, empty string
to auto detection. Default:
--energy-threshold int
The low limit for the energy (uJ) exposed by frames
subject to processing. If -1000, selection by pulse
energy is disabled. Default: -1000
--use-xgm-device str DoocsXGM device ID to obtain actual photon energy,
operating condition else.. Default:
--recast-image-data str
Cast data to a different dtype before saving. Default:
--compress-fields str [str ...]
Datasets in image group to compress.. Default:
['gain', 'mask']
--skip-plots exit after writing corrected files and metadata.
Default: False
--no-skip-plots Opposite of --skip-plots (default: False)
--cell-id-preview int
cell Id used for preview in single-shot plots.
Default: 1
--chunk-size int Size of chunk for image-wise correction. Default: 1000
--n-cores-correct int
Number of chunks to be processed in parallel. Default:
16
--n-cores-files int Number of files to be processed in parallel. Default:
4
--sequences-per-node int
number of sequence files per cluster node if run as
SLURM job, set to 0 to not run SLURM parallel.
Default: 2
--max-nodes int Maximum number of SLURM jobs to split correction work
into. Default: 8
--max-tasks-per-worker int
the number of tasks a correction pool worker process
can complete before it will exit and be replaced with
a fresh worker process. Leave as -1 to keep worker
alive as long as pool.. Default: 1
required arguments:
--in-folder str the folder to read data from, required. Default: None
--out-folder str the folder to output to, required. Default: None
--run int runs to process, required. Default: None
AGIPD Characterize Dark Images¶
Author: European XFEL Detector Group, Version: 2.0
The following code analyzes a set of dark images taken with the AGIPD detector to deduce detector offsets , noise, bad-pixel maps and thresholding. All four types of constants are evaluated per-pixel and per-memory cell. Data for the detector’s three gain stages needs to be present, separated into separate runs.
The evaluated calibration constants are stored locally and injected in the calibration data base.
To invoke this notebook and display help use:
xfel-calibrate AGIPD DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
[--modules str [str ...]] --run-high int --run-med int
--run-low int [--operation-mode str] [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-template str]
[--instrument-source-template str]
[--ctrl-source-template str] [--karabo-id-control str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--local-output | --no-local-output]
[--db-output | --no-db-output] [--mem-cells int]
[--bias-voltage int] [--gain-setting int]
[--gain-mode int] [--integration-time int]
[--acq-rate float] [--interlaced | --no-interlaced]
[--thresholds-offset-sigma float]
[--thresholds-offset-hard int [int ...]]
[--thresholds-offset-hard-hg int [int ...]]
[--thresholds-offset-hard-mg int [int ...]]
[--thresholds-offset-hard-lg int [int ...]]
[--thresholds-offset-hard-hg-fixed int [int ...]]
[--thresholds-offset-hard-mg-fixed int [int ...]]
[--thresholds-offset-hard-lg-fixed int [int ...]]
[--thresholds-noise-sigma float]
[--thresholds-noise-hard int [int ...]]
[--thresholds-noise-hard-hg int [int ...]]
[--thresholds-noise-hard-mg int [int ...]]
[--thresholds-noise-hard-lg int [int ...]]
[--thresholds-gain-sigma float] [--max-trains int]
[--min-trains int]
[--high-res-badpix-3d | --no-high-res-badpix-3d]
AGIPD DARK
# AGIPD Characterize Dark Images #
,
Author: European XFEL Detector Group, Version: 2.0,
,
The following code analyzes a set of dark images taken with the AGIPD
detector to deduce detector offsets noise, bad-pixel maps and thresholding.
All four types of constants are evaluated per-pixel and per-memory cell. Data
"for the detectors three gain stages needs to be present, separated into "
separate runs.,
,
The evaluated calibration constants are stored locally and injected in the
calibration data base.,
positional arguments:
AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--modules str [str ...]
list of modules to evaluate, RANGE ALLOWED. Default:
[-1]
--operation-mode str Detector operation mode, optional (defaults to
"ADAPTIVE_GAIN"). Default: ADAPTIVE_GAIN
--karabo-id str karabo karabo_id. Default: HED_DET_AGIPD500K2G
--karabo-da str [str ...]
a list of data aggregators names, Default [-1] for
selecting all data aggregators. Default: ['-1']
--receiver-template str
inset for receiver devices. Default: {}CH0
--instrument-source-template str
path in the HDF5 file to images. Default:
{}/DET/{}:xtdf
--ctrl-source-template str
path to control information. Default: {}/MDL/FPGA_COMP
--karabo-id-control str
karabo-id for control device '. Default:
HED_EXP_AGIPD500K2G
--use-dir-creation-date
use dir creation date as data production reference
date. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8020
--cal-db-timeout int timeout on caldb requests". Default: 3000000
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--mem-cells int number of memory cells used, set to 0 to automatically
infer. Default: 0
--bias-voltage int bias voltage, set to 0 to use stored value in slow
data.. Default: 0
--gain-setting int the gain setting, use -1 to use value stored in slow
data.. Default: -1
--gain-mode int gain mode, use -1 to use value stored in slow data..
Default: -1
--integration-time int
integration time, negative values for auto-detection..
Default: -1
--acq-rate float the detector acquisition rate, use 0 to try to auto-
determine. Default: 0.0
--interlaced assume interlaced data format, for data prior to Dec.
2017. Default: False
--no-interlaced Opposite of --interlaced (default: False)
--thresholds-offset-sigma float
offset sigma thresholds for offset deduced bad pixels.
Default: 3.0
--thresholds-offset-hard int [int ...]
For setting the same threshold offset for the 3 gains.
Left for backcompatability. Default [0, 0] to take the
following parameters.. Default: [0, 0]
--thresholds-offset-hard-hg int [int ...]
High-gain thresholds in absolute ADU terms for offset
deduced bad pixels. Default: [3000, 7000]
--thresholds-offset-hard-mg int [int ...]
Medium-gain thresholds in absolute ADU terms for
offset deduced bad pixels. Default: [6000, 10000]
--thresholds-offset-hard-lg int [int ...]
Low-gain thresholds in absolute ADU terms for offset
deduced bad pixels. Default: [6000, 10000]
--thresholds-offset-hard-hg-fixed int [int ...]
Same as thresholds_offset_hard_hg, but for fixed gain
operation. Default: [3500, 6500]
--thresholds-offset-hard-mg-fixed int [int ...]
Same as thresholds_offset_hard_mg, but for fixed gain
operation. Default: [3500, 6500]
--thresholds-offset-hard-lg-fixed int [int ...]
Same as thresholds_offset_hard_lg, but for fixed gain
operation. Default: [3500, 6500]
--thresholds-noise-sigma float
noise sigma thresholds for offset deduced bad pixels.
Default: 5.0
--thresholds-noise-hard int [int ...]
For setting the same threshold noise for the 3 gains.
Left for backcompatability. Default [0, 0] to take the
following parameters.. Default: [0, 0]
--thresholds-noise-hard-hg int [int ...]
High-gain thresholds in absolute ADU terms for offset
deduced bad pixels. Default: [4, 20]
--thresholds-noise-hard-mg int [int ...]
Medium-gain thresholds in absolute ADU terms for
offset deduced bad pixels. Default: [4, 20]
--thresholds-noise-hard-lg int [int ...]
Low-gain thresholds in absolute ADU terms for offset
deduced bad pixels. Default: [4, 20]
--thresholds-gain-sigma float
Gain separation sigma threshold. Default: 5.0
--max-trains int Maximum number of trains to use for processing dark.
Set to 0 to process all available trains. 550 added
for ~500GB nodes to temporarely avoid memory issues..
Default: 550
--min-trains int Miniumum number of trains for processing dark. If run
folder has less than minimum trains, processing is
stopped.. Default: 1
--high-res-badpix-3d set this to True if you need high-resolution 3d bad
pixel plots. ~7mins extra time for 64 memory cells.
Default: False
--no-high-res-badpix-3d
Opposite of --high-res-badpix-3d (default: False)
required arguments:
--in-folder str path to input data, required. Default: None
--out-folder str path to output to, required. Default: None
--run-high int run number in which high gain data was recorded,
required. Default: None
--run-med int run number in which medium gain data was recorded,
required. Default: None
--run-low int run number in which low gain data was recorded,
required. Default: None
Gain Characterization¶
To invoke this notebook and display help use:
xfel-calibrate AGIPD FF --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
[--hist-file-template str] [--modules str [str ...]]
[--raw-folder str] [--proc-folder str] [--run int]
[--karabo-id str] [--karabo-da str [str ...]]
[--receiver-id str] [--path-template str] [--h5path str]
[--h5path-idx str] [--h5path-ctrl str]
[--karabo-id-control str] [--karabo-da-control str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--local-output | --no-local-output]
[--db-output | --no-db-output]
[--peak-range int [int ...]]
[--peak-width-range int [int ...]]
[--peak-norm-range float [float ...]]
[--peak-lim int [int ...]] [--d0-lim int [int ...]]
[--peak-width-lim float [float ...]]
[--chi2-lim int [int ...]] [--intensity-lim int]
[--gain-lim float [float ...]]
[--cell-range int [int ...]]
[--pixel-range int [int ...]] [--max-bins int]
[--batch-size int [int ...]] [--fit-range int [int ...]]
[--n-peaks-fit int] [--fix-peaks | --no-fix-peaks]
[--do-minos | --no-do-minos] [--sigma-limit float]
[--mem-cells int] [--bias-voltage int]
[--acq-rate float] [--gain-setting int]
[--photon-energy float] [--integration-time int]
AGIPD FF
# Gain Characterization #
positional arguments:
AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
FF Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--hist-file-template str
the template to use to access histograms. Default:
hists_m{:02d}_sum.h5
--modules str [str ...]
modules to correct, set to -1 for all, range allowed.
Default: [10]
--raw-folder str Path to raw image data used to create histograms.
Default: /gpfs/exfel/exp/MID/202030/p900137/raw
--proc-folder str Path to corrected image data used to create
histograms. Default:
--run int of the run of image data used to create histograms.
Default: 449
--karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1
--karabo-da str [str ...]
a list of data aggregators names, Default [-1] for
selecting all data aggregators. Default: ['-1']
--receiver-id str inset for receiver devices. Default: {}CH0
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{:05d}.h5
--h5path str path in the HDF5 file to images. Default:
INSTRUMENT/{}/DET/{}:xtdf/
--h5path-idx str path in the HDF5 file to images. Default:
INDEX/{}/DET/{}:xtdf/
--h5path-ctrl str path to control information. Default:
/CONTROL/{}/MDL/FPGA_COMP
--karabo-id-control str
karabo-id for control device. Default:
MID_IRU_AGIPD1M1
--karabo-da-control str
karabo DA for control infromation. Default:
AGIPD1MCTRL00
--use-dir-creation-date
use the creation data of the input dir for database
queries. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8015#8045
--cal-db-timeout int in milli seconds. Default: 30000
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--peak-range int [int ...]
where to look for the peaks, [a0, b0, a1, b1, ...]
exactly 8 elements. Default: [-30, 30, 35, 70, 95,
135, 145, 220]
--peak-width-range int [int ...]
fit limits on the peak widths, [a0, b0, a1, b1, ...]
exactly 8 elements. Default: [0, 30, 0, 35, 0, 40, 0,
45]
--peak-norm-range float [float ...]
. Default: [0.0, -1, 0, -1, 0, -1, 0, -1]
--peak-lim int [int ...]
Limit of position of noise peak. Default: [-30, 30]
--d0-lim int [int ...]
hard limits for distance between noise and first peak.
Default: [10, 80]
--peak-width-lim float [float ...]
hard limits on the peak widths for first and second
peak, in units of the noise peak. 4 parameters..
Default: [0.9, 1.55, 0.95, 1.65]
--chi2-lim int [int ...]
Hard limit on chi2/nDOF value. Default: [0, 3.0]
--intensity-lim int Threshold on standard deviation of a histogram in ADU.
Contribute to BadPixel bit "No_Entry". Default: 15
--gain-lim float [float ...]
Threshold on gain in relative number. Contribute to
BadPixel bit "Gain_deviation". Default: [0.8, 1.2]
--cell-range int [int ...]
range of cell to be considered, [0,0] for all.
Default: [1, 3]
--pixel-range int [int ...]
range of pixels x1,y1,x2,y2 to consider [0,0,512,128]
for all. Default: [0, 0, 32, 32]
--max-bins int Maximum number of bins to consider, 0 for all bins.
Default: 0
--batch-size int [int ...]
batch size: [cell,x,y]. Default: [1, 8, 8]
--fit-range int [int ...]
range of a histogram considered for fitting in ADU.
Dynamically evaluated in case [0,0]. Default: [0, 0]
--n-peaks-fit int Number of gaussian peaks to fit including noise peak.
Default: 4
--fix-peaks Fix distance between photon peaks. Default: False
--no-fix-peaks Opposite of --fix-peaks (default: False)
--do-minos This is additional feature of minuit to evaluate
errors.. Default: False
--no-do-minos Opposite of --do-minos (default: False)
--sigma-limit float If >0, repeat fit keeping only bins within mu +-
sigma_limit*sigma. Default: 0.0
--mem-cells int number of memory cells used, negative values for auto-
detection.. Default: -1
--bias-voltage int Bias voltage.. Default: 300
--acq-rate float the detector acquisition rate, use 0 to try to auto-
determine.. Default: 0.0
--gain-setting int the gain setting, negative values for auto-detection..
Default: -1
--photon-energy float
photon energy in keV.. Default: 8.05
--integration-time int
integration time, negative values for auto-detection..
Default: -1
required arguments:
--in-folder str the folder to read histograms from, required. Default:
None
--out-folder str the folder to output to, required. Default: None
Histogramming of AGIPD FF data¶
Author: European XFEL Detector Group, Version: 1.0
Offline Calibration for the AGIPD Detector
To invoke this notebook and display help use:
xfel-calibrate AGIPD FF_HISTS --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
[--sequences str [str ...]] [--modules str [str ...]]
--runs int [int ...] [--cells-list str [str ...]]
[--karabo-id str] [--karabo-da str [str ...]]
[--receiver-id str] [--path-template str] [--h5path str]
[--n-bins-adu int] [--h-range int [int ...]]
[--n-cells int] [--hist-std-tresh int]
[--hist-mean-tresh int] [--h-sums-tresh int]
[--n-cores-hists int] [--n-cores-files int]
AGIPD FF_HISTS
# Histogramming of AGIPD FF data
,
Author: European XFEL Detector Group, Version: 1.0,
,
Offline Calibration for the AGIPD Detector
positional arguments:
AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
FF_HISTS Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequences str [str ...]
module to consider, set to -1 for all, range allowed.
Default: [-1]
--modules str [str ...]
module to consider, range allowed. Default: [5]
--cells-list str [str ...]
list of lists or any python expressions, which can be
converted to a list. E.g. 'range(0,15,5)'
'list(range(0,5))+list(range(50,60))'. Default:
['range(0,15)']
--karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1
--karabo-da str [str ...]
a list of data aggregators names, Default [-1] for
selecting all data aggregators. Default: ['-1']
--receiver-id str inset for receiver devices. Default: {}CH0
--path-template str the template to use to access data. Default:
CORR-R{:04d}-AGIPD{:02d}-S{}.h5
--h5path str path in the HDF5 file to images. Default:
INSTRUMENT/{}/DET/{}:xtdf/image/
--n-bins-adu int number of bins per ADU. Default: 1
--h-range int [int ...]
range of the histogram in ADU. Default: [-50, 450]
--n-cells int total number of memory cells (used to create summary
file). Default: 202
--hist-std-tresh int Threshold for histogram standard deviation. Default:
10
--hist-mean-tresh int
Threshold for histogram mean. Default: 15
--h-sums-tresh int Threshold for number of entries in histograms.
Default: 100
--n-cores-hists int Number of processes (cores) to create histograms.
Default: 20
--n-cores-files int Number of processes (cores) to read files. Default: 10
required arguments:
--in-folder str the folder to read data from, required. Default: None
--out-folder str the folder to output to, required. Default: None
--runs int [int ...] list of run numbers, required. Default: None
Characterize AGIPD Pulse Capacitor Data¶
Author: Detector Operations Group, Version 1.0
The following code characterizes AGIPD gain via data take with the pulse capacitor source (PCS). The PCS allows scanning through the high and medium gains of AGIPD, by subsequently intecreasing the number of charge pulses from a on-ASIC capicitor, thus increasing the charge a pixel sees in a given integration time.
Because induced charge does not originate from X-rays on the sensor, the gains evaluated here will later need to be rescaled with gains deduced from X-ray data.
PCS data is organized into multiple runs, as the on-ASIC current source cannot supply all pixels of a given module with charge at the same time. Hence, only certain pixel rows will have seen charge for a given image. These rows then first need to be combined into single module images again. This script uses new style of merging, which does not support old data format.
We then use a K-means clustering algorithm to identify components in the resulting per-pixel data series, matching to three general regions:
- a high gain slope
- a transition region, where gain switching occurs
- a medium gain slope.
The same regions are present in the gain-bit data and are used to deduce the switching threshold.
The resulting slopes are then fitted with a linear function and a combination of a linear and exponential decay function to determine the relative gains of the pixels with respect to the module. Additionally, we deduce masks for bad pixels form the data.
To invoke this notebook and display help use:
xfel-calibrate AGIPD PC --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str --runs str [str ...]
[--n-sequences int] [--modules str [str ...]]
[--karabo-da str [str ...]] [--karabo-da-control str]
[--karabo-id-control str] [--karabo-id str]
[--ctrl-source-template str]
[--instrument-source-template str]
[--receiver-template str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--delta-time int] [--cal-db-interface str]
[--local-output | --no-local-output]
[--db-output | --no-db-output] [--bias-voltage int]
[--mem-cells int] [--acq-rate float]
[--gain-setting int] [--integration-time int]
[--FF-gain float] [--fit-hook | --no-fit-hook]
[--high-res-badpix-3d | --no-high-res-badpix-3d]
[--hg-range int [int ...]] [--mg-range int [int ...]]
[--sigma-dev-cut float]
AGIPD PC
# Characterize AGIPD Pulse Capacitor Data #
,
Author: Detector Operations Group, Version 1.0,
,
The following code characterizes AGIPD gain via data take with the pulse
capacitor source (PCS). The PCS allows scanning through the high and medium
gains of AGIPD, by subsequently intecreasing the number of charge pulses from
a on-ASIC capicitor, thus increasing the charge a pixel sees in a given
integration time.,
,
Because induced charge does not originate from X-rays on the sensor, the
gains evaluated here will later need to be rescaled with gains deduced from
X-ray data.,
,
PCS data is organized into multiple runs, as the on-ASIC current source
cannot supply all pixels of a given module with charge at the same time.
Hence, only certain pixel rows will have seen charge for a given image. These
rows then first need to be combined into single module images again. This
script uses new style of merging, which does not support old data format.,
,
We then use a K-means clustering algorithm to identify components in the
resulting per-pixel data series, matching to three general regions:,
,
* a high gain slope,
* a transition region, where gain switching occurs,
* a medium gain slope.,
,
The same regions are present in the gain-bit data and are used to deduce the
switching threshold.
,
The resulting slopes are then fitted with a linear function and a combination
of a linear and exponential decay function to determine the relative gains of
the pixels with respect to the module. Additionally, we deduce masks for bad
pixels form the data.
positional arguments:
AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
PC Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
The ipcluster profile to use. Default: noDB
--n-sequences int number of sequence files, starting for 0 to evaluate.
Default: 5
--modules str [str ...]
modules to work on, required, range allowed. Default:
[-1]
--karabo-da str [str ...]
Default: ['all']
--karabo-da-control str
karabo DA for control infromation. Default:
AGIPD1MCTRL00
--karabo-id-control str
karabo-id for the control device e.g.
"MID_EXP_AGIPD1M1", or "SPB_IRU_AGIPD1M1". Default:
SPB_IRU_AGIPD1M1
--karabo-id str Default: SPB_DET_AGIPD1M-1
--ctrl-source-template str
path to control information. Default: {}/MDL/FPGA_COMP
--instrument-source-template str
path in the HDF5 file to images. Default:
{}/DET/{}:xtdf
--receiver-template str
inset for receiver devices. Default: {}CH0
--use-dir-creation-date
Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--delta-time int offset to the creation time (e.g. useful in case we
want to force the system to use diff. dark constants).
Default: 0
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8019
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--bias-voltage int detector bias voltage, negative values for auto-
detection.. Default: -1
--mem-cells int number of memory cells used, negative values for auto-
detection.. Default: -1
--acq-rate float the detector acquisition rate, negative values for
auto-detection.. Default: -1.0
--gain-setting int gain setting can have value 0 or 1, negative values
for auto-detection.. Default: -1
--integration-time int
integration time, negative values for auto-detection..
Default: -1
--FF-gain float ADU/keV, gain from FF to convert ADU to keV. Default:
7.3
--fit-hook fit a hook function to medium gain slope --> run
without hook. Default: False
--no-fit-hook Opposite of --fit-hook (default: False)
--high-res-badpix-3d set this to True if you need high-resolution 3d bad
pixel plots. Runtime: ~ 1h. Default: False
--no-high-res-badpix-3d
Opposite of --high-res-badpix-3d (default: False)
--hg-range int [int ...]
[0,-150] range for linear fit. If upper edge is
negative use clustering result (bound_hg - 20).
Default: [30, 210]
--mg-range int [int ...]
[-350,600] range for linear fit. If lower edge is
negative use clustering result (bound_mg + 20).
Default: [-277, 600]
--sigma-dev-cut float
parameters outside the range median +-
sigma_dev_cut*MAD are replaced with the median.
Default: 5.0
required arguments:
--in-folder str path to input data, required. Default: None
--out-folder str path to output to, required. Default: None
--runs str [str ...] runs to use, required, range allowed. Default: None
DSSC¶
DSSC Offline Correction¶
Author: European XFEL Detector Group, Version: 1.0
Offline Calibration for the DSSC Detector
To invoke this notebook and display help use:
xfel-calibrate DSSC CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--sequences int [int ...]]
[--modules str [str ...]] --run int [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-id str]
[--path-template str] [--h5path str] [--h5path-idx str]
[--slow-data-pattern str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--mem-cells int] [--overwrite | --no-overwrite]
[--max-pulses int] [--bias-voltage int]
[--sequences-per-node int] [--chunk-size-idim int]
[--mask-noisy-asic float] [--mask-cold-asic float]
[--noisy-pix-threshold float] [--geo-file str]
[--dinstance str]
[--slow-data-aggregators int [int ...]]
[--slow-data-path str]
DSSC CORRECT
# DSSC Offline Correction #
,
Author: European XFEL Detector Group, Version: 1.0,
,
Offline Calibration for the DSSC Detector
positional arguments:
DSSC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
The ipcluster profile to use. Default: noDB
--sequences int [int ...]
sequence files to evaluate.. Default: [-1]
--modules str [str ...]
modules to correct, set to -1 for all, range allowed.
Default: [-1]
--karabo-id str karabo karabo_id. Default: SQS_DET_DSSC1M-1
--karabo-da str [str ...]
a list of data aggregators names, Default [-1] for
selecting all data aggregators. Default: ['-1']
--receiver-id str inset for receiver devices. Default: {}CH0
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{:05d}.h5
--h5path str path in the HDF5 file to images. Default:
INSTRUMENT/{}/DET/{}:xtdf/image
--h5path-idx str path in the HDF5 file to images. Default:
/INDEX/{}/DET/{}:xtdf/image
--slow-data-pattern str
Default: RAW-R{}-DA{}-S00000.h5
--use-dir-creation-date
use the creation data of the input dir for database
queries. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8020#8025
--cal-db-timeout int in milli seconds. Default: 300000
--mem-cells int number of memory cells used, set to 0 to automatically
infer. Default: 0
--overwrite set to True if existing data should be overwritten.
Default: True
--no-overwrite Opposite of --overwrite (default: True)
--max-pulses int maximum number of pulses per train. Default: 800
--bias-voltage int detector bias voltage. Default: 100
--sequences-per-node int
number of sequence files per cluster node if run as
slurm job, set to 0 to not run SLURM parallel.
Default: 1
--chunk-size-idim int
chunking size of imaging dimension, adjust if user
software is sensitive to this.. Default: 1
--mask-noisy-asic float
set to a value other than 0 and below 1 to mask entire
ADC if fraction of noisy pixels is above. Default:
0.25
--mask-cold-asic float
mask cold ASICS if number of pixels with negligable
standard deviation is larger than this fraction.
Default: 0.25
--noisy-pix-threshold float
threshold above which ap pixel is considered noisy..
Default: 1.0
--geo-file str detector geometry file. Default:
/gpfs/exfel/data/scratch/xcal/dssc_geo_june19.h5
--dinstance str Default: DSSC1M1
--slow-data-aggregators int [int ...]
quadrant/aggregator. Default: [1, 2, 3, 4]
--slow-data-path str Default: SQS_NQS_DSSC/FPGA/PPT_Q
required arguments:
--in-folder str path to input data, required. Default: None
--out-folder str path to output to, required. Default: None
--run int runs to process, required. Default: None
DSSC Characterize Dark Images¶
Author: S. Hauf, Version: 0.1
The following code analyzes a set of dark images taken with the DSSC detector to deduce detector offsets and noise. Data for the detector is presented in one run and don’t acquire multiple gain stages.
The notebook explicitely does what pyDetLib provides in its offset calculation method for streaming data.
To invoke this notebook and display help use:
xfel-calibrate DSSC DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--sequences int [int ...]]
[--modules int [int ...]] --run int [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-id str]
[--path-template str] [--h5path str] [--h5path-idx str]
[--slow-data-pattern str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--local-output | --no-local-output]
[--db-output | --no-db-output] [--mem-cells int]
[--bias-voltage int] [--rawversion int]
[--thresholds-offset-sigma float]
[--thresholds-offset-hard int [int ...]]
[--thresholds-noise-sigma float]
[--thresholds-noise-hard float [float ...]]
[--offset-numpy-algorithm str] [--instrument str]
[--high-res-badpix-3d | --no-high-res-badpix-3d]
[--slow-data-aggregators int [int ...]]
[--slow-data-path str] [--operation-mode str]
DSSC DARK
# DSSC Characterize Dark Images #
,
Author: S. Hauf, Version: 0.1,
,
The following code analyzes a set of dark images taken with the DSSC detector
to deduce detector offsets and noise. Data for the detector is presented in
"one run and dont acquire multiple gain stages. ",
,
The notebook explicitely does what pyDetLib provides in its offset
calculation method for streaming data.
positional arguments:
DSSC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
The ipcluster profile to use. Default: noDB
--sequences int [int ...]
sequence files to evaluate.. Default: [0]
--modules int [int ...]
modules to run for. Default: [-1]
--karabo-id str karabo karabo_id. Default: SQS_DET_DSSC1M-1
--karabo-da str [str ...]
a list of data aggregators names, Default [-1] for
selecting all data aggregators. Default: ['-1']
--receiver-id str inset for receiver devices. Default: {}CH0
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{:05d}.h5
--h5path str path in the HDF5 file to images. Default:
/INSTRUMENT/{}/DET/{}:xtdf/image
--h5path-idx str path in the HDF5 file to images. Default:
/INDEX/{}/DET/{}:xtdf/image
--slow-data-pattern str
Default: RAW-R{}-DA{}-S00000.h5
--use-dir-creation-date
use the dir creation date for determining the creation
time. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8020
--cal-db-timeout int timeout on caldb requests". Default: 3000000
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--mem-cells int number of memory cells used, set to 0 to automatically
infer. Default: 0
--bias-voltage int detector bias voltage. Default: 100
--rawversion int RAW file format version. Default: 2
--thresholds-offset-sigma float
thresholds in terms of n sigma noise for offset
deduced bad pixels. Default: 3.0
--thresholds-offset-hard int [int ...]
thresholds in absolute ADU terms for offset deduced
bad pixels,. Default: [4, 125]
--thresholds-noise-sigma float
thresholds in terms of n sigma noise for offset
deduced bad pixels. Default: 3.0
--thresholds-noise-hard float [float ...]
thresholds in absolute ADU terms for offset deduced
bad pixels. Default: [0.001, 3]
--offset-numpy-algorithm str
Default: mean
--instrument str the instrument. Default: SQS
--high-res-badpix-3d set this to True if you need high-resolution 3d bad
pixel plots. Runtime: ~ 1h. Default: False
--no-high-res-badpix-3d
Opposite of --high-res-badpix-3d (default: False)
--slow-data-aggregators int [int ...]
quadrant/aggregator. Default: [1, 2, 3, 4]
--slow-data-path str Default: SQS_NQS_DSSC/FPGA/PPT_Q
--operation-mode str Detector operation mode, optional. Default:
required arguments:
--in-folder str path to input data, required. Default: None
--out-folder str path to output to, required. Default: None
--run int run number in which data was recorded, required.
Default: None
EPIX100¶
ePix100 Data Correction¶
Author: European XFEL Detector Group, Version: 2.0
The following notebook provides data correction of images acquired with the ePix100 detector.
To invoke this notebook and display help use:
xfel-calibrate EPIX100 CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
[--sequences str [str ...]] [--sequences-per-node int]
--run int [--karabo-id str] [--karabo-da str]
[--db-module str] [--receiver-template str]
[--path-template str] [--instrument-source-template str]
[--chunk-size-idim int] [--limit-images int]
[--cal-db-interface str] [--cal-db-timeout int]
[--creation-time str] [--bias-voltage int]
[--in-vacuum | --no-in-vacuum] [--integration-time int]
[--fix-temperature int] [--gain-photon-energy float]
[--photon-energy float]
[--pattern-classification | --no-pattern-classification]
[--relative-gain | --no-relative-gain]
[--absolute-gain | --no-absolute-gain]
[--common-mode | --no-common-mode] [--cm-min-frac float]
[--cm-noise-sigma float]
[--split-evt-primary-threshold float]
[--split-evt-secondary-threshold float]
[--split-evt-mip-threshold float]
EPIX100 CORRECT
# ePix100 Data Correction
,
Author: European XFEL Detector Group, Version: 2.0,
,
The following notebook provides data correction of images acquired with the
ePix100 detector.
positional arguments:
EPIX100 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequences str [str ...]
sequences to correct, set to -1 for all, range
allowed. Default: [-1]
--sequences-per-node int
number of sequence files per cluster node if run as
slurm job, set to 0 to not run SLURM parallel.
Default: 1
--karabo-id str karabo karabo_id. Default: MID_EXP_EPIX-1
--karabo-da str data aggregators. Default: EPIX01
--db-module str module id in the database. Default:
--receiver-template str
detector receiver template for accessing raw data
files. Default: RECEIVER
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--instrument-source-template str
instrument detector data source in h5files. Default:
{}/DET/{}:daqOutput
--chunk-size-idim int
H5 chunking size of output data. Default: 1
--limit-images int ONLY FOR TESTING. process only first N images, 0 -
process all.. Default: 0
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8015#8025
--cal-db-timeout int timeout on caldb requests. Default: 300000
--creation-time str The timestamp to use with Calibration DBe. Required
Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
11:02:41. Default:
--bias-voltage int bias voltage. Default: 200
--in-vacuum detector operated in vacuum. Default: False
--no-in-vacuum Opposite of --in-vacuum (default: False)
--integration-time int
Detector integration time, Default value -1 to use the
value from the slow data.. Default: -1
--fix-temperature int
fixed temperature value in Kelvin, Default value -1 to
use the value from files.. Default: -1
--gain-photon-energy float
Photon energy used for gain calibration. Default: 9.0
--photon-energy float
Photon energy to calibrate in number of photons, 0 for
calibration in keV. Default: 0.0
--pattern-classification
do clustering.. Default: True
--no-pattern-classification
Opposite of --pattern-classification (default: True)
--relative-gain Apply relative gain correction.. Default: True
--no-relative-gain Opposite of --relative-gain (default: True)
--absolute-gain Apply absolute gain correction (implies relative
gain).. Default: True
--no-absolute-gain Opposite of --absolute-gain (default: True)
--common-mode Apply common mode correction.. Default: True
--no-common-mode Opposite of --common-mode (default: True)
--cm-min-frac float No CM correction is performed if after masking the
ratio of good pixels falls below this. Default: 0.25
--cm-noise-sigma float
CM correction noise standard deviation. Default: 5.0
--split-evt-primary-threshold float
primary threshold for split event correction. Default:
7.0
--split-evt-secondary-threshold float
secondary threshold for split event correction.
Default: 5.0
--split-evt-mip-threshold float
minimum ionizing particle threshold. Default: 1000.0
required arguments:
--in-folder str input folder, required. Default: None
--out-folder str output folder, required. Default: None
--run int which run to read data from, required. Default: None
ePix100 Dark Characterization¶
Author: European XFEL Detector Group, Version: 2.0
The following notebook provides dark image analysis and calibration constants of the ePix100 detector.
Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.
Noise and bad pixels maps are calculated independently for each of the 4 ASICs of ePix100, since their noise behaviour can be significantly different.
Common mode correction can be applied to increase sensitivity to noise related bad pixels. Common mode correction is achieved by subtracting the median of all pixels that are read out at the same time along a row/column. This is done in an iterative process, by which a new bad pixels map is calculated and used to mask data as the common mode values across the rows/columns is updated.
Resulting maps are saved as .h5 files for a later use and injected to calibration DB.
To invoke this notebook and display help use:
xfel-calibrate EPIX100 DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
[--sequence int] --run int [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-template str]
[--path-template str] [--instrument-source-template str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--db-output | --no-db-output]
[--local-output | --no-local-output]
[--bias-voltage int] [--in-vacuum | --no-in-vacuum]
[--fix-integration-time int] [--fix-temperature int]
[--temp-limits int] [--badpixel-threshold-sigma float]
[--CM-N-iterations int] [--min-trains int]
[--max-trains int] [--operation-mode str]
[--db-module str]
EPIX100 DARK
# ePix100 Dark Characterization
,
Author: European XFEL Detector Group, Version: 2.0,
,
The following notebook provides dark image analysis and calibration constants
of the ePix100 detector.,
,
Dark characterization evaluates offset and noise of the detector and gives
information about bad pixels.
,
Noise and bad pixels maps are calculated independently for each of the 4
ASICs of ePix100, since their noise behaviour can be significantly different.,
,
Common mode correction can be applied to increase sensitivity to noise
related bad pixels. Common mode correction is achieved by subtracting the
median of all pixels that are read out at the same time along a row/column.
This is done in an iterative process, by which a new bad pixels map is
calculated and used to mask data as the common mode values across the
rows/columns is updated.,
,
Resulting maps are saved as .h5 files for a later use and injected to
calibration DB.
positional arguments:
EPIX100 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequence int sequence file to use. Default: 0
--karabo-id str karabo karabo_id. Default: HED_IA1_EPX100-1
--karabo-da str [str ...]
data aggregators. Default: ['EPIX01']
--receiver-template str
detector receiver template for accessing raw data
files. Default: RECEIVER
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--instrument-source-template str
instrument detector data source in h5files. Default:
{}/DET/{}:daqOutput
--use-dir-creation-date
Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8020
--cal-db-timeout int timeout on caldb requests. Default: 300000
--db-output Output constants to the calibration database. Default:
False
--no-db-output Opposite of --db-output (default: False)
--local-output Output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--bias-voltage int Bias voltage. Default: 200
--in-vacuum Detector operated in vacuum. Default: False
--no-in-vacuum Opposite of --in-vacuum (default: False)
--fix-integration-time int
Integration time. Set to -1 to read from .h5 file.
Default: -1
--fix-temperature int
Fixed temperature in Kelvin. Set to -1 to read from
.h5 file. Default: -1
--temp-limits int Limit for parameter Operational temperature. Default:
5
--badpixel-threshold-sigma float
Bad pixels defined by values outside n times this std
from median. Default: 5. Default: 5.0
--CM-N-iterations int
Number of iterations for common mode correction. Set
to 0 to skip it. Default: 2
--min-trains int Minimum number of trains that should be available to
process dark constants. Default 1.. Default: 1
--max-trains int Maximum number of trains to use for processing dark
constants. Set to 0 to use all available trains..
Default: 1000
--operation-mode str Detector operation mode, optional. Default:
--db-module str ID of module in calibration database, this parameter
is ignore in the notebook. TODO: remove from
calibration_configurations.. Default:
required arguments:
--in-folder str input folder, required. Default: None
--out-folder str output folder, required. Default: None
--run int which run to read data from, required. Default: None
EPIX10K¶
ePIX10K Data Correction¶
Authors: M. Karnevskiy, S. Hauf, Version 1.0
The following notebook provides Offset correction of images acquired with the ePix10K detector.
To invoke this notebook and display help use:
xfel-calibrate EPIX10K CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--sequences str [str ...]] --run
int [--karabo-id str] [--karabo-da str]
[--receiver-id str] [--path-template str] [--h5path str]
[--h5path-t str] [--h5path-cntrl str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--cpuCores int] [--chunk-size-idim int]
[--overwrite | --no-overwrite] [--limit-images int]
[--db-module str] [--sequences-per-node int]
[--bias-voltage int] [--in-vacuum | --no-in-vacuum]
[--fix-temperature float]
[--split-evt-primary-threshold float]
[--split-evt-secondary-threshold float]
[--split-evt-mip-threshold float]
EPIX10K CORRECT
# ePIX10K Data Correction #
,
Authors: M. Karnevskiy, S. Hauf, Version 1.0,
,
The following notebook provides Offset correction of images acquired with the
ePix10K detector.
positional arguments:
EPIX10K The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
ipcluster profile to use. Default: noDB
--sequences str [str ...]
sequences to correct, set to -1 for all, range
allowed. Default: [-1]
--karabo-id str karabo karabo_id. Default: HED_IA1_EPIX10K-1
--karabo-da str data aggregators. Default: EPIX03
--receiver-id str inset for receiver devices. Default: RECEIVER
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--h5path str path in the HDF5 file to images. Default:
/INSTRUMENT/{}/DET/{}:daqOutput/data/image
--h5path-t str path to find temperature at. Default:
/INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp
--h5path-cntrl str path to control data. Default: /CONTROL/{}/DET
--use-dir-creation-date
date constants injected before directory creation
time. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8015#8025
--cal-db-timeout int timeout on caldb requests. Default: 30000000
--cpuCores int Specifies the number of running cpu cores. Default: 4
--chunk-size-idim int
H5 chunking size of output data. Default: 1
--overwrite overwrite output folder. Default: True
--no-overwrite Opposite of --overwrite (default: True)
--limit-images int process only first N images, 0 - process all. Default:
0
--db-module str module id in the database. Default: ePix10K_M43
--sequences-per-node int
number of sequence files per cluster node if run as
slurm job, set to 0 to not run SLURM parallel.
Default: 1
--bias-voltage int bias voltage. Default: 200
--in-vacuum detector operated in vacuum. Default: False
--no-in-vacuum Opposite of --in-vacuum (default: False)
--fix-temperature float
fix temperature to this value. Default: 290.0
--split-evt-primary-threshold float
primary threshold for split event correction. Default:
7.0
--split-evt-secondary-threshold float
secondary threshold for split event correction.
Default: 5.0
--split-evt-mip-threshold float
minimum ionizing particle threshold. Default: 1000.0
required arguments:
--in-folder str input folder, required. Default: None
--out-folder str output folder, required. Default: None
--run int which run to read data from, required. Default: None
ePix10K Dark Characterization¶
Author: M. Karnevskiy, Version 1.0
The following notebook provides dark image analysis of the ePix10K detector.
Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Resulting maps are saved as .h5 files for a latter use and injected to the calibration DB.
To invoke this notebook and display help use:
xfel-calibrate EPIX10K DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--sequence int] --run int
[--karabo-id str] [--karabo-da str [str ...]]
[--receiver-id str] [--path-template str] [--h5path str]
[--h5path-t str] [--h5path-cntrl str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--db-output | --no-db-output]
[--local-output | --no-local-output] [--temp-limits int]
[--number-dark-frames int] [--db-module str]
[--bias-voltage int] [--in-vacuum | --no-in-vacuum]
[--fix-temperature float] [--operation-mode str]
EPIX10K DARK
# ePix10K Dark Characterization #
,
Author: M. Karnevskiy, Version 1.0,
,
The following notebook provides dark image analysis of the ePix10K detector.,
,
Dark characterization evaluates offset and noise of the detector and gives
information about bad pixels. Resulting maps are saved as .h5 files for a
latter use and injected to the calibration DB.
positional arguments:
EPIX10K The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
ipcluster profile to use. Default: noDB
--sequence int sequence file to use. Default: 0
--karabo-id str karabo karabo_id. Default: HED_IA1_EPIX10K-1
--karabo-da str [str ...]
data aggregators. Default: ['EPIX03']
--receiver-id str inset for receiver devices. Default: RECEIVER
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--h5path str path in the HDF5 file to images. Default:
/INSTRUMENT/{}/DET/{}:daqOutput/data/image/pixels
--h5path-t str path to find temperature at. Default:
/INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp
--h5path-cntrl str path to control data. Default: /CONTROL/{}/DET
--use-dir-creation-date
Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8020
--cal-db-timeout int timeout on caldb requests. Default: 300000
--db-output Output constants to the calibration database. Default:
False
--no-db-output Opposite of --db-output (default: False)
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--temp-limits int limit for parameter Operational temperature. Default:
5
--number-dark-frames int
number of images to be used, if set to 0 all available
images are used. Default: 0
--db-module str detector karabo_id. Default: ePix10K_M43
--bias-voltage int bias voltage. Default: 200
--in-vacuum detector operated in vacuum. Default: False
--no-in-vacuum Opposite of --in-vacuum (default: False)
--fix-temperature float
fix temperature to this value. Default: 290.0
--operation-mode str Detector operation mode, optional. Default:
required arguments:
--in-folder str input folder, required. Default: None
--out-folder str output folder, required. Default: None
--run int which run to read data from, required. Default: None
FASTCCD¶
FastCCD Data Correction¶
Authors: I. Klačková, S. Hauf, Version 1.0
The following notebook provides correction of images acquired with the FastCCD.
To invoke this notebook and display help use:
xfel-calibrate FASTCCD CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--sequences str [str ...]]
[--run int] [--karabo-da str] [--karabo-id str]
[--receiver-id str] [--path-template str] [--h5path str]
[--h5path-t str] [--h5path-cntrl str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--cpuCores int] [--operation-mode str]
[--split-evt-primary-threshold float]
[--split-evt-secondary-threshold float]
[--split-evt-mip-threshold float]
[--chunk-size-idim int] [--overwrite | --no-overwrite]
[--sequences-per-node int] [--limit-images int]
[--time-offset-days int]
[--photon-energy-gain-map float]
[--fix-temperature float]
[--flipped-between str [str ...]] [--temp-limits int]
[--commonModeAxis int]
[--only-offset | --no-only-offset]
[--cti-corr | --no-cti-corr]
[--relgain-corr | --no-relgain-corr]
[--common-mode-corr | --no-common-mode-corr]
[--correct-offset-drift | --no-correct-offset-drift]
[--do-pattern-classification | --no-do-pattern-classification]
FASTCCD CORRECT
# FastCCD Data Correction
,
Authors: I. Klačková, S. Hauf, Version 1.0,
,
The following notebook provides correction of images acquired with the
FastCCD.
positional arguments:
FASTCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
ipcluster profile to use. Default: noDB
--sequences str [str ...]
sequences to correct, set to -1 for all, range
allowed. Default: [-1]
--run int run number. Default: 277
--karabo-da str data aggregators. Default: DA05
--karabo-id str karabo prefix of PNCCD devices. Default:
SCS_CDIDET_FCCD2M
--receiver-id str inset for receiver devices. Default: FCCD
--path-template str path template in hdf5 file. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--h5path str path in HDF5 file. Default:
/INSTRUMENT/{}/DAQ/{}:daqOutput/data/image
--h5path-t str temperature path in HDF5 file. Default:
/CONTROL/{}/CTRL/LSLAN/inputA/crdg/value
--h5path-cntrl str path to control data. Default: /RUN/{}/DET/FCCD
--use-dir-creation-date
use dir creation data for calDB queries. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8015#8025
--cal-db-timeout int timeout on caldb requests. Default: 300000000
--cpuCores int Specifies the number of running cpu cores. Default: 16
--operation-mode str FS stands for frame-store and FF for full-frame
opeartion. Default: FF
--split-evt-primary-threshold float
primary threshold for split event classification in
terms of n sigma noise. Default: 7.0
--split-evt-secondary-threshold float
secondary threshold for split event classification in
terms of n sigma noise. Default: 4.0
--split-evt-mip-threshold float
MIP threshold for event classification. Default:
1000.0
--chunk-size-idim int
H5 chunking size of output data. Default: 1
--overwrite overwrite existing files. Default: True
--no-overwrite Opposite of --overwrite (default: True)
--sequences-per-node int
sequences to correct per node. Default: 1
--limit-images int limit images per file. Default: 0
--time-offset-days int
offset in days for calibration parameters. Default: 0
--photon-energy-gain-map float
energy in keV. Default: 5.9
--fix-temperature float
fix temperature to this value, set to 0 to use slow
control value. Default: 0.0
--flipped-between str [str ...]
detector was flipped during this timespan. Default:
['2019-02-01', '2019-04-02']
--temp-limits int limits within which temperature is considered the
same. Default: 5
--commonModeAxis int Axis along which common mode will be calculated (0:
along rows, 1: along columns). Default: 1
--only-offset Only apply offset correction. Default: False
--no-only-offset Opposite of --only-offset (default: False)
--cti-corr Apply CTI correction. Default: False
--no-cti-corr Opposite of --cti-corr (default: False)
--relgain-corr Apply relative gain correction. Default: True
--no-relgain-corr Opposite of --relgain-corr (default: True)
--common-mode-corr Apply commonMode correction. Default: False
--no-common-mode-corr
Opposite of --common-mode-corr (default: False)
--correct-offset-drift
correct for offset drifts. Default: False
--no-correct-offset-drift
Opposite of --correct-offset-drift (default: False)
--do-pattern-classification
classify split events. Default: True
--no-do-pattern-classification
Opposite of --do-pattern-classification (default:
True)
required arguments:
--in-folder str input folder, required. Default: None
--out-folder str output folder, required. Default: None
FastCCD Dark Characterization¶
Author: I. Klačková, S. Hauf, K. Setoodehnia and M. Cascella
The following notebook provides dark image analysis of the FastCCD detector.
Dark characterization evaluates offset and noise of the FastCCD detector, corrects the noise for Common Mode (CM), and defines bad pixels relative to offset and CM corrected noise. Bad pixels are then excluded and CM corrected noise is recalculated excluding the bad pixels. Resulting offset and CM corrected noise maps, as well as the bad pixel map are sent to the calibration database.
To invoke this notebook and display help use:
xfel-calibrate FASTCCD DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--sequence int] --run int
[--karabo-da str [str ...]] [--karabo-id str]
[--receiver-id str] [--path-template str] [--h5path str]
[--h5path-t str] [--h5path-cntrl str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--db-output | --no-db-output]
[--local-output | --no-local-output]
[--number-dark-frames int] [--operation-mode str]
[--temp-limits int] [--bad-pixel-offset-sigma float]
[--bad-pixel-noise-sigma float] [--sigmaNoise float]
[--fix-temperature float] [--chunkSize int]
[--cpuCores int] [--commonModeAxis int]
[--run-parallel | --no-run-parallel]
[--ADU-to-electron-upper-hg float]
[--ADU-to-electron-lower-hg float]
[--ADU-to-electron-upper-mg float]
[--ADU-to-electron-lower-mg float]
[--ADU-to-electron-upper-lg float]
[--ADU-to-electron-lower-lg float]
FASTCCD DARK
# FastCCD Dark Characterization
,
Author: I. Klačková, S. Hauf, K. Setoodehnia and M. Cascella,
,
The following notebook provides dark image analysis of the FastCCD detector.,
,
Dark characterization evaluates offset and noise of the FastCCD detector,
corrects the noise for Common Mode (CM), and defines bad pixels relative to
offset and CM corrected noise. Bad pixels are then excluded and CM corrected
noise is recalculated excluding the bad pixels. Resulting offset and CM
corrected noise maps, as well as the bad pixel map are sent to the
calibration database.
positional arguments:
FASTCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
ipcluster profile to use. Default: noDB
--sequence int sequence file to use. Default: 0
--karabo-da str [str ...]
data aggregators. Default: ['DA05']
--karabo-id str karabo prefix of PNCCD devices. Default:
SCS_CDIDET_FCCD2M
--receiver-id str inset for receiver devices. Default: FCCD
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--h5path str path to the data in the HDF5 file. Default:
/INSTRUMENT/{}/DAQ/{}:daqOutput/data/image/pixels
--h5path-t str path to find temperature. Default:
/CONTROL/{}/CTRL/LSLAN/inputA/crdg/value
--h5path-cntrl str path to find control data. Default: /RUN/{}/DET/FCCD
--use-dir-creation-date
To be used to retrieve calibration constants later on
(for database time derivation). Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the calibration database interface to use. Default:
tcp://max-exfl016:8020
--cal-db-timeout int timeout on calibration database requests. Default:
300000
--db-output Output constants to the calibration database. Default:
False
--no-db-output Opposite of --db-output (default: False)
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--number-dark-frames int
number of images to be used, if set to 0 all available
images are used. Default: 0
--operation-mode str FS stands for frame-store and FF for full-frame
operation.. Default: FF
--temp-limits int to find calibration constants later on, the sensor
temperature is allowed to vary by 5 units. Default: 5
--bad-pixel-offset-sigma float
Any pixel whose offset is beyond 5 standard
deviations, is a bad pixel. Default: 5.0
--bad-pixel-noise-sigma float
Any pixel whose noise is beyond 5 standard deviations,
is a bad pixel. Default: 5.0
--sigmaNoise float Any pixel whose signal exceeds 'sigmaNoise'*noiseCM
(common mode corrected noise) will be masked. Default:
5.0
--fix-temperature float
Fixed operation temperature in Kelvins. If set to 0,
mean value of the data file's temperature is. Default:
0.0
--chunkSize int Number of images to read per chunk. Default: 100
--cpuCores int Specifies the number of running cpu cores. Default: 40
--commonModeAxis int Axis along which common mode will be calculated (0:
along rows, 1: along columns). Default: 1
--run-parallel For parallel computation. Default: True
--no-run-parallel Opposite of --run-parallel (default: True)
--ADU-to-electron-upper-hg float
for upper hemisphere and high gain. Default: 6.3
--ADU-to-electron-lower-hg float
for lower hemisphere and high gain. Default: 6.4
--ADU-to-electron-upper-mg float
for upper hemisphere and medium gain. Default: 23.4
--ADU-to-electron-lower-mg float
for lower hemisphere and medium gain. Default: 23.4
--ADU-to-electron-upper-lg float
for upper hemisphere and low gain. Default: 49.3
--ADU-to-electron-lower-lg float
for lower hemisphere and low gain. Default: 47.3
required arguments:
--in-folder str input folder, required. Default: None
--out-folder str output folder, required. Default: None
--run int which run to read data from, required. Default: None
GENERIC¶
Constants from DB to HDF5¶
Version 0.1, Author: S. Hauf
Currently available instances are LPD1M1 and AGIPD1M1
To invoke this notebook and display help use:
xfel-calibrate GENERIC DB_TO_H5 --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --detector-instance str --out-file
str [--valid-at str] [--cal-db-interface str]
[--modules str [str ...]]
GENERIC DB_TO_H5
# Constants from DB to HDF5 #
,
Version 0.1, Author: S. Hauf,
,
Currently available instances are LPD1M1 and AGIPD1M1
positional arguments:
GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DB_TO_H5 Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--valid-at str ISO formatted date for which constants shoudl be
valid. Leave empty to get most current ones. Default:
--cal-db-interface str
Default: tcp://max-exfl015:5005
--modules str [str ...]
modules to get data from, in terms of numerical
quadrant indices, range allowed. Default: [-1]
required arguments:
--detector-instance str
the detector instance to get constants for e.g.
LPD1M1, required. Default: None
--out-file str HDF5 file to output constants into, required. Default:
None
Statistical analysis of calibration factors¶
Author: Mikhail Karnevskiy, Version 0.2
Plot calibration constants retrieved from the cal. DB.
To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant.
Values shown in plots are saved in h5 files.
To invoke this notebook and display help use:
xfel-calibrate GENERIC STATS_FROM_DB --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--start-date str] [--end-date str]
[--dclass str] [--modules str [str ...]]
[--submodules str [str ...]] [--constants str [str ...]]
[--nconstants int] [--max-time int] [--nMemToShow int]
[--gain-setting int [int ...]]
[--bias-voltage int [int ...]]
[--temperature int [int ...]]
[--integration-time float [float ...]]
[--pixels-x int [int ...]] [--pixels-y int [int ...]]
[--in-vacuum int [int ...]]
[--memory-cells int [int ...]]
[--acquisition-rate float [float ...]]
[--parameter-names str [str ...]]
[--separate-plot str [str ...]]
[--x-labels str [str ...]] [--photon-energy float]
[--out-folder str] [--use-existing str]
[--cal-db-interface str] [--cal-db-timeout int]
[--plot-range int] [--spShape int [int ...]]
[--sp-name str] [--gain-titles str [str ...]]
GENERIC STATS_FROM_DB
# Statistical analysis of calibration factors#
,
Author: Mikhail Karnevskiy, Version 0.2,
,
Plot calibration constants retrieved from the cal. DB.,
,
To be visualized, calibration constants are averaged per group of pixels.
Plots shows calibration constant over time for each constant.,
,
Values shown in plots are saved in h5 files.
positional arguments:
GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
STATS_FROM_DB Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--start-date str date to start investigation interval from. Default:
2019-01-01
--end-date str date to end investigation interval at, can be "now".
Default: NOW
--dclass str Detector class. Default: jungfrau
--modules str [str ...]
detector entry in the DB to investigate. Default:
['Jungfrau_M035']
--submodules str [str ...]
module index of a modular detector (1 for Q1M1 of
AGIPD), range allowed. Default: [2]
--constants str [str ...]
constants to plot. Default: ['RelativeGain']
--nconstants int Number of time stamps to plot. If not 0, overcome
start_date.. Default: 20
--max-time int max time margin in minutes to match bad pixels.
Default: 15
--nMemToShow int Number of memory cells to be shown in plots. Default:
16
--gain-setting int [int ...]
gain stages. Default: [0, 1, 2]
--bias-voltage int [int ...]
Bias voltage. Default: [90, 180]
--temperature int [int ...]
Operation temperature. Default: [291]
--integration-time float [float ...]
Integration time. Default: [4.96, 10, 50, 250]
--pixels-x int [int ...]
number of pixels along X axis. Default: [1024]
--pixels-y int [int ...]
number of pixels along Y axis. Default: [512]
--in-vacuum int [int ...]
0 if detector is operated in room pressure. Default:
[0]
--memory-cells int [int ...]
number of memory cells. Default: [1, 16]
--acquisition-rate float [float ...]
aquisition rate. Default: [1.1]
--parameter-names str [str ...]
names of parameters. Default: ['bias_voltage',
'integration_time', 'pixels_x', 'pixels_y',
'temperature', 'memory_cells']
--separate-plot str [str ...]
Plot on separate plots. Default: ['gain_setting',
'memory_cells', 'integration_time']
--x-labels str [str ...]
parameters to be shown on X axis: Acquisition rate,
Memory cells, Sensor Temperature, Integration Time.
Default: ['Sensor Temperature', 'Integration Time']
--photon-energy float
Photon energy of the beam. Default: 9.2
--out-folder str output folder. Default:
/gpfs/exfel/data/scratch/karnem/test_bla4/
--use-existing str If not empty, constants stored in given folder will be
used. Default:
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8016
--cal-db-timeout int timeout on caldb requests",. Default: 180000
--plot-range int range for plotting in units of median absolute
deviations. Default: 3
--spShape int [int ...]
Shape of superpixel. Default: [256, 64]
--sp-name str name of superpixel. Default: ASIC IDs
--gain-titles str [str ...]
Title inset related to gain. Default: ['High gain',
'Medium gain', 'Low gain']
Statistical analysis of calibration factors¶
Author: Mikhail Karnevskiy, Version 0.2
Plot calibration constants retrieved from the cal. DB.
To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant.
Values shown in plots are saved in h5 files.
To invoke this notebook and display help use:
xfel-calibrate GENERIC STATS_FROM_DB2 --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--start-date str] [--end-date str]
[--dclass str] [--modules str [str ...]]
[--submodules str [str ...]] [--constants str [str ...]]
[--nconstants int] [--max-time int] [--nMemToShow int]
[--gain-setting int [int ...]]
[--bias-voltage int [int ...]]
[--temperature int [int ...]]
[--integration-time float [float ...]]
[--pixels-x int [int ...]] [--pixels-y int [int ...]]
[--in-vacuum int [int ...]]
[--memory-cells int [int ...]]
[--acquisition-rate float [float ...]]
[--parameter-names str [str ...]]
[--separate-plot str [str ...]]
[--x-labels str [str ...]] [--photon-energy float]
[--out-folder str] [--use-existing str]
[--cal-db-interface str] [--cal-db-timeout int]
[--plot-range int] [--spShape int [int ...]]
[--sp-name str] [--gain-titles str [str ...]]
GENERIC STATS_FROM_DB2
# Statistical analysis of calibration factors#
,
Author: Mikhail Karnevskiy, Version 0.2,
,
Plot calibration constants retrieved from the cal. DB.,
,
To be visualized, calibration constants are averaged per group of pixels.
Plots shows calibration constant over time for each constant.,
,
Values shown in plots are saved in h5 files.
positional arguments:
GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
STATS_FROM_DB2 Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--start-date str date to start investigation interval from. Default:
2019-01-01
--end-date str date to end investigation interval at, can be "now".
Default: NOW
--dclass str Detector class. Default: jungfrau
--modules str [str ...]
detector entry in the DB to investigate. Default:
['Jungfrau_M035']
--submodules str [str ...]
module index of a modular detector (1 for Q1M1 of
AGIPD), range allowed. Default: [2]
--constants str [str ...]
constants to plot. Default: ['RelativeGain']
--nconstants int Number of time stamps to plot. If not 0, overcome
start_date.. Default: 20
--max-time int max time margin in minutes to match bad pixels.
Default: 15
--nMemToShow int Number of memory cells to be shown in plots. Default:
16
--gain-setting int [int ...]
gain stages. Default: [0, 1, 2]
--bias-voltage int [int ...]
Bias voltage. Default: [90, 180]
--temperature int [int ...]
Operation temperature. Default: [291]
--integration-time float [float ...]
Integration time. Default: [4.96, 10, 50, 250]
--pixels-x int [int ...]
number of pixels along X axis. Default: [1024]
--pixels-y int [int ...]
number of pixels along Y axis. Default: [512]
--in-vacuum int [int ...]
0 if detector is operated in room pressure. Default:
[0]
--memory-cells int [int ...]
number of memory cells. Default: [1, 16]
--acquisition-rate float [float ...]
aquisition rate. Default: [1.1]
--parameter-names str [str ...]
names of parameters. Default: ['bias_voltage',
'integration_time', 'pixels_x', 'pixels_y',
'temperature', 'memory_cells']
--separate-plot str [str ...]
Plot on separate plots. Default: ['gain_setting',
'memory_cells', 'integration_time']
--x-labels str [str ...]
parameters to be shown on X axis: Acquisition rate,
Memory cells, Sensor Temperature, Integration Time.
Default: ['Sensor Temperature', 'Integration Time']
--photon-energy float
Photon energy of the beam. Default: 9.2
--out-folder str output folder. Default:
/gpfs/exfel/data/scratch/karnem/test_bla4/
--use-existing str If not empty, constants stored in given folder will be
used. Default:
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8016
--cal-db-timeout int timeout on caldb requests",. Default: 180000
--plot-range int range for plotting in units of median absolute
deviations. Default: 3
--spShape int [int ...]
Shape of superpixel. Default: [256, 64]
--sp-name str name of superpixel. Default: ASIC IDs
--gain-titles str [str ...]
Title inset related to gain. Default: ['High gain',
'Medium gain', 'Low gain']
GOTTHARD2¶
Gotthard2 Offline Correction¶
Author: European XFEL Detector Group, Version: 1.0
Offline Calibration for the Gothard2 Detector
To invoke this notebook and display help use:
xfel-calibrate GOTTHARD2 CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
--run int [--sequences str [str ...]]
[--sequences-per-node int] [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-template str]
[--control-template str]
[--instrument-source-template str]
[--ctrl-source-template str] [--karabo-id-control str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--overwrite-creation-time str] [--constants-file str]
[--offset-correction | --no-offset-correction]
[--gain-correction | --no-gain-correction]
[--bias-voltage int] [--exposure-time float]
[--exposure-period float] [--acquisition-rate float]
[--single-photon int] [--skip-plots | --no-skip-plots]
[--pulse-idx-preview int]
GOTTHARD2 CORRECT
# Gotthard2 Offline Correction #
,
Author: European XFEL Detector Group, Version: 1.0,
,
Offline Calibration for the Gothard2 Detector
positional arguments:
GOTTHARD2 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequences str [str ...]
sequences to correct, set to [-1] for all, range
allowed. Default: [-1]
--sequences-per-node int
number of sequence files per node if notebook executed
through xfel-calibrate, set to 0 to not run SLURM
parallel. Default: 1
--karabo-id str karabo prefix of Gotthard-II devices. Default:
FXE_XAD_G2XES
--karabo-da str [str ...]
data aggregators. Default: ['GH201']
--receiver-template str
receiver template used to read INSTRUMENT keys..
Default: RECEIVER
--control-template str
control template used to read CONTROL keys.. Default:
CONTROL
--instrument-source-template str
template for source name (filled with karabo_id &
receiver_id). e.g.
'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
{}/DET/{}:daqOutput
--ctrl-source-template str
template for control source name (filled with
karabo_id_control). Default: {}/DET/{}
--karabo-id-control str
Control karabo ID. Set to empty string to use the
karabo-id. Default:
--use-dir-creation-date
use the creation data of the input dir for database
queries.. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use.. Default: tcp://max-
exfl016:8016#8025
--cal-db-timeout int timeout on caldb requests.. Default: 180000
--overwrite-creation-time str
To overwrite the measured creation_time. Required
Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28
13:00:00.00". Default:
--constants-file str Use constants in given constant file path. /gpfs/exfel
/data/scratch/ahmedk/dont_remove/gotthard2/constants/c
alibration_constants_GH2.h5. Default:
--offset-correction apply offset correction. This can be disabled to only
apply LUT or apply LUT and gain correction for non-
linear differential results.. Default: True
--no-offset-correction
Opposite of --offset-correction (default: True)
--gain-correction apply gain correction.. Default: True
--no-gain-correction Opposite of --gain-correction (default: True)
--bias-voltage int Detector bias voltage, set to -1 to use value in raw
file.. Default: -1
--exposure-time float
Detector exposure time, set to -1 to use value in raw
file.. Default: -1.0
--exposure-period float
Detector exposure period, set to -1 to use value in
raw file.. Default: -1.0
--acquisition-rate float
Detector acquisition rate (1.1/4.5), set to -1 to use
value in raw file.. Default: -1.0
--single-photon int Detector single photon mode (High/Low CDS), set to -1
to use value in raw file.. Default: -1
--skip-plots exit after writing corrected files. Default: False
--no-skip-plots Opposite of --skip-plots (default: False)
--pulse-idx-preview int
pulse index to preview. The following even/odd pulse
index is used for preview. TODO: update to pulseId
preview.. Default: 3
required arguments:
--in-folder str the folder to read data from, required. Default: None
--out-folder str the folder to output to, required. Default: None
--run int run to process, required. Default: None
Gotthard2 Dark Image Characterization¶
Author: European XFEL Detector Group, Version: 1.0
The following is a processing for offset, noise, and Badpixels maps using dark images taken with Gotthard2 detector. All constants are evaluated per strip, per pulse, and per memory cell. The maps are calculated for each gain stage that is acquired in 3 separate runs.
The three maps (calibration constants) can be injected to the database and stored locally.
To invoke this notebook and display help use:
xfel-calibrate GOTTHARD2 DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
--run-high int --run-med int --run-low int
[--karabo-id str] [--karabo-da str [str ...]]
[--receiver-template str] [--control-template str]
[--instrument-source-template str]
[--ctrl-source-template str] [--karabo-id-control str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--overwrite-creation-time str]
[--db-output | --no-db-output]
[--local-output | --no-local-output]
[--bias-voltage int] [--exposure-time float]
[--exposure-period float] [--acquisition-rate float]
[--single-photon int] [--min-trains int]
[--max-trains int] [--badpixel-threshold-sigma float]
[--operation-mode str]
GOTTHARD2 DARK
# Gotthard2 Dark Image Characterization #
,
Author: European XFEL Detector Group, Version: 1.0,
,
The following is a processing for offset, noise, and Badpixels maps using
dark images taken with Gotthard2 detector.,
All constants are evaluated per strip, per pulse, and per memory cell. The
maps are calculated for each gain stage that is acquired in 3 separate runs.,
,
The three maps (calibration constants) can be injected to the database and
stored locally.
positional arguments:
GOTTHARD2 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--karabo-id str karabo prefix of Gotthard-II devices. Default:
FXE_XAD_G2XES
--karabo-da str [str ...]
data aggregators. Default: ['GH201']
--receiver-template str
receiver template used to read INSTRUMENT keys..
Default: RECEIVER
--control-template str
control template used to read CONTROL keys.. Default:
CONTROL
--instrument-source-template str
template for source name (filled with karabo_id &
receiver_id). e.g.
'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput' noqa. Default:
{}/DET/{}:daqOutput
--ctrl-source-template str
template for control source name (filled with
karabo_id_control). Default: {}/DET/{}
--karabo-id-control str
Control karabo ID. Set to empty string to use the
karabo-id. Default:
--use-dir-creation-date
Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8020
--cal-db-timeout int timeout on caldb requests. Default: 300000
--overwrite-creation-time str
To overwrite the measured creation_time. Required
Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28
13:00:00.00". Default:
--db-output Output constants to the calibration database. Default:
False
--no-db-output Opposite of --db-output (default: False)
--local-output Output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--bias-voltage int Detector bias voltage, set to -1 to use value in raw
file.. Default: -1
--exposure-time float
Detector exposure time, set to -1 to use value in raw
file.. Default: -1.0
--exposure-period float
Detector exposure period, set to -1 to use value in
raw file.. Default: -1.0
--acquisition-rate float
Detector acquisition rate (1.1/4.5), set to -1 to use
value in raw file.. Default: -1.0
--single-photon int Detector single photon mode (High/Low CDS), set to -1
to use value in raw file.. Default: -1
--min-trains int Minimum number of trains that should be available to
process dark constants. Default 1.. Default: 1
--max-trains int Maximum number of trains to use for processing dark
constants. Set to 0 to use all available trains..
Default: 1000
--badpixel-threshold-sigma float
bad pixels defined by values outside n times this std
from median. Default: 5.0
--operation-mode str Detector dark run acquiring operation mode, optional.
Default:
required arguments:
--in-folder str the folder to read data from, required. Default: None
--out-folder str the folder to output to, required. Default: None
--run-high int run number for G0 dark run, required. Default: None
--run-med int run number for G1 dark run, required. Default: None
--run-low int run number for G2 dark run, required. Default: None
JUNGFRAU¶
Jungfrau Offline Correction¶
Author: European XFEL Detector Group, Version: 2.0
Offline Calibration for the Jungfrau Detector
To invoke this notebook and display help use:
xfel-calibrate JUNGFRAU CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
--run int [--sequences str [str ...]]
[--sequences-per-node int] [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-template str]
[--instrument-source-template str]
[--ctrl-source-template str] [--karabo-id-control str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--relative-gain | --no-relative-gain]
[--strixel-sensor | --no-strixel-sensor]
[--strixel-double-norm float] [--limit-trains int]
[--chunks-ids int] [--chunks-data int]
[--manual-slow-data | --no-manual-slow-data]
[--integration-time float] [--gain-setting int]
[--gain-mode int] [--mem-cells int] [--bias-voltage int]
[--skip-plots | --no-skip-plots] [--plot-trains int]
[--cell-id-preview int]
[--roi-definitions int [int ...]]
JUNGFRAU CORRECT
# Jungfrau Offline Correction #
,
Author: European XFEL Detector Group, Version: 2.0,
,
Offline Calibration for the Jungfrau Detector
positional arguments:
JUNGFRAU The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequences str [str ...]
sequences to correct, set to [-1] for all, range
allowed. Default: [-1]
--sequences-per-node int
number of sequence files per cluster node if run as
slurm job, set to 0 to not run SLURM parallel.
Default: 1
--karabo-id str karabo prefix of Jungfrau devices. Default:
SPB_IRDA_JF4M
--karabo-da str [str ...]
data aggregators. Default: ['JNGFR01', 'JNGFR02',
'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07',
'JNGFR08']
--receiver-template str
Detector receiver template for accessing raw data
files. e.g. "JNGFR{:02d}". Default: JNGFR{:02d}
--instrument-source-template str
template for source name (filled with karabo_id &
receiver_id). e.g.
'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
{}/DET/{}:daqOutput
--ctrl-source-template str
template for control source name (filled with
karabo_id_control). Default: {}/DET/CONTROL
--karabo-id-control str
if control is on a different ID, set to empty string
if it is the same a karabo-id. Default:
--use-dir-creation-date
use the creation data of the input dir for database
queries. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8017#8025
--cal-db-timeout int timeout on caldb requests. Default: 180000
--relative-gain do relative gain correction.. Default: True
--no-relative-gain Opposite of --relative-gain (default: True)
--strixel-sensor reordering for strixel detector layout.. Default:
False
--no-strixel-sensor Opposite of --strixel-sensor (default: False)
--strixel-double-norm float
normalization to use for double-size pixels, only
applied for strixel sensors.. Default: 2.0
--limit-trains int ONLY FOR TESTING. process only first N trains, Use 0
to process all.. Default: 0
--chunks-ids int HDF chunk size for memoryCell and frameNumber..
Default: 32
--chunks-data int HDF chunk size for pixel data in number of frames..
Default: 1
--manual-slow-data if true, use manually entered bias_voltage,
integration_time, gain_setting, and gain_mode values.
Default: False
--no-manual-slow-data
Opposite of --manual-slow-data (default: False)
--integration-time float
integration time in us, will be overwritten by value
in file. Default: 4.96
--gain-setting int 0 for dynamic gain, 1 for dynamic HG0, will be
overwritten by value in file. Default: 0
--gain-mode int 0 for runs with dynamic gain setting, 1 for fixgain.
It will be overwritten by value in file, if
manual_slow_data is set to True.. Default: 0
--mem-cells int Set mem_cells to -1 to automatically use the value
stored in RAW data.. Default: -1
--bias-voltage int will be overwritten by value in file. Default: 180
--skip-plots exit after writing corrected files. Default: False
--no-skip-plots Opposite of --skip-plots (default: False)
--plot-trains int Number of trains to plot for RAW and CORRECTED plots.
Set to -1 to automatically plot all trains.. Default:
500
--cell-id-preview int
cell Id used for preview in single-shot plots.
Default: 15
--roi-definitions int [int ...]
List with groups of 6 values defining ROIs, e.g. [3,
120, 180, 200, 550, -2] for module 3 (JNGFR03), slice
120:180, 200:550, average along axis -2 (slow scan, or
-1 for fast scan). Default: [-1]
required arguments:
--in-folder str the folder to read data from, required. Default: None
--out-folder str the folder to output to, required. Default: None
--run int run to process, required. Default: None
Jungfrau Dark Image Characterization¶
Author: European XFEL Detector Group, Version: 2.0
Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad pixel maps
To invoke this notebook and display help use:
xfel-calibrate JUNGFRAU DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
--run-high int --run-med int --run-low int
[--karabo-da str [str ...]] [--karabo-id str]
[--karabo-id-control str] [--receiver-template str]
[--instrument-source-template str]
[--ctrl-source-template str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--local-output | --no-local-output]
[--db-output | --no-db-output]
[--badpixel-threshold-sigma float]
[--offset-abs-threshold-low int [int ...]]
[--offset-abs-threshold-high int [int ...]]
[--max-trains int] [--min-trains int]
[--manual-slow-data | --no-manual-slow-data]
[--time-limits float] [--integration-time int]
[--gain-setting int] [--gain-mode int]
[--bias-voltage int] [--memory-cells int]
[--detailed-report | --no-detailed-report]
[--operation-mode str]
JUNGFRAU DARK
# Jungfrau Dark Image Characterization #
,
Author: European XFEL Detector Group, Version: 2.0,
,
Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad
pixel maps
positional arguments:
JUNGFRAU The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--karabo-da str [str ...]
list of data aggregators, which corresponds to
different JF modules. Default: ['JNGFR01', 'JNGFR02',
'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07',
'JNGFR08']
--karabo-id str karabo_id (detector identifier) prefix of Jungfrau
detector to process.. Default: SPB_IRDA_JF4M
--karabo-id-control str
if control is on a different ID, set to empty string
if it is the same a karabo-id. Default:
--receiver-template str
inset for receiver devices. Default: JNGFR{:02}
--instrument-source-template str
template for instrument source name (filled with
karabo_id & receiver_id). e.g.
'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
{}/DET/{}:daqOutput
--ctrl-source-template str
template for control source name (filled with
karabo_id_control). Default: {}/DET/CONTROL
--use-dir-creation-date
use dir creation date. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
calibrate db interface to connect to. Default:
tcp://max-exfl016:8016#8045
--cal-db-timeout int timeout on caldb requests. Default: 300000
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--badpixel-threshold-sigma float
bad pixels defined by values outside n times this std
from median. Default: 5.0
--offset-abs-threshold-low int [int ...]
absolute bad pixel threshold in terms of offset, lower
values. Default: [1000, 10000, 10000]
--offset-abs-threshold-high int [int ...]
absolute bad pixel threshold in terms of offset, upper
values. Default: [8000, 15000, 15000]
--max-trains int Maximum trains to process darks. Set to 0 to process
all available train images.. Default: 0
--min-trains int Minimum number of trains that should be available to
process dark constants. Default 1.. Default: 1
--manual-slow-data if true, use manually entered bias_voltage and
integration_time values. Default: False
--no-manual-slow-data
Opposite of --manual-slow-data (default: False)
--time-limits float to find calibration constants later on, the
integration time is allowed to vary by 0.5 us.
Default: 0.025
--integration-time int
integration time in us, will be overwritten by value
in file. Default: 1000
--gain-setting int 0 for dynamic, forceswitchg1, forceswitchg2, 1 for
dynamichg0, fixgain1, fixgain2. Will be overwritten by
value in file. Default: 0
--gain-mode int 1 if medium and low runs are fixgain1 and fixgain2,
otherwise 0. It will be overwritten by value in file,
if manual_slow_data. Default: 0
--bias-voltage int sensor bias voltage in V, will be overwritten by value
in file. Default: 90
--memory-cells int number of memory cells. Default: 16
--detailed-report Default: False
--no-detailed-report Opposite of --detailed-report (default: False)
--operation-mode str Detector operation mode, optional. Default:
ADAPTIVE_GAIN
required arguments:
--in-folder str folder under which runs are located, required.
Default: None
--out-folder str path to place reports at, required. Default: None
--run-high int run number for G0 dark run, required. Default: None
--run-med int run number for G1 dark run, required. Default: None
--run-low int run number for G2 dark run, required. Default: None
LPD¶
LPD Offline Correction¶
Author: European XFEL Data Analysis Group
To invoke this notebook and display help use:
xfel-calibrate LPD CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
[--sequences int [int ...]] [--modules int [int ...]]
[--karabo-da str [str ...]] --run int [--karabo-id str]
[--input-source str] [--output-source str]
[--creation-time str] [--cal-db-interface str]
[--cal-db-timeout int] [--cal-db-root str]
[--mem-cells int] [--bias-voltage float]
[--capacitor str] [--photon-energy float]
[--category int] [--offset-corr | --no-offset-corr]
[--rel-gain | --no-rel-gain] [--ff-map | --no-ff-map]
[--gain-amp-map | --no-gain-amp-map]
[--overwrite | --no-overwrite] [--chunks-data int]
[--chunks-ids int] [--create-virtual-cxi-in str]
[--sequences-per-node int] [--max-nodes int]
[--num-workers int] [--num-threads-per-worker int]
LPD CORRECT
# LPD Offline Correction #
Author: European XFEL Data Analysis Group
positional arguments:
LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequences int [int ...]
Sequences to correct, use [-1] for all. Default: [-1]
--modules int [int ...]
Modules indices to correct, use [-1] for all, only
used when karabo_da is empty. Default: [-1]
--karabo-da str [str ...]
Data aggregators names to correct, use [''] for all.
Default: ['']
--karabo-id str Karabo domain for detector.. Default: FXE_DET_LPD1M-1
--input-source str Input fast data source.. Default:
{karabo_id}/DET/{module_index}CH0:xtdf
--output-source str Output fast data source, empty to use same as input..
Default:
--creation-time str The timestamp to use with Calibration DB. Required
Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
11:02:41. Default:
--cal-db-interface str
Not needed, compatibility with current webservice..
Default:
--cal-db-timeout int Not needed, compatbility with current webservice..
Default: 0
--cal-db-root str Default: /gpfs/exfel/d/cal/caldb_store
--mem-cells int Memory cells, LPD constants are always taken with 512
cells.. Default: 512
--bias-voltage float Detector bias voltage.. Default: 250.0
--capacitor str Capacitor setting: 5pF or 50pF. Default: 5pF
--photon-energy float
Photon energy in keV.. Default: 9.2
--category int Whom to blame.. Default: 0
--offset-corr Offset correction.. Default: True
--no-offset-corr Opposite of --offset-corr (default: True)
--rel-gain Gain correction based on RelativeGain constant..
Default: True
--no-rel-gain Opposite of --rel-gain (default: True)
--ff-map Gain correction based on FFMap constant.. Default:
True
--no-ff-map Opposite of --ff-map (default: True)
--gain-amp-map Gain correction based on GainAmpMap constant..
Default: True
--no-gain-amp-map Opposite of --gain-amp-map (default: True)
--overwrite set to True if existing data should be overwritten.
Default: True
--no-overwrite Opposite of --overwrite (default: True)
--chunks-data int HDF chunk size for pixel data in number of frames..
Default: 1
--chunks-ids int HDF chunk size for cellId and pulseId datasets..
Default: 32
--create-virtual-cxi-in str
Folder to create virtual CXI files in (for each
sequence).. Default:
--sequences-per-node int
Sequence files to process per node. Default: 1
--max-nodes int Maximum number of SLURM jobs to split correction work
into. Default: 8
--num-workers int Worker processes per node, 8 is safe on 768G nodes but
won't work on 512G.. Default: 8
--num-threads-per-worker int
Number of threads per worker.. Default: 32
required arguments:
--in-folder str the folder to read data from, required. Default: None
--out-folder str the folder to output to, required. Default: None
--run int run to process, required. Default: None
LPD Offset, Noise and Dead Pixels Characterization¶
Author: M. Karnevskiy, S. Hauf
This notebook performs re-characterize of dark images to derive offset, noise and bad-pixel maps. All three types of constants are evaluated per-pixel and per-memory cell.
The notebook will correctly handle veto settings, but note that if you veto cells you will not be able to use these offsets for runs with different veto settings - vetoed cells will have zero offset.
The evaluated calibration constants are stored locally and injected in the calibration data base.
To invoke this notebook and display help use:
xfel-calibrate LPD DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--sequence int]
[--modules str [str ...]] --run-high int --run-med int
--run-low int [--karabo-id str]
[--karabo-da str [str ...]] [--receiver-id str]
[--path-template str] [--h5path str] [--h5path-idx str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--local-output | --no-local-output]
[--db-output | --no-db-output] [--capacitor-setting int]
[--mem-cells int] [--bias-voltage int]
[--thresholds-offset-sigma float]
[--thresholds-offset-hard int [int ...]]
[--thresholds-noise-sigma float]
[--thresholds-noise-hard int [int ...]]
[--skip-first-ntrains int] [--instrument str]
[--ntrains int]
[--high-res-badpix-3d | --no-high-res-badpix-3d]
[--test-for-normality | --no-test-for-normality]
[--operation-mode str]
LPD DARK
# LPD Offset, Noise and Dead Pixels Characterization #
,
Author: M. Karnevskiy, S. Hauf,
,
This notebook performs re-characterize of dark images to derive offset, noise
and bad-pixel maps. All three types of constants are evaluated per-pixel and
per-memory cell.,
,
The notebook will correctly handle veto settings, but note that if you veto
cells you will not be able to use these offsets for runs with different veto
settings - vetoed cells will have zero offset.,
,
The evaluated calibration constants are stored locally and injected in the
calibration data base.,
,
positional arguments:
LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
The ipcluster profile to use. Default: noDB
--sequence int sequence files to evaluate. Default: 0
--modules str [str ...]
list of modules to evaluate, RANGE ALLOWED. Default:
[-1]
--karabo-id str karabo karabo_id. Default: FXE_DET_LPD1M-1
--karabo-da str [str ...]
a list of data aggregators names, Default [-1] for
selecting all data aggregators. Default: ['-1']
--receiver-id str inset for receiver devices. Default: {}CH0
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{:05d}.h5
--h5path str path in the HDF5 file to images. Default:
/INSTRUMENT/{}/DET/{}:xtdf/image
--h5path-idx str path in the HDF5 file to images. Default:
/INDEX/{}/DET/{}:xtdf/image
--use-dir-creation-date
use the creation date of the directory for database
time derivation. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8015#8025
--cal-db-timeout int timeout on caldb requests". Default: 300000
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--capacitor-setting int
capacitor_setting for which data was taken. Default: 5
--mem-cells int number of memory cells used. Default: 512
--bias-voltage int detector bias voltage. Default: 250
--thresholds-offset-sigma float
bad pixel relative threshold in terms of n sigma
offset. Default: 3.0
--thresholds-offset-hard int [int ...]
bad pixel hard threshold. Default: [400, 1500]
--thresholds-noise-sigma float
bad pixel relative threshold in terms of n sigma
noise. Default: 7.0
--thresholds-noise-hard int [int ...]
bad pixel hard threshold. Default: [1, 35]
--skip-first-ntrains int
Number of first trains to skip. Default: 10
--instrument str instrument name. Default: FXE
--ntrains int number of trains to use. Default: 100
--high-res-badpix-3d plot bad-pixel summary in high resolution. Default:
False
--no-high-res-badpix-3d
Opposite of --high-res-badpix-3d (default: False)
--test-for-normality permorm normality test. Default: False
--no-test-for-normality
Opposite of --test-for-normality (default: False)
--operation-mode str Detector operation mode, optional. Default:
required arguments:
--in-folder str path to input data, required. Default: None
--out-folder str path to output to, required. Default: None
--run-high int run number in which high gain data was recorded,
required. Default: None
--run-med int run number in which medium gain data was recorded,
required. Default: None
--run-low int run number in which low gain data was recorded,
required. Default: None
LPD Radial X-ray Gain Evaluation¶
Author: S. Hauf, Version 0.5
Taking proper flat field for LPD can be difficult, as air scattering will always be present. Additionally, the large detector mandates a large distance to the source, in order to reduce \(1/r\) effects.
Because of this a radial evaluation method is used, which assumes that pixels a the same radial distance \(r\) should on average have the same signal \(S(r)\).
To invoke this notebook and display help use:
xfel-calibrate LPD FF --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
--run int [--sequences int [int ...]]
[--capacitor-setting int] [--mem-cells int]
[--local-output | --no-local-output]
[--db-output | --no-db-output] [--bias-voltage int]
[--cal-db-interface str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--instrument str] [--limit-trains int]
[--geometry-file str]
[--beam-center-offset float [float ...]]
[--allowed-gain-thresholds float [float ...]]
[--badpix-entire-asic-threshold float]
[--photon-energy float] [--max-images int]
LPD FF
# LPD Radial X-ray Gain Evaluation #
,
Author: S. Hauf, Version 0.5,
,
Taking proper flat field for LPD can be difficult, as air scattering will
always be present. Additionally, the large detector mandates a large distance
to the source, in order to reduce $1/r$ effects.
,
Because of this a radial evaluation method is used, which assumes that pixels
a the same radial distance $r$ should on average have the same signal $S(r)$.
positional arguments:
LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
FF Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--sequences int [int ...]
which sequence files to use. Default: [0]
--capacitor-setting int
capacitor_setting for which data was taken. Default: 5
--mem-cells int number of memory cells used. Default: 512
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--bias-voltage int detector bias voltage. Default: 250
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl015:5005
--use-dir-creation-date
use the creation date of the directory for database
time derivation. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--instrument str Default: FXE
--limit-trains int limit the number of train for the evaluation. Default:
10
--geometry-file str the geometry file to use, MAR 2018. Default:
/gpfs/exfel/d/cal/exchange/lpdMF_00.h5
--beam-center-offset float [float ...]
offset from the beam center, MAR 2018. Default: [1.5,
1]
--allowed-gain-thresholds float [float ...]
relative gain values within these bounds are valid.
Default: [0.5, 1.5]
--badpix-entire-asic-threshold float
if more than this fraction of pixels on an ASIC are
"bad" the entire ASIC is flagged. Default: 0.25
--photon-energy float
photon enery in keV. Default: 9.2
--max-images int maximum number of images to use in evaluation.
Default: 1024
required arguments:
--in-folder str path to already corrected input data, required.
Default: None
--out-folder str path to output to, required. Default: None
--run int runs to process, required. Default: None
Injecting calibration constant data to the database¶
Author: European XFEL Detector Group, Version: 1.0
Reading h5files of calibration constants to inject them to the database. Used for LPD
To invoke this notebook and display help use:
xfel-calibrate LPD INJECT_CONSTANTS --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --constant-names str [str ...]
--in-folder str --out-folder str [--proposal str]
[--runs str [str ...]] [--karabo-id str]
[--karabo-da str [str ...]] [--cal-db-interface str]
[--memory-cells int] [--bias-voltage int]
[--capacitor int] [--category int]
[--photon-energy float] --creation-time str
LPD INJECT_CONSTANTS
# Injecting calibration constant data to the database #
,
Author: European XFEL Detector Group, Version: 1.0,
,
Reading h5files of calibration constants to inject them to the database. Used
for LPD
positional arguments:
LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
INJECT_CONSTANTS Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--proposal str Add proposal number to be sent to the database as a
part of Raw data location.. Default:
--runs str [str ...] Add list of runs to be sent to the database as a part
of Raw data location.. Default: ['']
--karabo-id str detector identifier.. Default: FXE_DET_LPD1M-1
--karabo-da str [str ...]
karabo data aggregators. default "all" for all 16
karabo data aggregator names.. Default: ['all']
--cal-db-interface str
calibration DB zmq address.. Default: tcp://max-
exfl016:8015#8045
--memory-cells int Number of memory cells. Used for constant conditions..
Default: 512
--bias-voltage int bias voltage value. Used for constant conditions..
Default: 250
--capacitor int capacitor value. Used for constant conditions..
Default: 5
--category int calibration constant source category, 0 for European
XFEl and 1 for RAL. Used for constant conditions..
Default: 0
--photon-energy float
calibration constant photon energy. Used for constant
conditions.. Default: 9.2
required arguments:
--constant-names str [str ...]
calibration constant name, required.. Default: None
--in-folder str calibration constants folder, required.. Default: None
--out-folder str output folder to store report path in case the
notebook is executed by CLI, required.. Default: None
--creation-time str creation time for the injected constants. required
format '2019-01-20T14:12:06'. Default: None
LPD Gain Characterization (Charge Injection)¶
Author: S. Hauf, Version: 0.5
The following code characterizes the gain of the LPD detector from charge injection data, i.e. data with charge injected into the amplifiers, bypassing the sensor. The data needs to fulfil the following requirements:
- each file should represent one scan point for one mudle, defined by detector gain setting and charge injections setting
- settings need to overlap at at least one point for two neighboring gain ranges
- 100 samples or more per pixel and memory cell should be present for each setting.
The data is then analyzed by calcualting the per-pixel, per memory cell mean of the samples for each setting. These means are then normalized to the median peak position of a all means of the first module. Overlapping settings in neighboring gain ranges are used to deduce the slopes of the different gains with respect to the high gain setting.
To invoke this notebook and display help use:
xfel-calibrate LPD PC --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str] --in-folder
str --out-folder str [--offset-store str]
[--modules int [int ...]] [--capacitor-setting int]
[--mem-cells int] [--local-output | --no-local-output]
[--db-output | --no-db-output]
[--db-input | --no-db-input] [--bias-voltage int]
[--cal-db-interface str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--instrument str]
[--high-res-badpix-3d | --no-high-res-badpix-3d]
LPD PC
# LPD Gain Characterization (Charge Injection) #
,
Author: S. Hauf, Version: 0.5,
,
The following code characterizes the gain of the LPD detector from charge
injection data, i.e. data with charge injected into the amplifiers, bypassing
the sensor. The data needs to fulfil the following requirements:,
,
* each file should represent one scan point for one mudle, defined by
detector gain setting
and charge injections setting,
* settings need to overlap at at least one point for two neighboring gain
ranges,
* 100 samples or more per pixel and memory cell should be present for each
setting.,
,
The data is then analyzed by calcualting the per-pixel, per memory cell mean
of the samples for each setting. These means are then normalized to the
median peak position of a all means of the first module. Overlapping settings
in neighboring gain ranges are used to deduce the slopes of the different
gains with respect to the high gain setting.
positional arguments:
LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
PC Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
The ipcluster profile to use. Default: noDB
--offset-store str path to offset store, overwrite use_db_input. Default:
--modules int [int ...]
Default: [-1]
--capacitor-setting int
capacitor_setting for which data was taken. Default: 5
--mem-cells int number of memory cells used. Default: 512
--local-output output constants locally. Default: True
--no-local-output Opposite of --local-output (default: True)
--db-output output constants to database. Default: False
--no-db-output Opposite of --db-output (default: False)
--db-input Default: True
--no-db-input Opposite of --db-input (default: True)
--bias-voltage int detector bias voltage. Default: 300
--cal-db-interface str
the database interface to use. Default: tcp://max-
exfl016:8015
--use-dir-creation-date
use the creation date of the directory for database
time derivation. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--instrument str Default: FXE
--high-res-badpix-3d Default: False
--no-high-res-badpix-3d
Opposite of --high-res-badpix-3d (default: False)
required arguments:
--in-folder str path to input data, required. Default: None
--out-folder str path to output to, required. Default: None
Mine XGM Data and LPD Profile¶
Author: S. Hauf, Version: 0.1
To invoke this notebook and display help use:
xfel-calibrate LPD XGM_MINE --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--dc float [float ...]]
[--geometry-file str] [--proposal-folder str]
[--xgm-folder str] [--adc-folder str]
[--file-prefix str] [--raw-prefix str]
[--xgm-prefix str] [--adc-prefix str] [--data-path str]
[--gain-path str] [--xgm-path str] [--xgm-path-td str]
[--adc-root str] [--adc-paths str [str ...]]
[--temp-out-folder str] [--cluster-profile str]
[--chunk-size int] [--runs str [str ...]] [--method str]
[--sequences str [str ...]]
LPD XGM_MINE
# Mine XGM Data and LPD Profile #
Author: S. Hauf, Version: 0.1
positional arguments:
LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
XGM_MINE Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--dc float [float ...]
center offset. Default: [1.5, 1]
--geometry-file str the geometry file to use, MAR 2018. Default:
/gpfs/exfel/d/cal/exchange/lpdMF_00.h5
--proposal-folder str
folder under which runs can be found. Default:
/gpfs/exfel/exp/FXE/201701/p002045/proc/
--xgm-folder str folder under which runs can be found. Default:
/gpfs/exfel/exp/FXE/201701/p002045/raw/
--adc-folder str folder under which runs can be found. Default:
/gpfs/exfel/exp/FXE/201701/p002045/raw/
--file-prefix str prefix of each data file - should be a template to
replace run number. Default: CORR-R{:04d}-LPD
--raw-prefix str prefix of each data file - should be a template to
replace run number. Default: RAW-R{:04d}-LPD
--xgm-prefix str prefix for XGM file - should be a template to replace
run number. Default: RAW-R{:04d}-DA03
--adc-prefix str prefix for ADC file - should be a template to replace
run number. Default: RAW-R{:04d}-DA01
--data-path str path to data in file. Default:
INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/data
--gain-path str path to data in file. Default:
INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/gain
--xgm-path str path to XGM data. Default: /INSTRUMENT/SA1_XTD2_XGM/DO
OCS/MAIN:output/data/intensityTD
--xgm-path-td str path to XGM AUX data. Default: /INSTRUMENT/SA1_XTD2_XG
M/DOOCS/MAIN:output/data/intensityAUXTD
--adc-root str root path to ADC data. Default:
/INSTRUMENT/FXE_RR_DAQ/ADC/1:network/digitizers
--adc-paths str [str ...]
Default: ['channel_1_A/apd/pulseIntegral',
'channel_1_B/apd/pulseIntegral',
'channel_1_C/apd/pulseIntegral',
'channel_1_D/apd/pulseIntegral']
--temp-out-folder str
Default: /gpfs/exfel/data/scratch/haufs/test/tempout6
--cluster-profile str
Default: noDB
--chunk-size int read this amount of data at once. Default: 512
--runs str [str ...] Default: 69-72
--method str method to use for evaluation of images: radial,
average. Default: average
--sequences str [str ...]
range allowed. Default: [-1]
PNCCD¶
pnCCD Data Correction¶
Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0
The following notebook provides offset, common mode, relative gain, split events and pattern classification corrections of images acquired with the pnCCD. This notebook does not yet correct for charge transfer inefficiency.
To invoke this notebook and display help use:
xfel-calibrate PNCCD CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--in-folder str] [--out-folder str]
[--run int] [--sequences str [str ...]]
[--sequences-per-node int] [--karabo-da str]
[--karabo-id str] [--receiver-id str]
[--path-template str] [--instrument-source-template str]
[--commonModeAxis int]
[--commonModeBlockSize int [int ...]]
[--split-evt-primary-threshold float]
[--split-evt-secondary-threshold float]
[--saturated-threshold float]
[--fix-temperature-top float]
[--fix-temperature-bot float] [--gain int]
[--bias-voltage float] [--integration-time int]
[--photon-energy float] [--cal-db-interface str]
[--cal-db-timeout int] [--creation-time str]
[--only-offset | --no-only-offset]
[--common-mode | --no-common-mode]
[--relgain | --no-relgain]
[--pattern-classification | --no-pattern-classification]
[--chunk-size-idim int] [--limit-images int]
PNCCD CORRECT
# pnCCD Data Correction #
,
Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0,
,
The following notebook provides offset, common mode, relative gain, split
events and pattern classification corrections of images acquired with the
pnCCD. This notebook *does not* yet correct for charge transfer inefficiency.
positional arguments:
PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--in-folder str input folder. Default:
/gpfs/exfel/exp/SQS/202031/p900166/raw
--out-folder str output folder. Default: /gpfs/exfel/data/scratch/ahmed
k/test/remove/pnccd_correct
--run int which run to read data from. Default: 347
--sequences str [str ...]
sequences to correct, set to -1 for all, range
allowed. Default: [-1]
--sequences-per-node int
number of sequences running on the same slurm node..
Default: 1
--karabo-da str data aggregators. Default: PNCCD01
--karabo-id str karabo prefix of PNCCD devices. Default:
SQS_NQS_PNCCD1MP
--receiver-id str inset for receiver devices. Default: PNCCD_FMT-0
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{:05d}.h5
--instrument-source-template str
template for data source name, will be filled with
karabo_id and receiver_id.. Default: {}/CAL/{}:output
--commonModeAxis int axis along which common mode will be calculated, 0 =
row, and 1 = column. Default: 0
--commonModeBlockSize int [int ...]
size of the detector in pixels for common mode
calculations. Default: [512, 512]
--split-evt-primary-threshold float
primary threshold for split event classification in
terms of n sigma noise. Default: 4.0
--split-evt-secondary-threshold float
secondary threshold for split event classification in
terms of n sigma noise. Default: 3.0
--saturated-threshold float
full well capacity in ADU. Default: 32000.0
--fix-temperature-top float
fix temperature for top sensor in K, set to 0. to use
value from slow data.. Default: 0.0
--fix-temperature-bot float
fix temperature for bottom senspr in K, set to 0. to
use value from slow data.. Default: 0.0
--gain int the detector's gain setting. Set to -1 to use the
value from the slow data.. Default: -1
--bias-voltage float the detector's bias voltage. set to 0. to use value
from slow data.. Default: 0.0
--integration-time int
detector's integration time. Default: 70
--photon-energy float
Al fluorescence in keV. Default: 1.6
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8015
--cal-db-timeout int timeout on caldb requests. Default: 300000
--creation-time str The timestamp to use with Calibration DB. Required
Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
11:02:41. Default:
--only-offset Only, apply offset.. Default: False
--no-only-offset Opposite of --only-offset (default: False)
--common-mode Apply common mode correction. Default: True
--no-common-mode Opposite of --common-mode (default: True)
--relgain Apply relative gain correction. Default: True
--no-relgain Opposite of --relgain (default: True)
--pattern-classification
classify split events. Default: True
--no-pattern-classification
Opposite of --pattern-classification (default: True)
--chunk-size-idim int
H5 chunking size of output data. Default: 1
--limit-images int this parameter is used for limiting number of images
to correct from a sequence file. ONLY FOR TESTING..
Default: 0
pnCCD Dark Characterization¶
Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0
The following notebook provides dark image analysis of the pnCCD detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.
On the first iteration, the offset and noise maps are generated. Initial bad pixels map is obtained based on the offset and initial noise maps. Edge pixels are also added to the bad pixels map.
On the second iteration, the noise map is corrected for common mode. A second bad pixel map is generated based on the offset map and offset-and-common-mode-corrected noise map. Then, the hole in the center of the CCD is added to the second bad pixel map.
On the third and final iteration, the pixels that show up on the abovementioned bad pixels map are masked. Possible events due to cosmic rays are found and masked. The data are then again offset and common mode corrected and a new final noise and bad pixels maps are generated.
These latter resulting maps together with the offset map are saved as .h5 files to a local path for a later use. These dark constants are not automatically sent to the database.
To invoke this notebook and display help use:
xfel-calibrate PNCCD DARK --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] --in-folder str --out-folder str
--run int [--karabo-da str [str ...]] [--karabo-id str]
[--receiver-id str] [--path-template str]
[--instrument-source-template str]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--cal-db-interface str] [--cal-db-timeout int]
[--db-output | --no-db-output]
[--local-output | --no-local-output]
[--creation-time str] [--fix-temperature-top float]
[--fix-temperature-bot float] [--temp-limits int]
[--gain int] [--bias-voltage float]
[--integration-time int] [--commonModeAxis int]
[--commonModeBlockSize int [int ...]]
[--sigmaNoise float] [--bad-pixel-offset-sigma float]
[--bad-pixel-noise-sigma float] [--max-trains int]
[--min-trains int] [--operation-mode str]
[--db-module str]
PNCCD DARK
# pnCCD Dark Characterization
,
Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0,
,
The following notebook provides dark image analysis of the pnCCD detector.
Dark characterization evaluates offset and noise of the detector and gives
information about bad pixels.
,
On the first iteration, the offset and noise maps are generated. Initial bad
pixels map is obtained based on the offset and initial noise maps. Edge
pixels are also added to the bad pixels map.,
,
On the second iteration, the noise map is corrected for common mode. A second
bad pixel map is generated based on the offset map and
offset-and-common-mode-corrected noise map. Then, the hole in the center of
the CCD is added to the second bad pixel map.,
,
On the third and final iteration, the pixels that show up on the
abovementioned bad pixels map are masked. Possible events due to cosmic rays
are found and masked. The data are then again offset and common mode
corrected and a new final noise and bad pixels maps are generated.,
,
These latter resulting maps together with the offset map are saved as .h5
files to a local path for a later use. These dark constants are not
automatically sent to the database.
positional arguments:
PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
DARK Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--karabo-da str [str ...]
data aggregators. Default: ['PNCCD01']
--karabo-id str karabo prefix of PNCCD devices. Default:
SQS_NQS_PNCCD1MP
--receiver-id str inset for receiver devices. Default: PNCCD_FMT-0
--path-template str the template to use to access data. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--instrument-source-template str
data source path in h5file. Template filled with
karabo_id and receiver_id. Default: {}/CAL/{}:output
--use-dir-creation-date
use dir creation date as data production reference
date. Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8021
--cal-db-timeout int timeout on caldb requests. Default: 300000
--db-output if True, the notebook sends dark constants to the
calibration database. Default: False
--no-db-output Opposite of --db-output (default: False)
--local-output if True, the notebook saves dark constants locally.
Default: True
--no-local-output Opposite of --local-output (default: True)
--creation-time str To overwrite the measured creation_time. Required
Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04
11:02:41.00. Default:
--fix-temperature-top float
fix temperature of top pnCCD sensor in K. Set to 0, to
use the value from slow data. Default: 0.0
--fix-temperature-bot float
fix temperature of bottom pnCCD sensor in K. Set to 0,
to use the value from slow data. Default: 0.0
--temp-limits int temperature limits in which calibration parameters are
considered equal. Default: 5
--gain int the detector's gain setting. Set to -1 to use the
value from the slow data.. Default: -1
--bias-voltage float the detector's bias voltage. set to 0. to use the
value from slow data.. Default: 0.0
--integration-time int
detector's integration time.. Default: 70
--commonModeAxis int axis along which common mode will be calculated (0:
along rows, 1: along columns). Default: 0
--commonModeBlockSize int [int ...]
size of the detector in pixels for common mode
calculations. Default: [512, 512]
--sigmaNoise float pixels whose signal value exceeds sigmaNoise*noise
will be considered as cosmics and are masked. Default:
10.0
--bad-pixel-offset-sigma float
any pixel whose offset beyond this standard deviations
is a bad pixel. Default: 4.0
--bad-pixel-noise-sigma float
any pixel whose noise beyond this standard deviations
is a bad pixel. Default: 4.0
--max-trains int Maximum number of trains to use for dark processing.
Set to 0 to process all trains.. Default: 500
--min-trains int Minimum number of trains to proceed with dark
processing.. Default: 1
--operation-mode str Detector operation mode, optional. Default:
--db-module str the device name for pnCCD detector. Default:
required arguments:
--in-folder str input folder, required. Default: None
--out-folder str output folder, required. Default: None
--run int which run to read data from, required. Default: None
pnCCD Gain Characterization¶
Authors: DET Group, modified by Kiana Setoodehnia on December 2020 - Version 4.0
The following notebook provides gain characterization for the pnCCD. It relies on data which are previously corrected using the Meta Data Catalog web service interface or by running the Correct_pnCCD_NBC.ipynb notebook. Prior to running this notebook, the corrections which should be applied by the web service or the aforementioned notebook are as follows:
- offset correction
- common mode correction
- split pattern classification
To invoke this notebook and display help use:
xfel-calibrate PNCCD RELGAIN --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--cluster-profile str]
[--in-folder str] [--out-folder str] [--run int]
[--sequences str [str ...]] [--db-module str]
[--karabo-da str] [--karabo-da-control str]
[--karabo-id str] [--receiver-id str]
[--path-template str] [--path-template-ctrl str]
[--path-template-seqs str] [--h5path str]
[--h5path-ctrl str] [--cpuCores int]
[--use-dir-creation-date | --no-use-dir-creation-date]
[--sequences-per-node int] [--chunkSize int]
[--run-parallel | --no-run-parallel]
[--db-output | --no-db-output]
[--local-output | --no-local-output]
[--cal-db-interface str] [--cal-db-timeout int]
[--creation-time str] [--fix-temperature-top float]
[--fix-temperature-bot float] [--gain float]
[--bias-voltage float] [--integration-time int]
[--photon-energy float]
PNCCD RELGAIN
# pnCCD Gain Characterization #
,
Authors: DET Group, modified by Kiana Setoodehnia on December 2020 - Version
4.0,
,
The following notebook provides gain characterization for the pnCCD. It
relies on data which are previously corrected using the Meta Data Catalog
web service interface or by running the Correct_pnCCD_NBC.ipynb notebook.
Prior to running this notebook, the corrections which should be applied by
the web service or the aforementioned notebook are as follows:,
,
- offset correction,
- common mode correction,
- split pattern classification
positional arguments:
PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
RELGAIN Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--cluster-profile str
ipcluster profile to use. Default: noDB
--in-folder str input folder for the raw data. Default:
/gpfs/exfel/exp/SQS/202031/p900166/raw
--out-folder str output folder. Default:
/gpfs/exfel/data/scratch/setoodeh/Test
--run int which run to read data from. Default: 347
--sequences str [str ...]
sequences to correct, set to -1 for all, range
allowed. Default: [-1]
--db-module str Default: pnCCD_M205_M206
--karabo-da str data aggregators. Default: PNCCD01
--karabo-da-control str
file inset for control data. Default: PNCCD02
--karabo-id str karabo prefix of PNCCD devices. Default:
SQS_NQS_PNCCD1MP
--receiver-id str inset for receiver devices. Default: PNCCD_FMT-0
--path-template str the template to use to access data. Default:
CORR-R{:04d}-PNCCD01-S{{:05d}}.h5
--path-template-ctrl str
the template to use to access data. Default:
RAW-R{:04d}-{}-S{{:05d}}.h5
--path-template-seqs str
Default: {}/r{:04d}/*PNCCD01-S*.h5
--h5path str path to data in the HDF5 file. Default:
/INSTRUMENT/{}/CAL/{}:output/data/
--h5path-ctrl str Default: /CONTROL/{}/CTRL/TCTRL
--cpuCores int specifies the number of running cpu cores. Default: 40
--use-dir-creation-date
this is needed to obtain creation time of the run.
Default: True
--no-use-dir-creation-date
Opposite of --use-dir-creation-date (default: True)
--sequences-per-node int
Default: 1
--chunkSize int number of images to read per chunk. Default: 100
--run-parallel Default: True
--no-run-parallel Opposite of --run-parallel (default: True)
--db-output if True, the notebook injects dark constants into the
calibration database. Default: False
--no-db-output Opposite of --db-output (default: False)
--local-output if True, the notebook saves dark constants locally.
Default: True
--no-local-output Opposite of --local-output (default: True)
--cal-db-interface str
calibration DB interface to use. Default: tcp://max-
exfl016:8015
--cal-db-timeout int timeout on caldb requests. Default: 300000
--creation-time str To overwrite the measured creation_time. Required
Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04
11:02:41.00. Default:
--fix-temperature-top float
fix temperature of top pnCCD sensor in K, set to 0. to
use the value from slow data. Default: 0.0
--fix-temperature-bot float
fix temperature of bottom pnCCD sensor in K, set to 0.
to use the value from slow data. Default: 0.0
--gain float the detector's gain setting. Set to 0, to use the
value from slow data. Default: 0.1
--bias-voltage float the detector's bias voltage. set to 0. to use the
value from slow data.. Default: 0.0
--integration-time int
detector's integration time. Default: 70
--photon-energy float
Al fluorescence in keV. Default: 1.6
REMI¶
Transformation parameters¶
To invoke this notebook and display help use:
xfel-calibrate REMI CORRECT --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--run int] [--in-folder str]
[--out-folder str] [--calib-config-path str]
[--cycle str] [--cal-db-timeout int]
[--cal-db-interface str] [--karabo-da str]
[--karabo-id str] [--proposal str]
[--out-aggregator str] [--out-seq-len int]
[--det-device-id str] [--det-output-key str]
[--save-raw-triggers | --no-save-raw-triggers]
[--save-raw-edges | --no-save-raw-edges]
[--save-rec-signals | --no-save-rec-signals]
[--save-rec-hits | --no-save-rec-hits]
[--chunks-triggers int [int ...]]
[--chunks-edges int [int ...]]
[--chunks-hits int [int ...]]
[--chunks-signals int [int ...]]
[--dataset-compression str]
[--dataset-compression-opts int]
[--quad-anode | --no-quad-anode] [--ppt-source str]
[--ignore-fel | --no-ignore-fel]
[--ignore-ppl | --no-ignore-ppl] [--ppl-offset int]
[--laser-ppt-mask int] [--instrument-sase int]
[--first-pulse-offset int] [--single-pulse-length int]
[--mp-find-triggers float] [--mp-find-edges float]
[--mt-avg-trace int] [--mp-rec-hits float]
REMI CORRECT
# Transformation parameters
positional arguments:
REMI The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
CORRECT Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--run int Run ID.. Default: 104
--in-folder str Partial input path appended with run ID.. Default:
/gpfs/exfel/exp/SQS/202101/p002535/raw
--out-folder str Full path to output folder.. Default:
/gpfs/exfel/exp/SQS/202101/p002535/scratch/cal_test
--calib-config-path str
Path to correction and transform configuration.
Default: /gpfs/exfel/exp/SQS/202101/p002535/usr/config
_board2+4.yaml
--cycle str Proposal cycle, currently not used.. Default:
--cal-db-timeout int Calibration DB timeout, currently not used.. Default:
0
--cal-db-interface str
Calibration DB interface, currently not used..
Default: foo
--karabo-da str Karabo data aggregator name, currently not used.
Default: bar
--karabo-id str Karabo device ID root for virtual output device..
Default: SQS_REMI_DLD6
--proposal str Proposal, leave empty for auto detection based on
in_folder. Default:
--out-aggregator str Aggregator name for output files.. Default: REMI01
--out-seq-len int Number of trains per sequence file in output..
Default: 5000
--det-device-id str Karabo device ID for virtual output device.. Default:
{karabo_id}/DET/{det_name}
--det-output-key str Pipeline name for fast data output.. Default: output
--save-raw-triggers Whether to save trigger position in files.. Default:
True
--no-save-raw-triggers
Opposite of --save-raw-triggers (default: True)
--save-raw-edges Whether to save digitized edge positions in files..
Default: True
--no-save-raw-edges Opposite of --save-raw-edges (default: True)
--save-rec-signals Whether to save reconstructed signals (u1-w2, mcp) in
files.. Default: True
--no-save-rec-signals
Opposite of --save-rec-signals (default: True)
--save-rec-hits Whether to save reoncstructed hits (x,y,t,m) in
files.. Default: True
--no-save-rec-hits Opposite of --save-rec-hits (default: True)
--chunks-triggers int [int ...]
HDF chunk size for triggers.. Default: [500]
--chunks-edges int [int ...]
HDF chunk size for edges.. Default: [500, 7, 50]
--chunks-hits int [int ...]
HDF chunk size for hits.. Default: [50, 50]
--chunks-signals int [int ...]
HDF chunk size for signals.. Default: [50, 50]
--dataset-compression str
HDF compression method.. Default: gzip
--dataset-compression-opts int
HDF GZIP compression level.. Default: 3
--quad-anode Reconstruction assumes a hex anode by default, change
for quad anodes.. Default: False
--no-quad-anode Opposite of --quad-anode (default: False)
--ppt-source str Default: SQS_RR_UTC/TSYS/TIMESERVER:outputBunchPattern
--ignore-fel Ignore any FEL entries in the PPT.. Default: False
--no-ignore-fel Opposite of --ignore-fel (default: False)
--ignore-ppl Ignore any PPL entries in the PPT.. Default: False
--no-ignore-ppl Opposite of --ignore-ppl (default: False)
--ppl-offset int In units of the PPT.. Default: 0
--laser-ppt-mask int Bit mask for used laser, negative to auto-detect from
instrument.. Default: -1
--instrument-sase int
Default: 3
--first-pulse-offset int
Default: 1000
--single-pulse-length int
Default: 25000
--mp-find-triggers float
Parallelization for finding triggers.. Default: 0.5
--mp-find-edges float
Parallelization for digitizing analog signal..
Default: 0.5
--mt-avg-trace int Parallelization for trace averaging.. Default: 2
--mp-rec-hits float Parallelization for hit reconstruction.. Default: 1.0
TUTORIAL¶
Tutorial Calculation¶
Author: Astrid Muennich
Version: 0.1
A small example how to adapt a notebook to run with the offline calibration package “pycalibation”.
The first cell contains all parameters that should be exposed to the command line.
To run this notebooks with several different input parameters in parallel by submitting multiple slurm jobs, for example for various random seed we can do the following:
xfel-calibrate TUTORIAL TEST –random-seed 1,2,3,4
or
xfel-calibrate TUTORIAL TEST –random-seed 1-5
will produce 4 jobs:
Parsed input 1,2,3,4 to [1, 2, 3, 4]
Submitted job: 1169340
Submitted job: 1169341
Submitted job: 1169342
Submitted job: 1169343
Submitted the following SLURM jobs: 1169340,1169341,1169342,1169343
To invoke this notebook and display help use:
xfel-calibrate TUTORIAL TEST --help
The full parameter list of this notebook (with defaults is):
usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
[--report-to str] [--not-reproducible] [--skip-report]
[--skip-env-freeze] [--concurrency-par str]
[--constants-from str] [--vector-figs] [--slurm-mem int]
[--slurm-name str] [--slurm-scheduling int]
[--request-time str] [--slurm-partition str]
[--reservation str] [--out-folder str]
[--sensor-size int [int ...]]
[--random-seed str [str ...]] [--runs int]
[--cluster-profile str]
TUTORIAL TEST
# Tutorial Calculation #
,
Author: Astrid Muennich,
,
Version: 0.1,
,
A small example how to adapt a notebook to run with the offline calibration
package "pycalibation".,
,
The first cell contains all parameters that should be exposed to the command
line.,
,
To run this notebooks with several different input parameters in parallel by
submitting multiple slurm jobs, for example for various random seed we can do
the following:,
,
xfel-calibrate TUTORIAL TEST --random-seed 1,2,3,4,
,
or,
,
xfel-calibrate TUTORIAL TEST --random-seed 1-5,
,
will produce 4 jobs:,
,
Parsed input 1,2,3,4 to [1, 2, 3, 4],
,
Submitted job: 1169340,
,
Submitted job: 1169341,
,
Submitted job: 1169342,
,
Submitted job: 1169343,
,
Submitted the following SLURM jobs: 1169340,1169341,1169342,1169343
positional arguments:
TUTORIAL The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
TEST Type of calibration.
optional arguments:
-h, --help show this help message and exit
--no-cluster-job Do not run as a cluster job (default: False)
--prepare-only Prepare notebooks but don't run them (default: False)
--report-to str Filename (and optionally path) for output report
(default: None)
--not-reproducible Disable checks to allow the processing result to not
be reproducible based on its metadata. (default:
False)
--skip-report Skip report generation in finalize step. (default:
False)
--skip-env-freeze Skip recording the Python environment for
reproducibility purposes, requires --not-reproducible
to run. (default: False)
--concurrency-par str
Name of concurrency parameter.If not given, it is
taken from configuration. (default: None)
--constants-from str Path to a calibration-metadata.yml file. If given,
retrieved-constants will be copied to use for a new
correction. (default: None)
--vector-figs Use vector graphics for figures in the report.
(default: False)
--slurm-mem int Requested node RAM in GB (default: 500)
--slurm-name str Name of slurm job (default: xfel_calibrate)
--slurm-scheduling int
Change scheduling priority for a slurm job +-
2147483645 (negative value increases priority)
(default: 0)
--request-time str Time of request to process notebook. Iso format
(default: Now)
--slurm-partition str
Submit jobs in this Slurm partition (default: )
--reservation str Submit jobs in this Slurm reservation, overriding
--slurm-partition if both are set (default: )
--out-folder str output folder. Default:
/gpfs/exfel/data/scratch/amunnich/tutorial
--sensor-size int [int ...]
defining the picture size. Default: [10, 30]
--random-seed str [str ...]
random seed for filling of fake data array. Change it
to produce different results, range allowed. Default:
[2345]
--runs int how may iterations to fill histograms. Default: 500
--cluster-profile str
Default: tutorial