Available Notebooks

The following notebooks are currently integrated into the European XFEL Offline Calibration tool chain.

AGIPD

Combine Constants

Author: S. Hauf, Version: 0.1

This notebook combines constants from various evaluations

  • dark image analysis, yielding offset and noise
  • flat field analysis, yielding X-ray gain
  • pulse capacitor analysis, yielding medium gain stage slopes and thresholds
  • charge injection analysis, yielding low gain stage slopes and thresholds

into a single set of calibration constants. These constants do not include offset and noise as they need to be reevaluated more frequently.

Additionally, a bad pixel mask for all gain stages is deduced from the input. The mask contains dedicated entries for all pixels and memory cells as well as all three gains stages.

To invoke this notebook and display help use:

xfel-calibrate AGIPD COMBINE --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--modules str [str ...]]
                      --out-folder str [--local-output | --no-local-output]
                      [--db-output | --no-db-output] [--bias-voltage int]
                      [--cal-db-interface str] [--mem-cells int]
                      [--instrument str] [--photon-energy float]
                      [--offset-store str] [--gain-store str]
                      [--pc-store-base str] [--ci-store str]
                      [--high-res-badpix-3d | --no-high-res-badpix-3d]
                      [--db-input | --no-db-input]
                      [--deviation-threshold float]
                      [--max-dev-high-gain float] [--max-dev-med-gain float]
                      [--threshold-bounds-high-med int [int ...]]
                      [--thresholds-offset-sigma float]
                      [--thresholds-offset-hard int [int ...]]
                      [--thresholds-noise-sigma float]
                      [--thresholds-noise-hard int [int ...]]
                      [--thresholds-xraygain-hard-high float [float ...]]
                      [--thresholds-xraygain-hard-med float [float ...]]
                      [--thresholds-xraygain-hard-low float [float ...]]
                      [--no-flat-fields | --no-no-flat-fields]
                      AGIPD COMBINE

# Combine Constants #
,
Author: S. Hauf, Version: 0.1,
,
This notebook combines constants from various evaluations,
,
* dark image analysis, yielding offset and noise,
* flat field analysis, yielding X-ray gain,
* pulse capacitor analysis, yielding medium gain stage slopes and thresholds,
* charge injection analysis, yielding low gain stage slopes and thresholds,
,
into a single set of calibration constants. These constants do not include
offset and noise as they need to be reevaluated more frequently.,
,
Additionally, a bad pixel mask for all gain stages is deduced from the input.
The mask contains dedicated entries for all pixels and memory cells as well
as all three gains stages.

positional arguments:
  AGIPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  COMBINE                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --modules str [str ...]
                        modules to work on, range allowed. Default: [-1]
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --bias-voltage int    detector bias voltage. Default: 300
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:5005
  --mem-cells int       number of memory cells used. Default: 64
  --instrument str      Default: SPB
  --photon-energy float
                        the photon energy in keV. Default: 9.2
  --offset-store str    for file-based access. Default: /gpfs/exfel/exp/SPB/20
                        1830/p900019/usr/calibration0618/dark0.5MHz/agipd_offs
                        et_store_r0852_r0853_r0854.h5
  --gain-store str      for file-based access. Default: /gpfs/exfel/exp/SPB/20
                        1830/p900019/usr/calibration0618/FF/agipd_gain_store_r
                        0820_modules_CHANID.h5
  --pc-store-base str   for file-based access, use CHANID to indicate module
                        ranges. Default: /gpfs/exfel/exp/SPB/201831/p900039/us
                        r/AGIPD/PC/Veto_1MHZ/agipd_pc_store_38_39_40_41_42_43_
                        44_60_CHANID_CHANID.h5
  --ci-store str        for file based access. Default: /gpfs/exfel/data/scrat
                        ch/haufs/gain_Data/AGIPD_ci_data_74.h5
  --high-res-badpix-3d  set this to True if you need high-resolution 3d bad
                        pixel plots. Runtime: ~ 1h. Default: False
  --no-high-res-badpix-3d
                        Opposite of --high-res-badpix-3d (default: False)
  --db-input            retreive data from calibration database, setting
                        offset-store will overwrite this. Default: False
  --no-db-input         Opposite of --db-input (default: False)
  --deviation-threshold float
                        peaks with an absolute location deviation larger than
                        the medium are are considere bad. Default: 0.75
  --max-dev-high-gain float
                        Default: 0.2
  --max-dev-med-gain float
                        Default: 0.5
  --threshold-bounds-high-med int [int ...]
                        Default: [100, 8100]
  --thresholds-offset-sigma float
                        Default: 5.0
  --thresholds-offset-hard int [int ...]
                        Default: [4000, 9000]
  --thresholds-noise-sigma float
                        Default: 10.0
  --thresholds-noise-hard int [int ...]
                        Default: [1, 20]
  --thresholds-xraygain-hard-high float [float ...]
                        Default: [0.75, 1.25]
  --thresholds-xraygain-hard-med float [float ...]
                        Default: [0.001, 0.25]
  --thresholds-xraygain-hard-low float [float ...]
                        Default: [0.0001, 0.025]
  --no-flat-fields      Default: False
  --no-no-flat-fields   Opposite of --no-flat-fields (default: False)

required arguments:
  --out-folder str      path to output to, required. Default: None

AGIPD Offline Correction

Author: European XFEL Detector Group, Version: 2.0

Offline Calibration for the AGIPD Detector

To invoke this notebook and display help use:

xfel-calibrate AGIPD CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      [--sequences str [str ...]] [--modules str [str ...]]
                      [--train-ids str [str ...]] --run int [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-template str]
                      [--path-template str] [--instrument-source-template str]
                      [--index-source-template str]
                      [--ctrl-source-template str] [--karabo-id-control str]
                      [--slopes-ff-from-files str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--creation-date-offset str] [--mem-cells int]
                      [--bias-voltage int] [--acq-rate float]
                      [--gain-setting int] [--gain-mode int]
                      [--overwrite | --no-overwrite]
                      [--max-pulses int [int ...]] [--mem-cells-db int]
                      [--integration-time int] [--blc-noise-threshold int]
                      [--cm-dark-fraction float]
                      [--cm-dark-range float [float ...]] [--cm-n-itr int]
                      [--hg-hard-threshold int] [--mg-hard-threshold int]
                      [--noisy-adc-threshold float] [--ff-gain float]
                      [--photon-energy float]
                      [--only-offset | --no-only-offset]
                      [--rel-gain | --no-rel-gain]
                      [--xray-gain | --no-xray-gain]
                      [--blc-noise | --no-blc-noise]
                      [--blc-stripes | --no-blc-stripes]
                      [--blc-hmatch | --no-blc-hmatch]
                      [--match-asics | --no-match-asics]
                      [--adjust-mg-baseline | --no-adjust-mg-baseline]
                      [--zero-nans | --no-zero-nans]
                      [--zero-orange | --no-zero-orange]
                      [--blc-set-min | --no-blc-set-min]
                      [--corr-asic-diag | --no-corr-asic-diag]
                      [--force-hg-if-below | --no-force-hg-if-below]
                      [--force-mg-if-below | --no-force-mg-if-below]
                      [--mask-noisy-adc | --no-mask-noisy-adc]
                      [--common-mode | --no-common-mode]
                      [--melt-snow | --no-melt-snow]
                      [--mask-zero-std | --no-mask-zero-std]
                      [--low-medium-gap | --no-low-medium-gap]
                      [--round-photons | --no-round-photons]
                      [--use-ppu-device str] [--ppu-train-offset int]
                      [--use-litframe-finder str] [--litframe-device-id str]
                      [--energy-threshold int] [--use-xgm-device str]
                      [--recast-image-data str]
                      [--compress-fields str [str ...]]
                      [--skip-plots | --no-skip-plots] [--cell-id-preview int]
                      [--chunk-size int] [--n-cores-correct int]
                      [--n-cores-files int] [--sequences-per-node int]
                      [--max-nodes int] [--max-tasks-per-worker int]
                      AGIPD CORRECT

# AGIPD Offline Correction #
,
Author: European XFEL Detector Group, Version: 2.0,
,
Offline Calibration for the AGIPD Detector

positional arguments:
  AGIPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequences str [str ...]
                        sequences to correct, set to -1 for all, range
                        allowed. Default: [-1]
  --modules str [str ...]
                        modules to correct, set to -1 for all, range allowed.
                        Default: [-1]
  --train-ids str [str ...]
                        train IDs to correct, set to -1 for all, range
                        allowed. Default: [-1]
  --karabo-id str       karabo karabo_id. Default: MID_DET_AGIPD1M-1
  --karabo-da str [str ...]
                        a list of data aggregators names, Default [-1] for
                        selecting all data aggregators. Default: ['-1']
  --receiver-template str
                        inset for receiver devices. Default: {}CH0
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{:05d}.h5
  --instrument-source-template str
                        path in the HDF5 file to images. Default:
                        {}/DET/{}:xtdf
  --index-source-template str
                        path in the HDF5 file to images. Default:
                        INDEX/{}/DET/{}:xtdf/
  --ctrl-source-template str
                        path to control information. Default: {}/MDL/FPGA_COMP
  --karabo-id-control str
                        karabo-id for control device. Default:
                        MID_EXP_AGIPD1M1
  --slopes-ff-from-files str
                        Path to locally stored SlopesFF and BadPixelsFF
                        constants, loaded in precorrection notebook. Default:
  --use-dir-creation-date
                        use the creation data of the input dir for database
                        queries. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8015#8045
  --cal-db-timeout int  in milliseconds. Default: 30000
  --creation-date-offset str
                        add an offset to creation date, e.g. to get different
                        constants. Default: 00:00:00
  --mem-cells int       Number of memory cells used, set to 0 to automatically
                        infer. Default: 0
  --bias-voltage int    bias voltage, set to 0 to use stored value in slow
                        data.. Default: 0
  --acq-rate float      the detector acquisition rate, use 0 to try to auto-
                        determine. Default: 0.0
  --gain-setting int    the gain setting, use -1 to use value stored in slow
                        data.. Default: -1
  --gain-mode int       gain mode (0: adaptive, 1-3 fixed high/med/low, -1:
                        read from CONTROL data). Default: -1
  --overwrite           set to True if existing data should be overwritten.
                        Default: True
  --no-overwrite        Opposite of --overwrite (default: True)
  --max-pulses int [int ...]
                        range list [st, end, step] of memory cell indices to
                        be processed within a train. 3 allowed maximum list
                        input elements.. Default: [0, 352, 1]
  --mem-cells-db int    set to a value different than 0 to use this value for
                        DB queries. Default: 0
  --integration-time int
                        integration time, negative values for auto-detection..
                        Default: -1
  --blc-noise-threshold int
                        above this mean signal intensity now baseline
                        correction via noise is attempted. Default: 5000
  --cm-dark-fraction float
                        threshold for fraction of empty pixels to consider
                        module enough dark to perform CM correction. Default:
                        0.66
  --cm-dark-range float [float ...]
                        range for signal value ADU for pixel to be consider as
                        a dark pixel. Default: [-50.0, 30]
  --cm-n-itr int        number of iterations for common mode correction.
                        Default: 4
  --hg-hard-threshold int
                        threshold to force medium gain offset subtracted pixel
                        to high gain. Default: 1000
  --mg-hard-threshold int
                        threshold to force medium gain offset subtracted pixel
                        from low to medium gain. Default: 1000
  --noisy-adc-threshold float
                        threshold to mask complete adc. Default: 0.25
  --ff-gain float       conversion gain for absolute FlatField constants,
                        while applying xray_gain. Default: 7.2
  --photon-energy float
                        photon energy in keV, non-positive value for XGM
                        autodetection. Default: -1.0
  --only-offset         Apply only Offset correction. if False, Offset is
                        applied by Default. if True, Offset is only applied..
                        Default: False
  --no-only-offset      Opposite of --only-offset (default: False)
  --rel-gain            do relative gain correction based on PC data. Default:
                        False
  --no-rel-gain         Opposite of --rel-gain (default: False)
  --xray-gain           do relative gain correction based on xray data.
                        Default: False
  --no-xray-gain        Opposite of --xray-gain (default: False)
  --blc-noise           if set, baseline correction via noise peak location is
                        attempted. Default: False
  --no-blc-noise        Opposite of --blc-noise (default: False)
  --blc-stripes         if set, baseline corrected via stripes. Default: False
  --no-blc-stripes      Opposite of --blc-stripes (default: False)
  --blc-hmatch          if set, base line correction via histogram matching is
                        attempted. Default: False
  --no-blc-hmatch       Opposite of --blc-hmatch (default: False)
  --match-asics         if set, inner ASIC borders are matched to the same
                        signal level. Default: False
  --no-match-asics      Opposite of --match-asics (default: False)
  --adjust-mg-baseline  adjust medium gain baseline to match highest high gain
                        value. Default: False
  --no-adjust-mg-baseline
                        Opposite of --adjust-mg-baseline (default: False)
  --zero-nans           set NaN values in corrected data to 0. Default: False
  --no-zero-nans        Opposite of --zero-nans (default: False)
  --zero-orange         set to 0 very negative and very large values in
                        corrected data. Default: False
  --no-zero-orange      Opposite of --zero-orange (default: False)
  --blc-set-min         Shift to 0 negative medium gain pixels after offset
                        corr. Default: False
  --no-blc-set-min      Opposite of --blc-set-min (default: False)
  --corr-asic-diag      if set, diagonal drop offs on ASICs are corrected.
                        Default: False
  --no-corr-asic-diag   Opposite of --corr-asic-diag (default: False)
  --force-hg-if-below   set high gain if mg offset subtracted value is below
                        hg_hard_threshold. Default: False
  --no-force-hg-if-below
                        Opposite of --force-hg-if-below (default: False)
  --force-mg-if-below   set medium gain if mg offset subtracted value is below
                        mg_hard_threshold. Default: False
  --no-force-mg-if-below
                        Opposite of --force-mg-if-below (default: False)
  --mask-noisy-adc      Mask entire ADC if they are noise above a relative
                        threshold. Default: False
  --no-mask-noisy-adc   Opposite of --mask-noisy-adc (default: False)
  --common-mode         Common mode correction. Default: False
  --no-common-mode      Opposite of --common-mode (default: False)
  --melt-snow           Identify (and optionally interpolate) 'snowy' pixels.
                        Default: False
  --no-melt-snow        Opposite of --melt-snow (default: False)
  --mask-zero-std       Mask pixels with zero standard deviation across train.
                        Default: False
  --no-mask-zero-std    Opposite of --mask-zero-std (default: False)
  --low-medium-gap      5 sigma separation in thresholding between low and
                        medium gain. Default: False
  --no-low-medium-gap   Opposite of --low-medium-gap (default: False)
  --round-photons       Round to absolute number of photons, only use with
                        gain corrections. Default: False
  --no-round-photons    Opposite of --round-photons (default: False)
  --use-ppu-device str  Device ID for a pulse picker device to only process
                        picked trains, empty string to disable. Default:
  --ppu-train-offset int
                        When using the pulse picker, offset between the PPU's
                        sequence start and actually picked train. Default: 0
  --use-litframe-finder str
                        Process only illuminated frames: 'off' - disable,
                        'device' - use online device data, 'offline' - use
                        offline algorithm, 'auto' - choose online/offline
                        source automatically (default). Default: off
  --litframe-device-id str
                        Device ID for a lit frame finder device, empty string
                        to auto detection. Default:
  --energy-threshold int
                        The low limit for the energy (uJ) exposed by frames
                        subject to processing. If -1000, selection by pulse
                        energy is disabled. Default: -1000
  --use-xgm-device str  DoocsXGM device ID to obtain actual photon energy,
                        operating condition else.. Default:
  --recast-image-data str
                        Cast data to a different dtype before saving. Default:
  --compress-fields str [str ...]
                        Datasets in image group to compress.. Default:
                        ['gain', 'mask']
  --skip-plots          exit after writing corrected files and metadata.
                        Default: False
  --no-skip-plots       Opposite of --skip-plots (default: False)
  --cell-id-preview int
                        cell Id used for preview in single-shot plots.
                        Default: 1
  --chunk-size int      Size of chunk for image-wise correction. Default: 1000
  --n-cores-correct int
                        Number of chunks to be processed in parallel. Default:
                        16
  --n-cores-files int   Number of files to be processed in parallel. Default:
                        4
  --sequences-per-node int
                        number of sequence files per cluster node if run as
                        SLURM job, set to 0 to not run SLURM parallel.
                        Default: 2
  --max-nodes int       Maximum number of SLURM jobs to split correction work
                        into. Default: 8
  --max-tasks-per-worker int
                        the number of tasks a correction pool worker process
                        can complete before it will exit and be replaced with
                        a fresh worker process. Leave as -1 to keep worker
                        alive as long as pool.. Default: 1

required arguments:
  --in-folder str       the folder to read data from, required. Default: None
  --out-folder str      the folder to output to, required. Default: None
  --run int             runs to process, required. Default: None

AGIPD Characterize Dark Images

Author: European XFEL Detector Group, Version: 2.0

The following code analyzes a set of dark images taken with the AGIPD detector to deduce detector offsets , noise, bad-pixel maps and thresholding. All four types of constants are evaluated per-pixel and per-memory cell. Data for the detector’s three gain stages needs to be present, separated into separate runs.

The evaluated calibration constants are stored locally and injected in the calibration data base.

To invoke this notebook and display help use:

xfel-calibrate AGIPD DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      [--modules str [str ...]] --run-high int --run-med int
                      --run-low int [--operation-mode str] [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-template str]
                      [--instrument-source-template str]
                      [--ctrl-source-template str] [--karabo-id-control str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--local-output | --no-local-output]
                      [--db-output | --no-db-output] [--mem-cells int]
                      [--bias-voltage int] [--gain-setting int]
                      [--gain-mode int] [--integration-time int]
                      [--acq-rate float] [--interlaced | --no-interlaced]
                      [--thresholds-offset-sigma float]
                      [--thresholds-offset-hard int [int ...]]
                      [--thresholds-offset-hard-hg int [int ...]]
                      [--thresholds-offset-hard-mg int [int ...]]
                      [--thresholds-offset-hard-lg int [int ...]]
                      [--thresholds-offset-hard-hg-fixed int [int ...]]
                      [--thresholds-offset-hard-mg-fixed int [int ...]]
                      [--thresholds-offset-hard-lg-fixed int [int ...]]
                      [--thresholds-noise-sigma float]
                      [--thresholds-noise-hard int [int ...]]
                      [--thresholds-noise-hard-hg int [int ...]]
                      [--thresholds-noise-hard-mg int [int ...]]
                      [--thresholds-noise-hard-lg int [int ...]]
                      [--thresholds-gain-sigma float] [--max-trains int]
                      [--min-trains int]
                      [--high-res-badpix-3d | --no-high-res-badpix-3d]
                      AGIPD DARK

# AGIPD Characterize Dark Images #
,
Author: European XFEL Detector Group, Version: 2.0,
,
The following code analyzes a set of dark images taken with the AGIPD
detector to deduce detector offsets  noise, bad-pixel maps and thresholding.
All four types of constants are evaluated per-pixel and per-memory cell. Data
"for the detectors three gain stages needs to be present, separated into "
separate runs.,
,
The evaluated calibration constants are stored locally and injected in the
calibration data base.,

positional arguments:
  AGIPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --modules str [str ...]
                        list of modules to evaluate, RANGE ALLOWED. Default:
                        [-1]
  --operation-mode str  Detector operation mode, optional (defaults to
                        "ADAPTIVE_GAIN"). Default: ADAPTIVE_GAIN
  --karabo-id str       karabo karabo_id. Default: HED_DET_AGIPD500K2G
  --karabo-da str [str ...]
                        a list of data aggregators names, Default [-1] for
                        selecting all data aggregators. Default: ['-1']
  --receiver-template str
                        inset for receiver devices. Default: {}CH0
  --instrument-source-template str
                        path in the HDF5 file to images. Default:
                        {}/DET/{}:xtdf
  --ctrl-source-template str
                        path to control information. Default: {}/MDL/FPGA_COMP
  --karabo-id-control str
                        karabo-id for control device '. Default:
                        HED_EXP_AGIPD500K2G
  --use-dir-creation-date
                        use dir creation date as data production reference
                        date. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8020
  --cal-db-timeout int  timeout on caldb requests". Default: 3000000
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --mem-cells int       number of memory cells used, set to 0 to automatically
                        infer. Default: 0
  --bias-voltage int    bias voltage, set to 0 to use stored value in slow
                        data.. Default: 0
  --gain-setting int    the gain setting, use -1 to use value stored in slow
                        data.. Default: -1
  --gain-mode int       gain mode, use -1 to use value stored in slow data..
                        Default: -1
  --integration-time int
                        integration time, negative values for auto-detection..
                        Default: -1
  --acq-rate float      the detector acquisition rate, use 0 to try to auto-
                        determine. Default: 0.0
  --interlaced          assume interlaced data format, for data prior to Dec.
                        2017. Default: False
  --no-interlaced       Opposite of --interlaced (default: False)
  --thresholds-offset-sigma float
                        offset sigma thresholds for offset deduced bad pixels.
                        Default: 3.0
  --thresholds-offset-hard int [int ...]
                        For setting the same threshold offset for the 3 gains.
                        Left for backcompatability. Default [0, 0] to take the
                        following parameters.. Default: [0, 0]
  --thresholds-offset-hard-hg int [int ...]
                        High-gain thresholds in absolute ADU terms for offset
                        deduced bad pixels. Default: [3000, 7000]
  --thresholds-offset-hard-mg int [int ...]
                        Medium-gain thresholds in absolute ADU terms for
                        offset deduced bad pixels. Default: [6000, 10000]
  --thresholds-offset-hard-lg int [int ...]
                        Low-gain thresholds in absolute ADU terms for offset
                        deduced bad pixels. Default: [6000, 10000]
  --thresholds-offset-hard-hg-fixed int [int ...]
                        Same as thresholds_offset_hard_hg, but for fixed gain
                        operation. Default: [3500, 6500]
  --thresholds-offset-hard-mg-fixed int [int ...]
                        Same as thresholds_offset_hard_mg, but for fixed gain
                        operation. Default: [3500, 6500]
  --thresholds-offset-hard-lg-fixed int [int ...]
                        Same as thresholds_offset_hard_lg, but for fixed gain
                        operation. Default: [3500, 6500]
  --thresholds-noise-sigma float
                        noise sigma thresholds for offset deduced bad pixels.
                        Default: 5.0
  --thresholds-noise-hard int [int ...]
                        For setting the same threshold noise for the 3 gains.
                        Left for backcompatability. Default [0, 0] to take the
                        following parameters.. Default: [0, 0]
  --thresholds-noise-hard-hg int [int ...]
                        High-gain thresholds in absolute ADU terms for offset
                        deduced bad pixels. Default: [4, 20]
  --thresholds-noise-hard-mg int [int ...]
                        Medium-gain thresholds in absolute ADU terms for
                        offset deduced bad pixels. Default: [4, 20]
  --thresholds-noise-hard-lg int [int ...]
                        Low-gain thresholds in absolute ADU terms for offset
                        deduced bad pixels. Default: [4, 20]
  --thresholds-gain-sigma float
                        Gain separation sigma threshold. Default: 5.0
  --max-trains int      Maximum number of trains to use for processing dark.
                        Set to 0 to process all available trains. 550 added
                        for ~500GB nodes to temporarely avoid memory issues..
                        Default: 550
  --min-trains int      Miniumum number of trains for processing dark. If run
                        folder has less than minimum trains, processing is
                        stopped.. Default: 1
  --high-res-badpix-3d  set this to True if you need high-resolution 3d bad
                        pixel plots. ~7mins extra time for 64 memory cells.
                        Default: False
  --no-high-res-badpix-3d
                        Opposite of --high-res-badpix-3d (default: False)

required arguments:
  --in-folder str       path to input data, required. Default: None
  --out-folder str      path to output to, required. Default: None
  --run-high int        run number in which high gain data was recorded,
                        required. Default: None
  --run-med int         run number in which medium gain data was recorded,
                        required. Default: None
  --run-low int         run number in which low gain data was recorded,
                        required. Default: None

Gain Characterization

To invoke this notebook and display help use:

xfel-calibrate AGIPD FF --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      [--hist-file-template str] [--modules str [str ...]]
                      [--raw-folder str] [--proc-folder str] [--run int]
                      [--karabo-id str] [--karabo-da str [str ...]]
                      [--receiver-id str] [--path-template str] [--h5path str]
                      [--h5path-idx str] [--h5path-ctrl str]
                      [--karabo-id-control str] [--karabo-da-control str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--local-output | --no-local-output]
                      [--db-output | --no-db-output]
                      [--peak-range int [int ...]]
                      [--peak-width-range int [int ...]]
                      [--peak-norm-range float [float ...]]
                      [--peak-lim int [int ...]] [--d0-lim int [int ...]]
                      [--peak-width-lim float [float ...]]
                      [--chi2-lim int [int ...]] [--intensity-lim int]
                      [--gain-lim float [float ...]]
                      [--cell-range int [int ...]]
                      [--pixel-range int [int ...]] [--max-bins int]
                      [--batch-size int [int ...]] [--fit-range int [int ...]]
                      [--n-peaks-fit int] [--fix-peaks | --no-fix-peaks]
                      [--do-minos | --no-do-minos] [--sigma-limit float]
                      [--mem-cells int] [--bias-voltage int]
                      [--acq-rate float] [--gain-setting int]
                      [--photon-energy float] [--integration-time int]
                      AGIPD FF

# Gain Characterization #

positional arguments:
  AGIPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  FF                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --hist-file-template str
                        the template to use to access histograms. Default:
                        hists_m{:02d}_sum.h5
  --modules str [str ...]
                        modules to correct, set to -1 for all, range allowed.
                        Default: [10]
  --raw-folder str      Path to raw image data used to create histograms.
                        Default: /gpfs/exfel/exp/MID/202030/p900137/raw
  --proc-folder str     Path to corrected image data used to create
                        histograms. Default:
  --run int             of the run of image data used to create histograms.
                        Default: 449
  --karabo-id str       karabo karabo_id. Default: MID_DET_AGIPD1M-1
  --karabo-da str [str ...]
                        a list of data aggregators names, Default [-1] for
                        selecting all data aggregators. Default: ['-1']
  --receiver-id str     inset for receiver devices. Default: {}CH0
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{:05d}.h5
  --h5path str          path in the HDF5 file to images. Default:
                        INSTRUMENT/{}/DET/{}:xtdf/
  --h5path-idx str      path in the HDF5 file to images. Default:
                        INDEX/{}/DET/{}:xtdf/
  --h5path-ctrl str     path to control information. Default:
                        /CONTROL/{}/MDL/FPGA_COMP
  --karabo-id-control str
                        karabo-id for control device. Default:
                        MID_IRU_AGIPD1M1
  --karabo-da-control str
                        karabo DA for control infromation. Default:
                        AGIPD1MCTRL00
  --use-dir-creation-date
                        use the creation data of the input dir for database
                        queries. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8015#8045
  --cal-db-timeout int  in milli seconds. Default: 30000
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --peak-range int [int ...]
                        where to look for the peaks, [a0, b0, a1, b1, ...]
                        exactly 8 elements. Default: [-30, 30, 35, 70, 95,
                        135, 145, 220]
  --peak-width-range int [int ...]
                        fit limits on the peak widths, [a0, b0, a1, b1, ...]
                        exactly 8 elements. Default: [0, 30, 0, 35, 0, 40, 0,
                        45]
  --peak-norm-range float [float ...]
                        . Default: [0.0, -1, 0, -1, 0, -1, 0, -1]
  --peak-lim int [int ...]
                        Limit of position of noise peak. Default: [-30, 30]
  --d0-lim int [int ...]
                        hard limits for distance between noise and first peak.
                        Default: [10, 80]
  --peak-width-lim float [float ...]
                        hard limits on the peak widths for first and second
                        peak, in units of the noise peak. 4 parameters..
                        Default: [0.9, 1.55, 0.95, 1.65]
  --chi2-lim int [int ...]
                        Hard limit on chi2/nDOF value. Default: [0, 3.0]
  --intensity-lim int   Threshold on standard deviation of a histogram in ADU.
                        Contribute to BadPixel bit "No_Entry". Default: 15
  --gain-lim float [float ...]
                        Threshold on gain in relative number. Contribute to
                        BadPixel bit "Gain_deviation". Default: [0.8, 1.2]
  --cell-range int [int ...]
                        range of cell to be considered, [0,0] for all.
                        Default: [1, 3]
  --pixel-range int [int ...]
                        range of pixels x1,y1,x2,y2 to consider [0,0,512,128]
                        for all. Default: [0, 0, 32, 32]
  --max-bins int        Maximum number of bins to consider, 0 for all bins.
                        Default: 0
  --batch-size int [int ...]
                        batch size: [cell,x,y]. Default: [1, 8, 8]
  --fit-range int [int ...]
                        range of a histogram considered for fitting in ADU.
                        Dynamically evaluated in case [0,0]. Default: [0, 0]
  --n-peaks-fit int     Number of gaussian peaks to fit including noise peak.
                        Default: 4
  --fix-peaks           Fix distance between photon peaks. Default: False
  --no-fix-peaks        Opposite of --fix-peaks (default: False)
  --do-minos            This is additional feature of minuit to evaluate
                        errors.. Default: False
  --no-do-minos         Opposite of --do-minos (default: False)
  --sigma-limit float   If >0, repeat fit keeping only bins within mu +-
                        sigma_limit*sigma. Default: 0.0
  --mem-cells int       number of memory cells used, negative values for auto-
                        detection.. Default: -1
  --bias-voltage int    Bias voltage.. Default: 300
  --acq-rate float      the detector acquisition rate, use 0 to try to auto-
                        determine.. Default: 0.0
  --gain-setting int    the gain setting, negative values for auto-detection..
                        Default: -1
  --photon-energy float
                        photon energy in keV.. Default: 8.05
  --integration-time int
                        integration time, negative values for auto-detection..
                        Default: -1

required arguments:
  --in-folder str       the folder to read histograms from, required. Default:
                        None
  --out-folder str      the folder to output to, required. Default: None

Histogramming of AGIPD FF data

Author: European XFEL Detector Group, Version: 1.0

Offline Calibration for the AGIPD Detector

To invoke this notebook and display help use:

xfel-calibrate AGIPD FF_HISTS --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      [--sequences str [str ...]] [--modules str [str ...]]
                      --runs int [int ...] [--cells-list str [str ...]]
                      [--karabo-id str] [--karabo-da str [str ...]]
                      [--receiver-id str] [--path-template str] [--h5path str]
                      [--n-bins-adu int] [--h-range int [int ...]]
                      [--n-cells int] [--hist-std-tresh int]
                      [--hist-mean-tresh int] [--h-sums-tresh int]
                      [--n-cores-hists int] [--n-cores-files int]
                      AGIPD FF_HISTS

# Histogramming of AGIPD FF data
,
Author: European XFEL Detector Group, Version: 1.0,
,
Offline Calibration for the AGIPD Detector

positional arguments:
  AGIPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  FF_HISTS                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequences str [str ...]
                        module to consider, set to -1 for all, range allowed.
                        Default: [-1]
  --modules str [str ...]
                        module to consider, range allowed. Default: [5]
  --cells-list str [str ...]
                        list of lists or any python expressions, which can be
                        converted to a list. E.g. 'range(0,15,5)'
                        'list(range(0,5))+list(range(50,60))'. Default:
                        ['range(0,15)']
  --karabo-id str       karabo karabo_id. Default: MID_DET_AGIPD1M-1
  --karabo-da str [str ...]
                        a list of data aggregators names, Default [-1] for
                        selecting all data aggregators. Default: ['-1']
  --receiver-id str     inset for receiver devices. Default: {}CH0
  --path-template str   the template to use to access data. Default:
                        CORR-R{:04d}-AGIPD{:02d}-S{}.h5
  --h5path str          path in the HDF5 file to images. Default:
                        INSTRUMENT/{}/DET/{}:xtdf/image/
  --n-bins-adu int      number of bins per ADU. Default: 1
  --h-range int [int ...]
                        range of the histogram in ADU. Default: [-50, 450]
  --n-cells int         total number of memory cells (used to create summary
                        file). Default: 202
  --hist-std-tresh int  Threshold for histogram standard deviation. Default:
                        10
  --hist-mean-tresh int
                        Threshold for histogram mean. Default: 15
  --h-sums-tresh int    Threshold for number of entries in histograms.
                        Default: 100
  --n-cores-hists int   Number of processes (cores) to create histograms.
                        Default: 20
  --n-cores-files int   Number of processes (cores) to read files. Default: 10

required arguments:
  --in-folder str       the folder to read data from, required. Default: None
  --out-folder str      the folder to output to, required. Default: None
  --runs int [int ...]  list of run numbers, required. Default: None

Characterize AGIPD Pulse Capacitor Data

Author: Detector Operations Group, Version 1.0

The following code characterizes AGIPD gain via data take with the pulse capacitor source (PCS). The PCS allows scanning through the high and medium gains of AGIPD, by subsequently intecreasing the number of charge pulses from a on-ASIC capicitor, thus increasing the charge a pixel sees in a given integration time.

Because induced charge does not originate from X-rays on the sensor, the gains evaluated here will later need to be rescaled with gains deduced from X-ray data.

PCS data is organized into multiple runs, as the on-ASIC current source cannot supply all pixels of a given module with charge at the same time. Hence, only certain pixel rows will have seen charge for a given image. These rows then first need to be combined into single module images again. This script uses new style of merging, which does not support old data format.

We then use a K-means clustering algorithm to identify components in the resulting per-pixel data series, matching to three general regions:

  • a high gain slope
  • a transition region, where gain switching occurs
  • a medium gain slope.

The same regions are present in the gain-bit data and are used to deduce the switching threshold.

The resulting slopes are then fitted with a linear function and a combination of a linear and exponential decay function to determine the relative gains of the pixels with respect to the module. Additionally, we deduce masks for bad pixels form the data.

To invoke this notebook and display help use:

xfel-calibrate AGIPD PC --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str --runs str [str ...]
                      [--n-sequences int] [--modules str [str ...]]
                      [--karabo-da str [str ...]] [--karabo-da-control str]
                      [--karabo-id-control str] [--karabo-id str]
                      [--ctrl-source-template str]
                      [--instrument-source-template str]
                      [--receiver-template str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--delta-time int] [--cal-db-interface str]
                      [--local-output | --no-local-output]
                      [--db-output | --no-db-output] [--bias-voltage int]
                      [--mem-cells int] [--acq-rate float]
                      [--gain-setting int] [--integration-time int]
                      [--FF-gain float] [--fit-hook | --no-fit-hook]
                      [--high-res-badpix-3d | --no-high-res-badpix-3d]
                      [--hg-range int [int ...]] [--mg-range int [int ...]]
                      [--sigma-dev-cut float]
                      AGIPD PC

# Characterize AGIPD Pulse Capacitor Data #
,
Author: Detector Operations Group, Version 1.0,
,
The following code characterizes AGIPD gain via data take with the pulse
capacitor source (PCS). The PCS allows scanning through the high and medium
gains of AGIPD, by subsequently intecreasing the number of charge pulses from
a on-ASIC capicitor, thus increasing the charge a pixel sees in a given
integration time.,
,
Because induced charge does not originate from X-rays on the sensor, the
gains evaluated here will later need to be rescaled with gains deduced from
X-ray data.,
,
PCS data is organized into multiple runs, as the on-ASIC current source
cannot supply all pixels of a given module with charge at the same time.
Hence, only certain pixel rows will have seen charge for a given image. These
rows then first need to be combined into single module images again. This
script uses new style of merging, which does not support old data format.,
,
We then use a K-means clustering algorithm to identify components in the
resulting per-pixel data series, matching to three general regions:,
,
* a high gain slope,
* a transition region, where gain switching occurs,
* a medium gain slope.,
,
The same regions are present in the gain-bit data and are used to deduce the
switching threshold.
,
The resulting slopes are then fitted with a linear function and a combination
of a linear and exponential decay function to determine the relative gains of
the pixels with respect to the module. Additionally, we deduce masks for bad
pixels form the data.

positional arguments:
  AGIPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  PC                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        The ipcluster profile to use. Default: noDB
  --n-sequences int     number of sequence files, starting for 0 to evaluate.
                        Default: 5
  --modules str [str ...]
                        modules to work on, required, range allowed. Default:
                        [-1]
  --karabo-da str [str ...]
                        Default: ['all']
  --karabo-da-control str
                        karabo DA for control infromation. Default:
                        AGIPD1MCTRL00
  --karabo-id-control str
                        karabo-id for the control device e.g.
                        "MID_EXP_AGIPD1M1", or "SPB_IRU_AGIPD1M1". Default:
                        SPB_IRU_AGIPD1M1
  --karabo-id str       Default: SPB_DET_AGIPD1M-1
  --ctrl-source-template str
                        path to control information. Default: {}/MDL/FPGA_COMP
  --instrument-source-template str
                        path in the HDF5 file to images. Default:
                        {}/DET/{}:xtdf
  --receiver-template str
                        inset for receiver devices. Default: {}CH0
  --use-dir-creation-date
                        Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --delta-time int      offset to the creation time (e.g. useful in case we
                        want to force the system to use diff. dark constants).
                        Default: 0
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8019
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --bias-voltage int    detector bias voltage, negative values for auto-
                        detection.. Default: -1
  --mem-cells int       number of memory cells used, negative values for auto-
                        detection.. Default: -1
  --acq-rate float      the detector acquisition rate, negative values for
                        auto-detection.. Default: -1.0
  --gain-setting int    gain setting can have value 0 or 1, negative values
                        for auto-detection.. Default: -1
  --integration-time int
                        integration time, negative values for auto-detection..
                        Default: -1
  --FF-gain float       ADU/keV, gain from FF to convert ADU to keV. Default:
                        7.3
  --fit-hook            fit a hook function to medium gain slope --> run
                        without hook. Default: False
  --no-fit-hook         Opposite of --fit-hook (default: False)
  --high-res-badpix-3d  set this to True if you need high-resolution 3d bad
                        pixel plots. Runtime: ~ 1h. Default: False
  --no-high-res-badpix-3d
                        Opposite of --high-res-badpix-3d (default: False)
  --hg-range int [int ...]
                        [0,-150] range for linear fit. If upper edge is
                        negative use clustering result (bound_hg - 20).
                        Default: [30, 210]
  --mg-range int [int ...]
                        [-350,600] range for linear fit. If lower edge is
                        negative use clustering result (bound_mg + 20).
                        Default: [-277, 600]
  --sigma-dev-cut float
                        parameters outside the range median +-
                        sigma_dev_cut*MAD are replaced with the median.
                        Default: 5.0

required arguments:
  --in-folder str       path to input data, required. Default: None
  --out-folder str      path to output to, required. Default: None
  --runs str [str ...]  runs to use, required, range allowed. Default: None

DSSC

DSSC Offline Correction

Author: European XFEL Detector Group, Version: 1.0

Offline Calibration for the DSSC Detector

To invoke this notebook and display help use:

xfel-calibrate DSSC CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--sequences int [int ...]]
                      [--modules str [str ...]] --run int [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-id str]
                      [--path-template str] [--h5path str] [--h5path-idx str]
                      [--slow-data-pattern str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--mem-cells int] [--overwrite | --no-overwrite]
                      [--max-pulses int] [--bias-voltage int]
                      [--sequences-per-node int] [--chunk-size-idim int]
                      [--mask-noisy-asic float] [--mask-cold-asic float]
                      [--noisy-pix-threshold float] [--geo-file str]
                      [--dinstance str]
                      [--slow-data-aggregators int [int ...]]
                      [--slow-data-path str]
                      DSSC CORRECT

# DSSC Offline Correction #
,
Author: European XFEL Detector Group, Version: 1.0,
,
Offline Calibration for the DSSC Detector

positional arguments:
  DSSC              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        The ipcluster profile to use. Default: noDB
  --sequences int [int ...]
                        sequence files to evaluate.. Default: [-1]
  --modules str [str ...]
                        modules to correct, set to -1 for all, range allowed.
                        Default: [-1]
  --karabo-id str       karabo karabo_id. Default: SQS_DET_DSSC1M-1
  --karabo-da str [str ...]
                        a list of data aggregators names, Default [-1] for
                        selecting all data aggregators. Default: ['-1']
  --receiver-id str     inset for receiver devices. Default: {}CH0
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{:05d}.h5
  --h5path str          path in the HDF5 file to images. Default:
                        INSTRUMENT/{}/DET/{}:xtdf/image
  --h5path-idx str      path in the HDF5 file to images. Default:
                        /INDEX/{}/DET/{}:xtdf/image
  --slow-data-pattern str
                        Default: RAW-R{}-DA{}-S00000.h5
  --use-dir-creation-date
                        use the creation data of the input dir for database
                        queries. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8020#8025
  --cal-db-timeout int  in milli seconds. Default: 300000
  --mem-cells int       number of memory cells used, set to 0 to automatically
                        infer. Default: 0
  --overwrite           set to True if existing data should be overwritten.
                        Default: True
  --no-overwrite        Opposite of --overwrite (default: True)
  --max-pulses int      maximum number of pulses per train. Default: 800
  --bias-voltage int    detector bias voltage. Default: 100
  --sequences-per-node int
                        number of sequence files per cluster node if run as
                        slurm job, set to 0 to not run SLURM parallel.
                        Default: 1
  --chunk-size-idim int
                        chunking size of imaging dimension, adjust if user
                        software is sensitive to this.. Default: 1
  --mask-noisy-asic float
                        set to a value other than 0 and below 1 to mask entire
                        ADC if fraction of noisy pixels is above. Default:
                        0.25
  --mask-cold-asic float
                        mask cold ASICS if number of pixels with negligable
                        standard deviation is larger than this fraction.
                        Default: 0.25
  --noisy-pix-threshold float
                        threshold above which ap pixel is considered noisy..
                        Default: 1.0
  --geo-file str        detector geometry file. Default:
                        /gpfs/exfel/data/scratch/xcal/dssc_geo_june19.h5
  --dinstance str       Default: DSSC1M1
  --slow-data-aggregators int [int ...]
                        quadrant/aggregator. Default: [1, 2, 3, 4]
  --slow-data-path str  Default: SQS_NQS_DSSC/FPGA/PPT_Q

required arguments:
  --in-folder str       path to input data, required. Default: None
  --out-folder str      path to output to, required. Default: None
  --run int             runs to process, required. Default: None

DSSC Characterize Dark Images

Author: S. Hauf, Version: 0.1

The following code analyzes a set of dark images taken with the DSSC detector to deduce detector offsets and noise. Data for the detector is presented in one run and don’t acquire multiple gain stages.

The notebook explicitely does what pyDetLib provides in its offset calculation method for streaming data.

To invoke this notebook and display help use:

xfel-calibrate DSSC DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--sequences int [int ...]]
                      [--modules int [int ...]] --run int [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-id str]
                      [--path-template str] [--h5path str] [--h5path-idx str]
                      [--slow-data-pattern str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--local-output | --no-local-output]
                      [--db-output | --no-db-output] [--mem-cells int]
                      [--bias-voltage int] [--rawversion int]
                      [--thresholds-offset-sigma float]
                      [--thresholds-offset-hard int [int ...]]
                      [--thresholds-noise-sigma float]
                      [--thresholds-noise-hard float [float ...]]
                      [--offset-numpy-algorithm str] [--instrument str]
                      [--high-res-badpix-3d | --no-high-res-badpix-3d]
                      [--slow-data-aggregators int [int ...]]
                      [--slow-data-path str] [--operation-mode str]
                      DSSC DARK

# DSSC Characterize Dark Images #
,
Author: S. Hauf, Version: 0.1,
,
The following code analyzes a set of dark images taken with the DSSC detector
to deduce detector offsets and noise. Data for the detector is presented in
"one run and dont acquire multiple gain stages. ",
,
The notebook explicitely does what pyDetLib provides in its offset
calculation method for streaming data.

positional arguments:
  DSSC              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        The ipcluster profile to use. Default: noDB
  --sequences int [int ...]
                        sequence files to evaluate.. Default: [0]
  --modules int [int ...]
                        modules to run for. Default: [-1]
  --karabo-id str       karabo karabo_id. Default: SQS_DET_DSSC1M-1
  --karabo-da str [str ...]
                        a list of data aggregators names, Default [-1] for
                        selecting all data aggregators. Default: ['-1']
  --receiver-id str     inset for receiver devices. Default: {}CH0
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{:05d}.h5
  --h5path str          path in the HDF5 file to images. Default:
                        /INSTRUMENT/{}/DET/{}:xtdf/image
  --h5path-idx str      path in the HDF5 file to images. Default:
                        /INDEX/{}/DET/{}:xtdf/image
  --slow-data-pattern str
                        Default: RAW-R{}-DA{}-S00000.h5
  --use-dir-creation-date
                        use the dir creation date for determining the creation
                        time. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8020
  --cal-db-timeout int  timeout on caldb requests". Default: 3000000
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --mem-cells int       number of memory cells used, set to 0 to automatically
                        infer. Default: 0
  --bias-voltage int    detector bias voltage. Default: 100
  --rawversion int      RAW file format version. Default: 2
  --thresholds-offset-sigma float
                        thresholds in terms of n sigma noise for offset
                        deduced bad pixels. Default: 3.0
  --thresholds-offset-hard int [int ...]
                        thresholds in absolute ADU terms for offset deduced
                        bad pixels,. Default: [4, 125]
  --thresholds-noise-sigma float
                        thresholds in terms of n sigma noise for offset
                        deduced bad pixels. Default: 3.0
  --thresholds-noise-hard float [float ...]
                        thresholds in absolute ADU terms for offset deduced
                        bad pixels. Default: [0.001, 3]
  --offset-numpy-algorithm str
                        Default: mean
  --instrument str      the instrument. Default: SQS
  --high-res-badpix-3d  set this to True if you need high-resolution 3d bad
                        pixel plots. Runtime: ~ 1h. Default: False
  --no-high-res-badpix-3d
                        Opposite of --high-res-badpix-3d (default: False)
  --slow-data-aggregators int [int ...]
                        quadrant/aggregator. Default: [1, 2, 3, 4]
  --slow-data-path str  Default: SQS_NQS_DSSC/FPGA/PPT_Q
  --operation-mode str  Detector operation mode, optional. Default:

required arguments:
  --in-folder str       path to input data, required. Default: None
  --out-folder str      path to output to, required. Default: None
  --run int             run number in which data was recorded, required.
                        Default: None

EPIX100

ePix100 Data Correction

Author: European XFEL Detector Group, Version: 2.0

The following notebook provides data correction of images acquired with the ePix100 detector.

To invoke this notebook and display help use:

xfel-calibrate EPIX100 CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      [--sequences str [str ...]] [--sequences-per-node int]
                      --run int [--karabo-id str] [--karabo-da str]
                      [--db-module str] [--receiver-template str]
                      [--path-template str] [--instrument-source-template str]
                      [--chunk-size-idim int] [--limit-images int]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--creation-time str] [--bias-voltage int]
                      [--in-vacuum | --no-in-vacuum] [--integration-time int]
                      [--fix-temperature int] [--gain-photon-energy float]
                      [--photon-energy float]
                      [--pattern-classification | --no-pattern-classification]
                      [--relative-gain | --no-relative-gain]
                      [--absolute-gain | --no-absolute-gain]
                      [--common-mode | --no-common-mode] [--cm-min-frac float]
                      [--cm-noise-sigma float]
                      [--split-evt-primary-threshold float]
                      [--split-evt-secondary-threshold float]
                      [--split-evt-mip-threshold float]
                      EPIX100 CORRECT

# ePix100 Data Correction
,
Author: European XFEL Detector Group, Version: 2.0,
,
The following notebook provides data correction of images acquired with the
ePix100 detector.

positional arguments:
  EPIX100              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequences str [str ...]
                        sequences to correct, set to -1 for all, range
                        allowed. Default: [-1]
  --sequences-per-node int
                        number of sequence files per cluster node if run as
                        slurm job, set to 0 to not run SLURM parallel.
                        Default: 1
  --karabo-id str       karabo karabo_id. Default: MID_EXP_EPIX-1
  --karabo-da str       data aggregators. Default: EPIX01
  --db-module str       module id in the database. Default:
  --receiver-template str
                        detector receiver template for accessing raw data
                        files. Default: RECEIVER
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --instrument-source-template str
                        instrument detector data source in h5files. Default:
                        {}/DET/{}:daqOutput
  --chunk-size-idim int
                        H5 chunking size of output data. Default: 1
  --limit-images int    ONLY FOR TESTING. process only first N images, 0 -
                        process all.. Default: 0
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8015#8025
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --creation-time str   The timestamp to use with Calibration DBe. Required
                        Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
                        11:02:41. Default:
  --bias-voltage int    bias voltage. Default: 200
  --in-vacuum           detector operated in vacuum. Default: False
  --no-in-vacuum        Opposite of --in-vacuum (default: False)
  --integration-time int
                        Detector integration time, Default value -1 to use the
                        value from the slow data.. Default: -1
  --fix-temperature int
                        fixed temperature value in Kelvin, Default value -1 to
                        use the value from files.. Default: -1
  --gain-photon-energy float
                        Photon energy used for gain calibration. Default: 9.0
  --photon-energy float
                        Photon energy to calibrate in number of photons, 0 for
                        calibration in keV. Default: 0.0
  --pattern-classification
                        do clustering.. Default: True
  --no-pattern-classification
                        Opposite of --pattern-classification (default: True)
  --relative-gain       Apply relative gain correction.. Default: True
  --no-relative-gain    Opposite of --relative-gain (default: True)
  --absolute-gain       Apply absolute gain correction (implies relative
                        gain).. Default: True
  --no-absolute-gain    Opposite of --absolute-gain (default: True)
  --common-mode         Apply common mode correction.. Default: True
  --no-common-mode      Opposite of --common-mode (default: True)
  --cm-min-frac float   No CM correction is performed if after masking the
                        ratio of good pixels falls below this. Default: 0.25
  --cm-noise-sigma float
                        CM correction noise standard deviation. Default: 5.0
  --split-evt-primary-threshold float
                        primary threshold for split event correction. Default:
                        7.0
  --split-evt-secondary-threshold float
                        secondary threshold for split event correction.
                        Default: 5.0
  --split-evt-mip-threshold float
                        minimum ionizing particle threshold. Default: 1000.0

required arguments:
  --in-folder str       input folder, required. Default: None
  --out-folder str      output folder, required. Default: None
  --run int             which run to read data from, required. Default: None

ePix100 Dark Characterization

Author: European XFEL Detector Group, Version: 2.0

The following notebook provides dark image analysis and calibration constants of the ePix100 detector.

Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.

Noise and bad pixels maps are calculated independently for each of the 4 ASICs of ePix100, since their noise behaviour can be significantly different.

Common mode correction can be applied to increase sensitivity to noise related bad pixels. Common mode correction is achieved by subtracting the median of all pixels that are read out at the same time along a row/column. This is done in an iterative process, by which a new bad pixels map is calculated and used to mask data as the common mode values across the rows/columns is updated.

Resulting maps are saved as .h5 files for a later use and injected to calibration DB.

To invoke this notebook and display help use:

xfel-calibrate EPIX100 DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      [--sequence int] --run int [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-template str]
                      [--path-template str] [--instrument-source-template str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--db-output | --no-db-output]
                      [--local-output | --no-local-output]
                      [--bias-voltage int] [--in-vacuum | --no-in-vacuum]
                      [--fix-integration-time int] [--fix-temperature int]
                      [--temp-limits int] [--badpixel-threshold-sigma float]
                      [--CM-N-iterations int] [--min-trains int]
                      [--max-trains int] [--operation-mode str]
                      [--db-module str]
                      EPIX100 DARK

# ePix100 Dark Characterization
,
Author: European XFEL Detector Group, Version: 2.0,
,
The following notebook provides dark image analysis and calibration constants
of the ePix100 detector.,
,
Dark characterization evaluates offset and noise of the detector and gives
information about bad pixels.
,
Noise and bad pixels maps are calculated independently for each of the 4
ASICs of ePix100, since their noise behaviour can be significantly different.,
,
Common mode correction can be applied to increase sensitivity to noise
related bad pixels. Common mode correction is achieved by subtracting the
median of all pixels that are read out at the same time along a row/column.
This is done in an iterative process, by which a new bad pixels map is
calculated and used to mask data as the common mode values across the
rows/columns is updated.,
,
Resulting maps are saved as .h5 files for a later use and injected to
calibration DB.

positional arguments:
  EPIX100              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequence int        sequence file to use. Default: 0
  --karabo-id str       karabo karabo_id. Default: HED_IA1_EPX100-1
  --karabo-da str [str ...]
                        data aggregators. Default: ['EPIX01']
  --receiver-template str
                        detector receiver template for accessing raw data
                        files. Default: RECEIVER
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --instrument-source-template str
                        instrument detector data source in h5files. Default:
                        {}/DET/{}:daqOutput
  --use-dir-creation-date
                        Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8020
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --db-output           Output constants to the calibration database. Default:
                        False
  --no-db-output        Opposite of --db-output (default: False)
  --local-output        Output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --bias-voltage int    Bias voltage. Default: 200
  --in-vacuum           Detector operated in vacuum. Default: False
  --no-in-vacuum        Opposite of --in-vacuum (default: False)
  --fix-integration-time int
                        Integration time. Set to -1 to read from .h5 file.
                        Default: -1
  --fix-temperature int
                        Fixed temperature in Kelvin. Set to -1 to read from
                        .h5 file. Default: -1
  --temp-limits int     Limit for parameter Operational temperature. Default:
                        5
  --badpixel-threshold-sigma float
                        Bad pixels defined by values outside n times this std
                        from median. Default: 5. Default: 5.0
  --CM-N-iterations int
                        Number of iterations for common mode correction. Set
                        to 0 to skip it. Default: 2
  --min-trains int      Minimum number of trains that should be available to
                        process dark constants. Default 1.. Default: 1
  --max-trains int      Maximum number of trains to use for processing dark
                        constants. Set to 0 to use all available trains..
                        Default: 1000
  --operation-mode str  Detector operation mode, optional. Default:
  --db-module str       ID of module in calibration database, this parameter
                        is ignore in the notebook. TODO: remove from
                        calibration_configurations.. Default:

required arguments:
  --in-folder str       input folder, required. Default: None
  --out-folder str      output folder, required. Default: None
  --run int             which run to read data from, required. Default: None

EPIX10K

ePIX10K Data Correction

Authors: M. Karnevskiy, S. Hauf, Version 1.0

The following notebook provides Offset correction of images acquired with the ePix10K detector.

To invoke this notebook and display help use:

xfel-calibrate EPIX10K CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--sequences str [str ...]] --run
                      int [--karabo-id str] [--karabo-da str]
                      [--receiver-id str] [--path-template str] [--h5path str]
                      [--h5path-t str] [--h5path-cntrl str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--cpuCores int] [--chunk-size-idim int]
                      [--overwrite | --no-overwrite] [--limit-images int]
                      [--db-module str] [--sequences-per-node int]
                      [--bias-voltage int] [--in-vacuum | --no-in-vacuum]
                      [--fix-temperature float]
                      [--split-evt-primary-threshold float]
                      [--split-evt-secondary-threshold float]
                      [--split-evt-mip-threshold float]
                      EPIX10K CORRECT

# ePIX10K Data Correction #
,
Authors: M. Karnevskiy, S. Hauf, Version 1.0,
,
The following notebook provides Offset correction of images acquired with the
ePix10K detector.

positional arguments:
  EPIX10K              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        ipcluster profile to use. Default: noDB
  --sequences str [str ...]
                        sequences to correct, set to -1 for all, range
                        allowed. Default: [-1]
  --karabo-id str       karabo karabo_id. Default: HED_IA1_EPIX10K-1
  --karabo-da str       data aggregators. Default: EPIX03
  --receiver-id str     inset for receiver devices. Default: RECEIVER
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --h5path str          path in the HDF5 file to images. Default:
                        /INSTRUMENT/{}/DET/{}:daqOutput/data/image
  --h5path-t str        path to find temperature at. Default:
                        /INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp
  --h5path-cntrl str    path to control data. Default: /CONTROL/{}/DET
  --use-dir-creation-date
                        date constants injected before directory creation
                        time. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8015#8025
  --cal-db-timeout int  timeout on caldb requests. Default: 30000000
  --cpuCores int        Specifies the number of running cpu cores. Default: 4
  --chunk-size-idim int
                        H5 chunking size of output data. Default: 1
  --overwrite           overwrite output folder. Default: True
  --no-overwrite        Opposite of --overwrite (default: True)
  --limit-images int    process only first N images, 0 - process all. Default:
                        0
  --db-module str       module id in the database. Default: ePix10K_M43
  --sequences-per-node int
                        number of sequence files per cluster node if run as
                        slurm job, set to 0 to not run SLURM parallel.
                        Default: 1
  --bias-voltage int    bias voltage. Default: 200
  --in-vacuum           detector operated in vacuum. Default: False
  --no-in-vacuum        Opposite of --in-vacuum (default: False)
  --fix-temperature float
                        fix temperature to this value. Default: 290.0
  --split-evt-primary-threshold float
                        primary threshold for split event correction. Default:
                        7.0
  --split-evt-secondary-threshold float
                        secondary threshold for split event correction.
                        Default: 5.0
  --split-evt-mip-threshold float
                        minimum ionizing particle threshold. Default: 1000.0

required arguments:
  --in-folder str       input folder, required. Default: None
  --out-folder str      output folder, required. Default: None
  --run int             which run to read data from, required. Default: None

ePix10K Dark Characterization

Author: M. Karnevskiy, Version 1.0

The following notebook provides dark image analysis of the ePix10K detector.

Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Resulting maps are saved as .h5 files for a latter use and injected to the calibration DB.

To invoke this notebook and display help use:

xfel-calibrate EPIX10K DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--sequence int] --run int
                      [--karabo-id str] [--karabo-da str [str ...]]
                      [--receiver-id str] [--path-template str] [--h5path str]
                      [--h5path-t str] [--h5path-cntrl str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--db-output | --no-db-output]
                      [--local-output | --no-local-output] [--temp-limits int]
                      [--number-dark-frames int] [--db-module str]
                      [--bias-voltage int] [--in-vacuum | --no-in-vacuum]
                      [--fix-temperature float] [--operation-mode str]
                      EPIX10K DARK

# ePix10K Dark Characterization #
,
Author: M. Karnevskiy, Version 1.0,
,
The following notebook provides dark image analysis of the ePix10K detector.,
,
Dark characterization evaluates offset and noise of the detector and gives
information about bad pixels. Resulting maps are saved as .h5 files for a
latter use and injected to the calibration DB.

positional arguments:
  EPIX10K              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        ipcluster profile to use. Default: noDB
  --sequence int        sequence file to use. Default: 0
  --karabo-id str       karabo karabo_id. Default: HED_IA1_EPIX10K-1
  --karabo-da str [str ...]
                        data aggregators. Default: ['EPIX03']
  --receiver-id str     inset for receiver devices. Default: RECEIVER
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --h5path str          path in the HDF5 file to images. Default:
                        /INSTRUMENT/{}/DET/{}:daqOutput/data/image/pixels
  --h5path-t str        path to find temperature at. Default:
                        /INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp
  --h5path-cntrl str    path to control data. Default: /CONTROL/{}/DET
  --use-dir-creation-date
                        Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8020
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --db-output           Output constants to the calibration database. Default:
                        False
  --no-db-output        Opposite of --db-output (default: False)
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --temp-limits int     limit for parameter Operational temperature. Default:
                        5
  --number-dark-frames int
                        number of images to be used, if set to 0 all available
                        images are used. Default: 0
  --db-module str       detector karabo_id. Default: ePix10K_M43
  --bias-voltage int    bias voltage. Default: 200
  --in-vacuum           detector operated in vacuum. Default: False
  --no-in-vacuum        Opposite of --in-vacuum (default: False)
  --fix-temperature float
                        fix temperature to this value. Default: 290.0
  --operation-mode str  Detector operation mode, optional. Default:

required arguments:
  --in-folder str       input folder, required. Default: None
  --out-folder str      output folder, required. Default: None
  --run int             which run to read data from, required. Default: None

FASTCCD

FastCCD Data Correction

Authors: I. Klačková, S. Hauf, Version 1.0

The following notebook provides correction of images acquired with the FastCCD.

To invoke this notebook and display help use:

xfel-calibrate FASTCCD CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--sequences str [str ...]]
                      [--run int] [--karabo-da str] [--karabo-id str]
                      [--receiver-id str] [--path-template str] [--h5path str]
                      [--h5path-t str] [--h5path-cntrl str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--cpuCores int] [--operation-mode str]
                      [--split-evt-primary-threshold float]
                      [--split-evt-secondary-threshold float]
                      [--split-evt-mip-threshold float]
                      [--chunk-size-idim int] [--overwrite | --no-overwrite]
                      [--sequences-per-node int] [--limit-images int]
                      [--time-offset-days int]
                      [--photon-energy-gain-map float]
                      [--fix-temperature float]
                      [--flipped-between str [str ...]] [--temp-limits int]
                      [--commonModeAxis int]
                      [--only-offset | --no-only-offset]
                      [--cti-corr | --no-cti-corr]
                      [--relgain-corr | --no-relgain-corr]
                      [--common-mode-corr | --no-common-mode-corr]
                      [--correct-offset-drift | --no-correct-offset-drift]
                      [--do-pattern-classification | --no-do-pattern-classification]
                      FASTCCD CORRECT

# FastCCD Data Correction
,
Authors: I. Klačková, S. Hauf, Version 1.0,
,
The following notebook provides correction of images acquired with the
FastCCD.

positional arguments:
  FASTCCD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        ipcluster profile to use. Default: noDB
  --sequences str [str ...]
                        sequences to correct, set to -1 for all, range
                        allowed. Default: [-1]
  --run int             run number. Default: 277
  --karabo-da str       data aggregators. Default: DA05
  --karabo-id str       karabo prefix of PNCCD devices. Default:
                        SCS_CDIDET_FCCD2M
  --receiver-id str     inset for receiver devices. Default: FCCD
  --path-template str   path template in hdf5 file. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --h5path str          path in HDF5 file. Default:
                        /INSTRUMENT/{}/DAQ/{}:daqOutput/data/image
  --h5path-t str        temperature path in HDF5 file. Default:
                        /CONTROL/{}/CTRL/LSLAN/inputA/crdg/value
  --h5path-cntrl str    path to control data. Default: /RUN/{}/DET/FCCD
  --use-dir-creation-date
                        use dir creation data for calDB queries. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8015#8025
  --cal-db-timeout int  timeout on caldb requests. Default: 300000000
  --cpuCores int        Specifies the number of running cpu cores. Default: 16
  --operation-mode str  FS stands for frame-store and FF for full-frame
                        opeartion. Default: FF
  --split-evt-primary-threshold float
                        primary threshold for split event classification in
                        terms of n sigma noise. Default: 7.0
  --split-evt-secondary-threshold float
                        secondary threshold for split event classification in
                        terms of n sigma noise. Default: 4.0
  --split-evt-mip-threshold float
                        MIP threshold for event classification. Default:
                        1000.0
  --chunk-size-idim int
                        H5 chunking size of output data. Default: 1
  --overwrite           overwrite existing files. Default: True
  --no-overwrite        Opposite of --overwrite (default: True)
  --sequences-per-node int
                        sequences to correct per node. Default: 1
  --limit-images int    limit images per file. Default: 0
  --time-offset-days int
                        offset in days for calibration parameters. Default: 0
  --photon-energy-gain-map float
                        energy in keV. Default: 5.9
  --fix-temperature float
                        fix temperature to this value, set to 0 to use slow
                        control value. Default: 0.0
  --flipped-between str [str ...]
                        detector was flipped during this timespan. Default:
                        ['2019-02-01', '2019-04-02']
  --temp-limits int     limits within which temperature is considered the
                        same. Default: 5
  --commonModeAxis int  Axis along which common mode will be calculated (0:
                        along rows, 1: along columns). Default: 1
  --only-offset         Only apply offset correction. Default: False
  --no-only-offset      Opposite of --only-offset (default: False)
  --cti-corr            Apply CTI correction. Default: False
  --no-cti-corr         Opposite of --cti-corr (default: False)
  --relgain-corr        Apply relative gain correction. Default: True
  --no-relgain-corr     Opposite of --relgain-corr (default: True)
  --common-mode-corr    Apply commonMode correction. Default: False
  --no-common-mode-corr
                        Opposite of --common-mode-corr (default: False)
  --correct-offset-drift
                        correct for offset drifts. Default: False
  --no-correct-offset-drift
                        Opposite of --correct-offset-drift (default: False)
  --do-pattern-classification
                        classify split events. Default: True
  --no-do-pattern-classification
                        Opposite of --do-pattern-classification (default:
                        True)

required arguments:
  --in-folder str       input folder, required. Default: None
  --out-folder str      output folder, required. Default: None

FastCCD Dark Characterization

Author: I. Klačková, S. Hauf, K. Setoodehnia and M. Cascella

The following notebook provides dark image analysis of the FastCCD detector.

Dark characterization evaluates offset and noise of the FastCCD detector, corrects the noise for Common Mode (CM), and defines bad pixels relative to offset and CM corrected noise. Bad pixels are then excluded and CM corrected noise is recalculated excluding the bad pixels. Resulting offset and CM corrected noise maps, as well as the bad pixel map are sent to the calibration database.

To invoke this notebook and display help use:

xfel-calibrate FASTCCD DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--sequence int] --run int
                      [--karabo-da str [str ...]] [--karabo-id str]
                      [--receiver-id str] [--path-template str] [--h5path str]
                      [--h5path-t str] [--h5path-cntrl str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--db-output | --no-db-output]
                      [--local-output | --no-local-output]
                      [--number-dark-frames int] [--operation-mode str]
                      [--temp-limits int] [--bad-pixel-offset-sigma float]
                      [--bad-pixel-noise-sigma float] [--sigmaNoise float]
                      [--fix-temperature float] [--chunkSize int]
                      [--cpuCores int] [--commonModeAxis int]
                      [--run-parallel | --no-run-parallel]
                      [--ADU-to-electron-upper-hg float]
                      [--ADU-to-electron-lower-hg float]
                      [--ADU-to-electron-upper-mg float]
                      [--ADU-to-electron-lower-mg float]
                      [--ADU-to-electron-upper-lg float]
                      [--ADU-to-electron-lower-lg float]
                      FASTCCD DARK

# FastCCD Dark Characterization
,
Author: I. Klačková, S. Hauf, K. Setoodehnia and M. Cascella,
,
The following notebook provides dark image analysis of the FastCCD detector.,
,
Dark characterization evaluates offset and noise of the FastCCD detector,
corrects the noise for Common Mode (CM), and defines bad pixels relative to
offset and CM corrected noise. Bad pixels are then excluded and CM corrected
noise is recalculated excluding the bad pixels. Resulting offset and CM
corrected noise maps, as well as the bad pixel map are sent to the
calibration database.

positional arguments:
  FASTCCD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        ipcluster profile to use. Default: noDB
  --sequence int        sequence file to use. Default: 0
  --karabo-da str [str ...]
                        data aggregators. Default: ['DA05']
  --karabo-id str       karabo prefix of PNCCD devices. Default:
                        SCS_CDIDET_FCCD2M
  --receiver-id str     inset for receiver devices. Default: FCCD
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --h5path str          path to the data in the HDF5 file. Default:
                        /INSTRUMENT/{}/DAQ/{}:daqOutput/data/image/pixels
  --h5path-t str        path to find temperature. Default:
                        /CONTROL/{}/CTRL/LSLAN/inputA/crdg/value
  --h5path-cntrl str    path to find control data. Default: /RUN/{}/DET/FCCD
  --use-dir-creation-date
                        To be used to retrieve calibration constants later on
                        (for database time derivation). Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the calibration database interface to use. Default:
                        tcp://max-exfl016:8020
  --cal-db-timeout int  timeout on calibration database requests. Default:
                        300000
  --db-output           Output constants to the calibration database. Default:
                        False
  --no-db-output        Opposite of --db-output (default: False)
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --number-dark-frames int
                        number of images to be used, if set to 0 all available
                        images are used. Default: 0
  --operation-mode str  FS stands for frame-store and FF for full-frame
                        operation.. Default: FF
  --temp-limits int     to find calibration constants later on, the sensor
                        temperature is allowed to vary by 5 units. Default: 5
  --bad-pixel-offset-sigma float
                        Any pixel whose offset is beyond 5 standard
                        deviations, is a bad pixel. Default: 5.0
  --bad-pixel-noise-sigma float
                        Any pixel whose noise is beyond 5 standard deviations,
                        is a bad pixel. Default: 5.0
  --sigmaNoise float    Any pixel whose signal exceeds 'sigmaNoise'*noiseCM
                        (common mode corrected noise) will be masked. Default:
                        5.0
  --fix-temperature float
                        Fixed operation temperature in Kelvins. If set to 0,
                        mean value of the data file's temperature is. Default:
                        0.0
  --chunkSize int       Number of images to read per chunk. Default: 100
  --cpuCores int        Specifies the number of running cpu cores. Default: 40
  --commonModeAxis int  Axis along which common mode will be calculated (0:
                        along rows, 1: along columns). Default: 1
  --run-parallel        For parallel computation. Default: True
  --no-run-parallel     Opposite of --run-parallel (default: True)
  --ADU-to-electron-upper-hg float
                        for upper hemisphere and high gain. Default: 6.3
  --ADU-to-electron-lower-hg float
                        for lower hemisphere and high gain. Default: 6.4
  --ADU-to-electron-upper-mg float
                        for upper hemisphere and medium gain. Default: 23.4
  --ADU-to-electron-lower-mg float
                        for lower hemisphere and medium gain. Default: 23.4
  --ADU-to-electron-upper-lg float
                        for upper hemisphere and low gain. Default: 49.3
  --ADU-to-electron-lower-lg float
                        for lower hemisphere and low gain. Default: 47.3

required arguments:
  --in-folder str       input folder, required. Default: None
  --out-folder str      output folder, required. Default: None
  --run int             which run to read data from, required. Default: None

GENERIC

Constants from DB to HDF5

Version 0.1, Author: S. Hauf

Currently available instances are LPD1M1 and AGIPD1M1

To invoke this notebook and display help use:

xfel-calibrate GENERIC DB_TO_H5 --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --detector-instance str --out-file
                      str [--valid-at str] [--cal-db-interface str]
                      [--modules str [str ...]]
                      GENERIC DB_TO_H5

# Constants from DB to HDF5 #
,
Version 0.1, Author: S. Hauf,
,
Currently available instances are LPD1M1 and AGIPD1M1

positional arguments:
  GENERIC              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DB_TO_H5                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --valid-at str        ISO formatted date for which constants shoudl be
                        valid. Leave empty to get most current ones. Default:
  --cal-db-interface str
                        Default: tcp://max-exfl015:5005
  --modules str [str ...]
                        modules to get data from, in terms of numerical
                        quadrant indices, range allowed. Default: [-1]

required arguments:
  --detector-instance str
                        the detector instance to get constants for e.g.
                        LPD1M1, required. Default: None
  --out-file str        HDF5 file to output constants into, required. Default:
                        None

Statistical analysis of calibration factors

Author: Mikhail Karnevskiy, Version 0.2

Plot calibration constants retrieved from the cal. DB.

To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant.

Values shown in plots are saved in h5 files.

To invoke this notebook and display help use:

xfel-calibrate GENERIC STATS_FROM_DB --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--start-date str] [--end-date str]
                      [--dclass str] [--modules str [str ...]]
                      [--submodules str [str ...]] [--constants str [str ...]]
                      [--nconstants int] [--max-time int] [--nMemToShow int]
                      [--gain-setting int [int ...]]
                      [--bias-voltage int [int ...]]
                      [--temperature int [int ...]]
                      [--integration-time float [float ...]]
                      [--pixels-x int [int ...]] [--pixels-y int [int ...]]
                      [--in-vacuum int [int ...]]
                      [--memory-cells int [int ...]]
                      [--acquisition-rate float [float ...]]
                      [--parameter-names str [str ...]]
                      [--separate-plot str [str ...]]
                      [--x-labels str [str ...]] [--photon-energy float]
                      [--out-folder str] [--use-existing str]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--plot-range int] [--spShape int [int ...]]
                      [--sp-name str] [--gain-titles str [str ...]]
                      GENERIC STATS_FROM_DB

# Statistical analysis of calibration factors#
,
Author: Mikhail Karnevskiy, Version 0.2,
,
Plot calibration constants retrieved from the cal. DB.,
,
To be visualized, calibration constants are averaged per group of pixels.
Plots shows calibration constant over time for each constant.,
,
Values shown in plots are saved in h5 files.

positional arguments:
  GENERIC              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  STATS_FROM_DB                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --start-date str      date to start investigation interval from. Default:
                        2019-01-01
  --end-date str        date to end investigation interval at, can be "now".
                        Default: NOW
  --dclass str          Detector class. Default: jungfrau
  --modules str [str ...]
                        detector entry in the DB to investigate. Default:
                        ['Jungfrau_M035']
  --submodules str [str ...]
                        module index of a modular detector (1 for Q1M1 of
                        AGIPD), range allowed. Default: [2]
  --constants str [str ...]
                        constants to plot. Default: ['RelativeGain']
  --nconstants int      Number of time stamps to plot. If not 0, overcome
                        start_date.. Default: 20
  --max-time int        max time margin in minutes to match bad pixels.
                        Default: 15
  --nMemToShow int      Number of memory cells to be shown in plots. Default:
                        16
  --gain-setting int [int ...]
                        gain stages. Default: [0, 1, 2]
  --bias-voltage int [int ...]
                        Bias voltage. Default: [90, 180]
  --temperature int [int ...]
                        Operation temperature. Default: [291]
  --integration-time float [float ...]
                        Integration time. Default: [4.96, 10, 50, 250]
  --pixels-x int [int ...]
                        number of pixels along X axis. Default: [1024]
  --pixels-y int [int ...]
                        number of pixels along Y axis. Default: [512]
  --in-vacuum int [int ...]
                        0 if detector is operated in room pressure. Default:
                        [0]
  --memory-cells int [int ...]
                        number of memory cells. Default: [1, 16]
  --acquisition-rate float [float ...]
                        aquisition rate. Default: [1.1]
  --parameter-names str [str ...]
                        names of parameters. Default: ['bias_voltage',
                        'integration_time', 'pixels_x', 'pixels_y',
                        'temperature', 'memory_cells']
  --separate-plot str [str ...]
                        Plot on separate plots. Default: ['gain_setting',
                        'memory_cells', 'integration_time']
  --x-labels str [str ...]
                        parameters to be shown on X axis: Acquisition rate,
                        Memory cells, Sensor Temperature, Integration Time.
                        Default: ['Sensor Temperature', 'Integration Time']
  --photon-energy float
                        Photon energy of the beam. Default: 9.2
  --out-folder str      output folder. Default:
                        /gpfs/exfel/data/scratch/karnem/test_bla4/
  --use-existing str    If not empty, constants stored in given folder will be
                        used. Default:
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8016
  --cal-db-timeout int  timeout on caldb requests",. Default: 180000
  --plot-range int      range for plotting in units of median absolute
                        deviations. Default: 3
  --spShape int [int ...]
                        Shape of superpixel. Default: [256, 64]
  --sp-name str         name of superpixel. Default: ASIC IDs
  --gain-titles str [str ...]
                        Title inset related to gain. Default: ['High gain',
                        'Medium gain', 'Low gain']

Statistical analysis of calibration factors

Author: Mikhail Karnevskiy, Version 0.2

Plot calibration constants retrieved from the cal. DB.

To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant.

Values shown in plots are saved in h5 files.

To invoke this notebook and display help use:

xfel-calibrate GENERIC STATS_FROM_DB2 --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--start-date str] [--end-date str]
                      [--dclass str] [--modules str [str ...]]
                      [--submodules str [str ...]] [--constants str [str ...]]
                      [--nconstants int] [--max-time int] [--nMemToShow int]
                      [--gain-setting int [int ...]]
                      [--bias-voltage int [int ...]]
                      [--temperature int [int ...]]
                      [--integration-time float [float ...]]
                      [--pixels-x int [int ...]] [--pixels-y int [int ...]]
                      [--in-vacuum int [int ...]]
                      [--memory-cells int [int ...]]
                      [--acquisition-rate float [float ...]]
                      [--parameter-names str [str ...]]
                      [--separate-plot str [str ...]]
                      [--x-labels str [str ...]] [--photon-energy float]
                      [--out-folder str] [--use-existing str]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--plot-range int] [--spShape int [int ...]]
                      [--sp-name str] [--gain-titles str [str ...]]
                      GENERIC STATS_FROM_DB2

# Statistical analysis of calibration factors#
,
Author: Mikhail Karnevskiy, Version 0.2,
,
Plot calibration constants retrieved from the cal. DB.,
,
To be visualized, calibration constants are averaged per group of pixels.
Plots shows calibration constant over time for each constant.,
,
Values shown in plots are saved in h5 files.

positional arguments:
  GENERIC              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  STATS_FROM_DB2                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --start-date str      date to start investigation interval from. Default:
                        2019-01-01
  --end-date str        date to end investigation interval at, can be "now".
                        Default: NOW
  --dclass str          Detector class. Default: jungfrau
  --modules str [str ...]
                        detector entry in the DB to investigate. Default:
                        ['Jungfrau_M035']
  --submodules str [str ...]
                        module index of a modular detector (1 for Q1M1 of
                        AGIPD), range allowed. Default: [2]
  --constants str [str ...]
                        constants to plot. Default: ['RelativeGain']
  --nconstants int      Number of time stamps to plot. If not 0, overcome
                        start_date.. Default: 20
  --max-time int        max time margin in minutes to match bad pixels.
                        Default: 15
  --nMemToShow int      Number of memory cells to be shown in plots. Default:
                        16
  --gain-setting int [int ...]
                        gain stages. Default: [0, 1, 2]
  --bias-voltage int [int ...]
                        Bias voltage. Default: [90, 180]
  --temperature int [int ...]
                        Operation temperature. Default: [291]
  --integration-time float [float ...]
                        Integration time. Default: [4.96, 10, 50, 250]
  --pixels-x int [int ...]
                        number of pixels along X axis. Default: [1024]
  --pixels-y int [int ...]
                        number of pixels along Y axis. Default: [512]
  --in-vacuum int [int ...]
                        0 if detector is operated in room pressure. Default:
                        [0]
  --memory-cells int [int ...]
                        number of memory cells. Default: [1, 16]
  --acquisition-rate float [float ...]
                        aquisition rate. Default: [1.1]
  --parameter-names str [str ...]
                        names of parameters. Default: ['bias_voltage',
                        'integration_time', 'pixels_x', 'pixels_y',
                        'temperature', 'memory_cells']
  --separate-plot str [str ...]
                        Plot on separate plots. Default: ['gain_setting',
                        'memory_cells', 'integration_time']
  --x-labels str [str ...]
                        parameters to be shown on X axis: Acquisition rate,
                        Memory cells, Sensor Temperature, Integration Time.
                        Default: ['Sensor Temperature', 'Integration Time']
  --photon-energy float
                        Photon energy of the beam. Default: 9.2
  --out-folder str      output folder. Default:
                        /gpfs/exfel/data/scratch/karnem/test_bla4/
  --use-existing str    If not empty, constants stored in given folder will be
                        used. Default:
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8016
  --cal-db-timeout int  timeout on caldb requests",. Default: 180000
  --plot-range int      range for plotting in units of median absolute
                        deviations. Default: 3
  --spShape int [int ...]
                        Shape of superpixel. Default: [256, 64]
  --sp-name str         name of superpixel. Default: ASIC IDs
  --gain-titles str [str ...]
                        Title inset related to gain. Default: ['High gain',
                        'Medium gain', 'Low gain']

GOTTHARD2

Gotthard2 Offline Correction

Author: European XFEL Detector Group, Version: 1.0

Offline Calibration for the Gothard2 Detector

To invoke this notebook and display help use:

xfel-calibrate GOTTHARD2 CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      --run int [--sequences str [str ...]]
                      [--sequences-per-node int] [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-template str]
                      [--control-template str]
                      [--instrument-source-template str]
                      [--ctrl-source-template str] [--karabo-id-control str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--overwrite-creation-time str] [--constants-file str]
                      [--offset-correction | --no-offset-correction]
                      [--gain-correction | --no-gain-correction]
                      [--bias-voltage int] [--exposure-time float]
                      [--exposure-period float] [--acquisition-rate float]
                      [--single-photon int] [--skip-plots | --no-skip-plots]
                      [--pulse-idx-preview int]
                      GOTTHARD2 CORRECT

# Gotthard2 Offline Correction #
,
Author: European XFEL Detector Group, Version: 1.0,
,
Offline Calibration for the Gothard2 Detector

positional arguments:
  GOTTHARD2              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequences str [str ...]
                        sequences to correct, set to [-1] for all, range
                        allowed. Default: [-1]
  --sequences-per-node int
                        number of sequence files per node if notebook executed
                        through xfel-calibrate, set to 0 to not run SLURM
                        parallel. Default: 1
  --karabo-id str       karabo prefix of Gotthard-II devices. Default:
                        FXE_XAD_G2XES
  --karabo-da str [str ...]
                        data aggregators. Default: ['GH201']
  --receiver-template str
                        receiver template used to read INSTRUMENT keys..
                        Default: RECEIVER
  --control-template str
                        control template used to read CONTROL keys.. Default:
                        CONTROL
  --instrument-source-template str
                        template for source name (filled with karabo_id &
                        receiver_id). e.g.
                        'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
                        {}/DET/{}:daqOutput
  --ctrl-source-template str
                        template for control source name (filled with
                        karabo_id_control). Default: {}/DET/{}
  --karabo-id-control str
                        Control karabo ID. Set to empty string to use the
                        karabo-id. Default:
  --use-dir-creation-date
                        use the creation data of the input dir for database
                        queries.. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use.. Default: tcp://max-
                        exfl016:8016#8025
  --cal-db-timeout int  timeout on caldb requests.. Default: 180000
  --overwrite-creation-time str
                        To overwrite the measured creation_time. Required
                        Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28
                        13:00:00.00". Default:
  --constants-file str  Use constants in given constant file path. /gpfs/exfel
                        /data/scratch/ahmedk/dont_remove/gotthard2/constants/c
                        alibration_constants_GH2.h5. Default:
  --offset-correction   apply offset correction. This can be disabled to only
                        apply LUT or apply LUT and gain correction for non-
                        linear differential results.. Default: True
  --no-offset-correction
                        Opposite of --offset-correction (default: True)
  --gain-correction     apply gain correction.. Default: True
  --no-gain-correction  Opposite of --gain-correction (default: True)
  --bias-voltage int    Detector bias voltage, set to -1 to use value in raw
                        file.. Default: -1
  --exposure-time float
                        Detector exposure time, set to -1 to use value in raw
                        file.. Default: -1.0
  --exposure-period float
                        Detector exposure period, set to -1 to use value in
                        raw file.. Default: -1.0
  --acquisition-rate float
                        Detector acquisition rate (1.1/4.5), set to -1 to use
                        value in raw file.. Default: -1.0
  --single-photon int   Detector single photon mode (High/Low CDS), set to -1
                        to use value in raw file.. Default: -1
  --skip-plots          exit after writing corrected files. Default: False
  --no-skip-plots       Opposite of --skip-plots (default: False)
  --pulse-idx-preview int
                        pulse index to preview. The following even/odd pulse
                        index is used for preview. TODO: update to pulseId
                        preview.. Default: 3

required arguments:
  --in-folder str       the folder to read data from, required. Default: None
  --out-folder str      the folder to output to, required. Default: None
  --run int             run to process, required. Default: None

Gotthard2 Dark Image Characterization

Author: European XFEL Detector Group, Version: 1.0

The following is a processing for offset, noise, and Badpixels maps using dark images taken with Gotthard2 detector. All constants are evaluated per strip, per pulse, and per memory cell. The maps are calculated for each gain stage that is acquired in 3 separate runs.

The three maps (calibration constants) can be injected to the database and stored locally.

To invoke this notebook and display help use:

xfel-calibrate GOTTHARD2 DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      --run-high int --run-med int --run-low int
                      [--karabo-id str] [--karabo-da str [str ...]]
                      [--receiver-template str] [--control-template str]
                      [--instrument-source-template str]
                      [--ctrl-source-template str] [--karabo-id-control str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--overwrite-creation-time str]
                      [--db-output | --no-db-output]
                      [--local-output | --no-local-output]
                      [--bias-voltage int] [--exposure-time float]
                      [--exposure-period float] [--acquisition-rate float]
                      [--single-photon int] [--min-trains int]
                      [--max-trains int] [--badpixel-threshold-sigma float]
                      [--operation-mode str]
                      GOTTHARD2 DARK

# Gotthard2 Dark Image Characterization #
,
Author: European XFEL Detector Group, Version: 1.0,
,
The following is a processing for offset, noise, and Badpixels maps using
dark images taken with Gotthard2 detector.,
All constants are evaluated per strip, per pulse, and per memory cell. The
maps are calculated for each gain stage that is acquired in 3 separate runs.,
,
The three maps (calibration constants) can be injected to the database and
stored locally.

positional arguments:
  GOTTHARD2              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --karabo-id str       karabo prefix of Gotthard-II devices. Default:
                        FXE_XAD_G2XES
  --karabo-da str [str ...]
                        data aggregators. Default: ['GH201']
  --receiver-template str
                        receiver template used to read INSTRUMENT keys..
                        Default: RECEIVER
  --control-template str
                        control template used to read CONTROL keys.. Default:
                        CONTROL
  --instrument-source-template str
                        template for source name (filled with karabo_id &
                        receiver_id). e.g.
                        'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput' noqa. Default:
                        {}/DET/{}:daqOutput
  --ctrl-source-template str
                        template for control source name (filled with
                        karabo_id_control). Default: {}/DET/{}
  --karabo-id-control str
                        Control karabo ID. Set to empty string to use the
                        karabo-id. Default:
  --use-dir-creation-date
                        Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8020
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --overwrite-creation-time str
                        To overwrite the measured creation_time. Required
                        Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28
                        13:00:00.00". Default:
  --db-output           Output constants to the calibration database. Default:
                        False
  --no-db-output        Opposite of --db-output (default: False)
  --local-output        Output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --bias-voltage int    Detector bias voltage, set to -1 to use value in raw
                        file.. Default: -1
  --exposure-time float
                        Detector exposure time, set to -1 to use value in raw
                        file.. Default: -1.0
  --exposure-period float
                        Detector exposure period, set to -1 to use value in
                        raw file.. Default: -1.0
  --acquisition-rate float
                        Detector acquisition rate (1.1/4.5), set to -1 to use
                        value in raw file.. Default: -1.0
  --single-photon int   Detector single photon mode (High/Low CDS), set to -1
                        to use value in raw file.. Default: -1
  --min-trains int      Minimum number of trains that should be available to
                        process dark constants. Default 1.. Default: 1
  --max-trains int      Maximum number of trains to use for processing dark
                        constants. Set to 0 to use all available trains..
                        Default: 1000
  --badpixel-threshold-sigma float
                        bad pixels defined by values outside n times this std
                        from median. Default: 5.0
  --operation-mode str  Detector dark run acquiring operation mode, optional.
                        Default:

required arguments:
  --in-folder str       the folder to read data from, required. Default: None
  --out-folder str      the folder to output to, required. Default: None
  --run-high int        run number for G0 dark run, required. Default: None
  --run-med int         run number for G1 dark run, required. Default: None
  --run-low int         run number for G2 dark run, required. Default: None

JUNGFRAU

Jungfrau Offline Correction

Author: European XFEL Detector Group, Version: 2.0

Offline Calibration for the Jungfrau Detector

To invoke this notebook and display help use:

xfel-calibrate JUNGFRAU CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      --run int [--sequences str [str ...]]
                      [--sequences-per-node int] [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-template str]
                      [--instrument-source-template str]
                      [--ctrl-source-template str] [--karabo-id-control str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--relative-gain | --no-relative-gain]
                      [--strixel-sensor | --no-strixel-sensor]
                      [--strixel-double-norm float] [--limit-trains int]
                      [--chunks-ids int] [--chunks-data int]
                      [--manual-slow-data | --no-manual-slow-data]
                      [--integration-time float] [--gain-setting int]
                      [--gain-mode int] [--mem-cells int] [--bias-voltage int]
                      [--skip-plots | --no-skip-plots] [--plot-trains int]
                      [--cell-id-preview int]
                      [--roi-definitions int [int ...]]
                      JUNGFRAU CORRECT

# Jungfrau Offline Correction #
,
Author: European XFEL Detector Group, Version: 2.0,
,
Offline Calibration for the Jungfrau Detector

positional arguments:
  JUNGFRAU              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequences str [str ...]
                        sequences to correct, set to [-1] for all, range
                        allowed. Default: [-1]
  --sequences-per-node int
                        number of sequence files per cluster node if run as
                        slurm job, set to 0 to not run SLURM parallel.
                        Default: 1
  --karabo-id str       karabo prefix of Jungfrau devices. Default:
                        SPB_IRDA_JF4M
  --karabo-da str [str ...]
                        data aggregators. Default: ['JNGFR01', 'JNGFR02',
                        'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07',
                        'JNGFR08']
  --receiver-template str
                        Detector receiver template for accessing raw data
                        files. e.g. "JNGFR{:02d}". Default: JNGFR{:02d}
  --instrument-source-template str
                        template for source name (filled with karabo_id &
                        receiver_id). e.g.
                        'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
                        {}/DET/{}:daqOutput
  --ctrl-source-template str
                        template for control source name (filled with
                        karabo_id_control). Default: {}/DET/CONTROL
  --karabo-id-control str
                        if control is on a different ID, set to empty string
                        if it is the same a karabo-id. Default:
  --use-dir-creation-date
                        use the creation data of the input dir for database
                        queries. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8017#8025
  --cal-db-timeout int  timeout on caldb requests. Default: 180000
  --relative-gain       do relative gain correction.. Default: True
  --no-relative-gain    Opposite of --relative-gain (default: True)
  --strixel-sensor      reordering for strixel detector layout.. Default:
                        False
  --no-strixel-sensor   Opposite of --strixel-sensor (default: False)
  --strixel-double-norm float
                        normalization to use for double-size pixels, only
                        applied for strixel sensors.. Default: 2.0
  --limit-trains int    ONLY FOR TESTING. process only first N trains, Use 0
                        to process all.. Default: 0
  --chunks-ids int      HDF chunk size for memoryCell and frameNumber..
                        Default: 32
  --chunks-data int     HDF chunk size for pixel data in number of frames..
                        Default: 1
  --manual-slow-data    if true, use manually entered bias_voltage,
                        integration_time, gain_setting, and gain_mode values.
                        Default: False
  --no-manual-slow-data
                        Opposite of --manual-slow-data (default: False)
  --integration-time float
                        integration time in us, will be overwritten by value
                        in file. Default: 4.96
  --gain-setting int    0 for dynamic gain, 1 for dynamic HG0, will be
                        overwritten by value in file. Default: 0
  --gain-mode int       0 for runs with dynamic gain setting, 1 for fixgain.
                        It will be overwritten by value in file, if
                        manual_slow_data is set to True.. Default: 0
  --mem-cells int       Set mem_cells to -1 to automatically use the value
                        stored in RAW data.. Default: -1
  --bias-voltage int    will be overwritten by value in file. Default: 180
  --skip-plots          exit after writing corrected files. Default: False
  --no-skip-plots       Opposite of --skip-plots (default: False)
  --plot-trains int     Number of trains to plot for RAW and CORRECTED plots.
                        Set to -1 to automatically plot all trains.. Default:
                        500
  --cell-id-preview int
                        cell Id used for preview in single-shot plots.
                        Default: 15
  --roi-definitions int [int ...]
                        List with groups of 6 values defining ROIs, e.g. [3,
                        120, 180, 200, 550, -2] for module 3 (JNGFR03), slice
                        120:180, 200:550, average along axis -2 (slow scan, or
                        -1 for fast scan). Default: [-1]

required arguments:
  --in-folder str       the folder to read data from, required. Default: None
  --out-folder str      the folder to output to, required. Default: None
  --run int             run to process, required. Default: None

Jungfrau Dark Image Characterization

Author: European XFEL Detector Group, Version: 2.0

Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad pixel maps

To invoke this notebook and display help use:

xfel-calibrate JUNGFRAU DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      --run-high int --run-med int --run-low int
                      [--karabo-da str [str ...]] [--karabo-id str]
                      [--karabo-id-control str] [--receiver-template str]
                      [--instrument-source-template str]
                      [--ctrl-source-template str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--local-output | --no-local-output]
                      [--db-output | --no-db-output]
                      [--badpixel-threshold-sigma float]
                      [--offset-abs-threshold-low int [int ...]]
                      [--offset-abs-threshold-high int [int ...]]
                      [--max-trains int] [--min-trains int]
                      [--manual-slow-data | --no-manual-slow-data]
                      [--time-limits float] [--integration-time int]
                      [--gain-setting int] [--gain-mode int]
                      [--bias-voltage int] [--memory-cells int]
                      [--detailed-report | --no-detailed-report]
                      [--operation-mode str]
                      JUNGFRAU DARK

# Jungfrau Dark Image Characterization #
,
Author: European XFEL Detector Group, Version: 2.0,
,
Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad
pixel maps

positional arguments:
  JUNGFRAU              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --karabo-da str [str ...]
                        list of data aggregators, which corresponds to
                        different JF modules. Default: ['JNGFR01', 'JNGFR02',
                        'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07',
                        'JNGFR08']
  --karabo-id str       karabo_id (detector identifier) prefix of Jungfrau
                        detector to process.. Default: SPB_IRDA_JF4M
  --karabo-id-control str
                        if control is on a different ID, set to empty string
                        if it is the same a karabo-id. Default:
  --receiver-template str
                        inset for receiver devices. Default: JNGFR{:02}
  --instrument-source-template str
                        template for instrument source name (filled with
                        karabo_id & receiver_id). e.g.
                        'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
                        {}/DET/{}:daqOutput
  --ctrl-source-template str
                        template for control source name (filled with
                        karabo_id_control). Default: {}/DET/CONTROL
  --use-dir-creation-date
                        use dir creation date. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        calibrate db interface to connect to. Default:
                        tcp://max-exfl016:8016#8045
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --badpixel-threshold-sigma float
                        bad pixels defined by values outside n times this std
                        from median. Default: 5.0
  --offset-abs-threshold-low int [int ...]
                        absolute bad pixel threshold in terms of offset, lower
                        values. Default: [1000, 10000, 10000]
  --offset-abs-threshold-high int [int ...]
                        absolute bad pixel threshold in terms of offset, upper
                        values. Default: [8000, 15000, 15000]
  --max-trains int      Maximum trains to process darks. Set to 0 to process
                        all available train images.. Default: 0
  --min-trains int      Minimum number of trains that should be available to
                        process dark constants. Default 1.. Default: 1
  --manual-slow-data    if true, use manually entered bias_voltage and
                        integration_time values. Default: False
  --no-manual-slow-data
                        Opposite of --manual-slow-data (default: False)
  --time-limits float   to find calibration constants later on, the
                        integration time is allowed to vary by 0.5 us.
                        Default: 0.025
  --integration-time int
                        integration time in us, will be overwritten by value
                        in file. Default: 1000
  --gain-setting int    0 for dynamic, forceswitchg1, forceswitchg2, 1 for
                        dynamichg0, fixgain1, fixgain2. Will be overwritten by
                        value in file. Default: 0
  --gain-mode int       1 if medium and low runs are fixgain1 and fixgain2,
                        otherwise 0. It will be overwritten by value in file,
                        if manual_slow_data. Default: 0
  --bias-voltage int    sensor bias voltage in V, will be overwritten by value
                        in file. Default: 90
  --memory-cells int    number of memory cells. Default: 16
  --detailed-report     Default: False
  --no-detailed-report  Opposite of --detailed-report (default: False)
  --operation-mode str  Detector operation mode, optional. Default:
                        ADAPTIVE_GAIN

required arguments:
  --in-folder str       folder under which runs are located, required.
                        Default: None
  --out-folder str      path to place reports at, required. Default: None
  --run-high int        run number for G0 dark run, required. Default: None
  --run-med int         run number for G1 dark run, required. Default: None
  --run-low int         run number for G2 dark run, required. Default: None

LPD

LPD Offline Correction

Author: European XFEL Data Analysis Group

To invoke this notebook and display help use:

xfel-calibrate LPD CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      [--sequences int [int ...]] [--modules int [int ...]]
                      [--karabo-da str [str ...]] --run int [--karabo-id str]
                      [--input-source str] [--output-source str]
                      [--creation-time str] [--cal-db-interface str]
                      [--cal-db-timeout int] [--cal-db-root str]
                      [--mem-cells int] [--bias-voltage float]
                      [--capacitor str] [--photon-energy float]
                      [--category int] [--offset-corr | --no-offset-corr]
                      [--rel-gain | --no-rel-gain] [--ff-map | --no-ff-map]
                      [--gain-amp-map | --no-gain-amp-map]
                      [--overwrite | --no-overwrite] [--chunks-data int]
                      [--chunks-ids int] [--create-virtual-cxi-in str]
                      [--sequences-per-node int] [--max-nodes int]
                      [--num-workers int] [--num-threads-per-worker int]
                      LPD CORRECT

# LPD Offline Correction #
Author: European XFEL Data Analysis Group

positional arguments:
  LPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequences int [int ...]
                        Sequences to correct, use [-1] for all. Default: [-1]
  --modules int [int ...]
                        Modules indices to correct, use [-1] for all, only
                        used when karabo_da is empty. Default: [-1]
  --karabo-da str [str ...]
                        Data aggregators names to correct, use [''] for all.
                        Default: ['']
  --karabo-id str       Karabo domain for detector.. Default: FXE_DET_LPD1M-1
  --input-source str    Input fast data source.. Default:
                        {karabo_id}/DET/{module_index}CH0:xtdf
  --output-source str   Output fast data source, empty to use same as input..
                        Default:
  --creation-time str   The timestamp to use with Calibration DB. Required
                        Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
                        11:02:41. Default:
  --cal-db-interface str
                        Not needed, compatibility with current webservice..
                        Default:
  --cal-db-timeout int  Not needed, compatbility with current webservice..
                        Default: 0
  --cal-db-root str     Default: /gpfs/exfel/d/cal/caldb_store
  --mem-cells int       Memory cells, LPD constants are always taken with 512
                        cells.. Default: 512
  --bias-voltage float  Detector bias voltage.. Default: 250.0
  --capacitor str       Capacitor setting: 5pF or 50pF. Default: 5pF
  --photon-energy float
                        Photon energy in keV.. Default: 9.2
  --category int        Whom to blame.. Default: 0
  --offset-corr         Offset correction.. Default: True
  --no-offset-corr      Opposite of --offset-corr (default: True)
  --rel-gain            Gain correction based on RelativeGain constant..
                        Default: True
  --no-rel-gain         Opposite of --rel-gain (default: True)
  --ff-map              Gain correction based on FFMap constant.. Default:
                        True
  --no-ff-map           Opposite of --ff-map (default: True)
  --gain-amp-map        Gain correction based on GainAmpMap constant..
                        Default: True
  --no-gain-amp-map     Opposite of --gain-amp-map (default: True)
  --overwrite           set to True if existing data should be overwritten.
                        Default: True
  --no-overwrite        Opposite of --overwrite (default: True)
  --chunks-data int     HDF chunk size for pixel data in number of frames..
                        Default: 1
  --chunks-ids int      HDF chunk size for cellId and pulseId datasets..
                        Default: 32
  --create-virtual-cxi-in str
                        Folder to create virtual CXI files in (for each
                        sequence).. Default:
  --sequences-per-node int
                        Sequence files to process per node. Default: 1
  --max-nodes int       Maximum number of SLURM jobs to split correction work
                        into. Default: 8
  --num-workers int     Worker processes per node, 8 is safe on 768G nodes but
                        won't work on 512G.. Default: 8
  --num-threads-per-worker int
                        Number of threads per worker.. Default: 32

required arguments:
  --in-folder str       the folder to read data from, required. Default: None
  --out-folder str      the folder to output to, required. Default: None
  --run int             run to process, required. Default: None

LPD Offset, Noise and Dead Pixels Characterization

Author: M. Karnevskiy, S. Hauf

This notebook performs re-characterize of dark images to derive offset, noise and bad-pixel maps. All three types of constants are evaluated per-pixel and per-memory cell.

The notebook will correctly handle veto settings, but note that if you veto cells you will not be able to use these offsets for runs with different veto settings - vetoed cells will have zero offset.

The evaluated calibration constants are stored locally and injected in the calibration data base.

To invoke this notebook and display help use:

xfel-calibrate LPD DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--sequence int]
                      [--modules str [str ...]] --run-high int --run-med int
                      --run-low int [--karabo-id str]
                      [--karabo-da str [str ...]] [--receiver-id str]
                      [--path-template str] [--h5path str] [--h5path-idx str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--local-output | --no-local-output]
                      [--db-output | --no-db-output] [--capacitor-setting int]
                      [--mem-cells int] [--bias-voltage int]
                      [--thresholds-offset-sigma float]
                      [--thresholds-offset-hard int [int ...]]
                      [--thresholds-noise-sigma float]
                      [--thresholds-noise-hard int [int ...]]
                      [--skip-first-ntrains int] [--instrument str]
                      [--ntrains int]
                      [--high-res-badpix-3d | --no-high-res-badpix-3d]
                      [--test-for-normality | --no-test-for-normality]
                      [--operation-mode str]
                      LPD DARK

# LPD Offset, Noise and Dead Pixels Characterization #
,
Author: M. Karnevskiy, S. Hauf,
,
This notebook performs re-characterize of dark images to derive offset, noise
and bad-pixel maps. All three types of constants are evaluated per-pixel and
per-memory cell.,
,
The notebook will correctly handle veto settings, but note that if you veto
cells you will not be able to use these offsets for runs with different veto
settings - vetoed cells will have zero offset.,
,
The evaluated calibration constants are stored locally and injected in the
calibration data base.,
,

positional arguments:
  LPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        The ipcluster profile to use. Default: noDB
  --sequence int        sequence files to evaluate. Default: 0
  --modules str [str ...]
                        list of modules to evaluate, RANGE ALLOWED. Default:
                        [-1]
  --karabo-id str       karabo karabo_id. Default: FXE_DET_LPD1M-1
  --karabo-da str [str ...]
                        a list of data aggregators names, Default [-1] for
                        selecting all data aggregators. Default: ['-1']
  --receiver-id str     inset for receiver devices. Default: {}CH0
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{:05d}.h5
  --h5path str          path in the HDF5 file to images. Default:
                        /INSTRUMENT/{}/DET/{}:xtdf/image
  --h5path-idx str      path in the HDF5 file to images. Default:
                        /INDEX/{}/DET/{}:xtdf/image
  --use-dir-creation-date
                        use the creation date of the directory for database
                        time derivation. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8015#8025
  --cal-db-timeout int  timeout on caldb requests". Default: 300000
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --capacitor-setting int
                        capacitor_setting for which data was taken. Default: 5
  --mem-cells int       number of memory cells used. Default: 512
  --bias-voltage int    detector bias voltage. Default: 250
  --thresholds-offset-sigma float
                        bad pixel relative threshold in terms of n sigma
                        offset. Default: 3.0
  --thresholds-offset-hard int [int ...]
                        bad pixel hard threshold. Default: [400, 1500]
  --thresholds-noise-sigma float
                        bad pixel relative threshold in terms of n sigma
                        noise. Default: 7.0
  --thresholds-noise-hard int [int ...]
                        bad pixel hard threshold. Default: [1, 35]
  --skip-first-ntrains int
                        Number of first trains to skip. Default: 10
  --instrument str      instrument name. Default: FXE
  --ntrains int         number of trains to use. Default: 100
  --high-res-badpix-3d  plot bad-pixel summary in high resolution. Default:
                        False
  --no-high-res-badpix-3d
                        Opposite of --high-res-badpix-3d (default: False)
  --test-for-normality  permorm normality test. Default: False
  --no-test-for-normality
                        Opposite of --test-for-normality (default: False)
  --operation-mode str  Detector operation mode, optional. Default:

required arguments:
  --in-folder str       path to input data, required. Default: None
  --out-folder str      path to output to, required. Default: None
  --run-high int        run number in which high gain data was recorded,
                        required. Default: None
  --run-med int         run number in which medium gain data was recorded,
                        required. Default: None
  --run-low int         run number in which low gain data was recorded,
                        required. Default: None

LPD Radial X-ray Gain Evaluation

Author: S. Hauf, Version 0.5

Taking proper flat field for LPD can be difficult, as air scattering will always be present. Additionally, the large detector mandates a large distance to the source, in order to reduce \(1/r\) effects.

Because of this a radial evaluation method is used, which assumes that pixels a the same radial distance \(r\) should on average have the same signal \(S(r)\).

To invoke this notebook and display help use:

xfel-calibrate LPD FF --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      --run int [--sequences int [int ...]]
                      [--capacitor-setting int] [--mem-cells int]
                      [--local-output | --no-local-output]
                      [--db-output | --no-db-output] [--bias-voltage int]
                      [--cal-db-interface str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--instrument str] [--limit-trains int]
                      [--geometry-file str]
                      [--beam-center-offset float [float ...]]
                      [--allowed-gain-thresholds float [float ...]]
                      [--badpix-entire-asic-threshold float]
                      [--photon-energy float] [--max-images int]
                      LPD FF

# LPD Radial X-ray Gain Evaluation #
,
Author: S. Hauf, Version 0.5,
,
Taking proper flat field for LPD can be difficult, as air scattering will
always be present. Additionally, the large detector mandates a large distance
to the source, in order to reduce $1/r$ effects.
,
Because of this a radial evaluation method is used, which assumes that pixels
a the same radial distance $r$ should on average have the same signal $S(r)$.

positional arguments:
  LPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  FF                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --sequences int [int ...]
                        which sequence files to use. Default: [0]
  --capacitor-setting int
                        capacitor_setting for which data was taken. Default: 5
  --mem-cells int       number of memory cells used. Default: 512
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --bias-voltage int    detector bias voltage. Default: 250
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl015:5005
  --use-dir-creation-date
                        use the creation date of the directory for database
                        time derivation. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --instrument str      Default: FXE
  --limit-trains int    limit the number of train for the evaluation. Default:
                        10
  --geometry-file str   the geometry file to use, MAR 2018. Default:
                        /gpfs/exfel/d/cal/exchange/lpdMF_00.h5
  --beam-center-offset float [float ...]
                        offset from the beam center, MAR 2018. Default: [1.5,
                        1]
  --allowed-gain-thresholds float [float ...]
                        relative gain values within these bounds are valid.
                        Default: [0.5, 1.5]
  --badpix-entire-asic-threshold float
                        if more than this fraction of pixels on an ASIC are
                        "bad" the entire ASIC is flagged. Default: 0.25
  --photon-energy float
                        photon enery in keV. Default: 9.2
  --max-images int      maximum number of images to use in evaluation.
                        Default: 1024

required arguments:
  --in-folder str       path to already corrected input data, required.
                        Default: None
  --out-folder str      path to output to, required. Default: None
  --run int             runs to process, required. Default: None

Injecting calibration constant data to the database

Author: European XFEL Detector Group, Version: 1.0

Reading h5files of calibration constants to inject them to the database. Used for LPD

To invoke this notebook and display help use:

xfel-calibrate LPD INJECT_CONSTANTS --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --constant-names str [str ...]
                      --in-folder str --out-folder str [--proposal str]
                      [--runs str [str ...]] [--karabo-id str]
                      [--karabo-da str [str ...]] [--cal-db-interface str]
                      [--memory-cells int] [--bias-voltage int]
                      [--capacitor int] [--category int]
                      [--photon-energy float] --creation-time str
                      LPD INJECT_CONSTANTS

# Injecting calibration constant data to the database #
,
Author: European XFEL Detector Group, Version: 1.0,
,
Reading h5files of calibration constants to inject them to the database. Used
for LPD

positional arguments:
  LPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  INJECT_CONSTANTS                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --proposal str        Add proposal number to be sent to the database as a
                        part of Raw data location.. Default:
  --runs str [str ...]  Add list of runs to be sent to the database as a part
                        of Raw data location.. Default: ['']
  --karabo-id str       detector identifier.. Default: FXE_DET_LPD1M-1
  --karabo-da str [str ...]
                        karabo data aggregators. default "all" for all 16
                        karabo data aggregator names.. Default: ['all']
  --cal-db-interface str
                        calibration DB zmq address.. Default: tcp://max-
                        exfl016:8015#8045
  --memory-cells int    Number of memory cells. Used for constant conditions..
                        Default: 512
  --bias-voltage int    bias voltage value. Used for constant conditions..
                        Default: 250
  --capacitor int       capacitor value. Used for constant conditions..
                        Default: 5
  --category int        calibration constant source category, 0 for European
                        XFEl and 1 for RAL. Used for constant conditions..
                        Default: 0
  --photon-energy float
                        calibration constant photon energy. Used for constant
                        conditions.. Default: 9.2

required arguments:
  --constant-names str [str ...]
                        calibration constant name, required.. Default: None
  --in-folder str       calibration constants folder, required.. Default: None
  --out-folder str      output folder to store report path in case the
                        notebook is executed by CLI, required.. Default: None
  --creation-time str   creation time for the injected constants. required
                        format '2019-01-20T14:12:06'. Default: None

LPD Gain Characterization (Charge Injection)

Author: S. Hauf, Version: 0.5

The following code characterizes the gain of the LPD detector from charge injection data, i.e. data with charge injected into the amplifiers, bypassing the sensor. The data needs to fulfil the following requirements:

  • each file should represent one scan point for one mudle, defined by detector gain setting and charge injections setting
  • settings need to overlap at at least one point for two neighboring gain ranges
  • 100 samples or more per pixel and memory cell should be present for each setting.

The data is then analyzed by calcualting the per-pixel, per memory cell mean of the samples for each setting. These means are then normalized to the median peak position of a all means of the first module. Overlapping settings in neighboring gain ranges are used to deduce the slopes of the different gains with respect to the high gain setting.

To invoke this notebook and display help use:

xfel-calibrate LPD PC --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str] --in-folder
                      str --out-folder str [--offset-store str]
                      [--modules int [int ...]] [--capacitor-setting int]
                      [--mem-cells int] [--local-output | --no-local-output]
                      [--db-output | --no-db-output]
                      [--db-input | --no-db-input] [--bias-voltage int]
                      [--cal-db-interface str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--instrument str]
                      [--high-res-badpix-3d | --no-high-res-badpix-3d]
                      LPD PC

# LPD Gain Characterization (Charge Injection) #
,
Author: S. Hauf, Version: 0.5,
,
The following code characterizes the gain of the LPD detector from charge
injection data, i.e. data with charge injected into the amplifiers, bypassing
the sensor. The data needs to fulfil the following requirements:,
,
* each file should represent one scan point for one mudle, defined by
detector gain setting
  and charge injections setting,
* settings need to overlap at at least one point for two neighboring gain
ranges,
* 100 samples or more per pixel and memory cell should be present for each
setting.,
,
The data is then analyzed by calcualting the per-pixel, per memory cell mean
of the samples for each setting. These means are then normalized to the
median peak position of a all means of the first module. Overlapping settings
in neighboring gain ranges are used to deduce the slopes of the different
gains with respect to the high gain setting.

positional arguments:
  LPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  PC                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        The ipcluster profile to use. Default: noDB
  --offset-store str    path to offset store, overwrite use_db_input. Default:
  --modules int [int ...]
                        Default: [-1]
  --capacitor-setting int
                        capacitor_setting for which data was taken. Default: 5
  --mem-cells int       number of memory cells used. Default: 512
  --local-output        output constants locally. Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --db-output           output constants to database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --db-input            Default: True
  --no-db-input         Opposite of --db-input (default: True)
  --bias-voltage int    detector bias voltage. Default: 300
  --cal-db-interface str
                        the database interface to use. Default: tcp://max-
                        exfl016:8015
  --use-dir-creation-date
                        use the creation date of the directory for database
                        time derivation. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --instrument str      Default: FXE
  --high-res-badpix-3d  Default: False
  --no-high-res-badpix-3d
                        Opposite of --high-res-badpix-3d (default: False)

required arguments:
  --in-folder str       path to input data, required. Default: None
  --out-folder str      path to output to, required. Default: None

Mine XGM Data and LPD Profile

Author: S. Hauf, Version: 0.1

To invoke this notebook and display help use:

xfel-calibrate LPD XGM_MINE --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--dc float [float ...]]
                      [--geometry-file str] [--proposal-folder str]
                      [--xgm-folder str] [--adc-folder str]
                      [--file-prefix str] [--raw-prefix str]
                      [--xgm-prefix str] [--adc-prefix str] [--data-path str]
                      [--gain-path str] [--xgm-path str] [--xgm-path-td str]
                      [--adc-root str] [--adc-paths str [str ...]]
                      [--temp-out-folder str] [--cluster-profile str]
                      [--chunk-size int] [--runs str [str ...]] [--method str]
                      [--sequences str [str ...]]
                      LPD XGM_MINE

# Mine XGM Data and LPD Profile #
Author: S. Hauf, Version: 0.1

positional arguments:
  LPD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  XGM_MINE                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --dc float [float ...]
                        center offset. Default: [1.5, 1]
  --geometry-file str   the geometry file to use, MAR 2018. Default:
                        /gpfs/exfel/d/cal/exchange/lpdMF_00.h5
  --proposal-folder str
                        folder under which runs can be found. Default:
                        /gpfs/exfel/exp/FXE/201701/p002045/proc/
  --xgm-folder str      folder under which runs can be found. Default:
                        /gpfs/exfel/exp/FXE/201701/p002045/raw/
  --adc-folder str      folder under which runs can be found. Default:
                        /gpfs/exfel/exp/FXE/201701/p002045/raw/
  --file-prefix str     prefix of each data file - should be a template to
                        replace run number. Default: CORR-R{:04d}-LPD
  --raw-prefix str      prefix of each data file - should be a template to
                        replace run number. Default: RAW-R{:04d}-LPD
  --xgm-prefix str      prefix for XGM file - should be a template to replace
                        run number. Default: RAW-R{:04d}-DA03
  --adc-prefix str      prefix for ADC file - should be a template to replace
                        run number. Default: RAW-R{:04d}-DA01
  --data-path str       path to data in file. Default:
                        INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/data
  --gain-path str       path to data in file. Default:
                        INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/gain
  --xgm-path str        path to XGM data. Default: /INSTRUMENT/SA1_XTD2_XGM/DO
                        OCS/MAIN:output/data/intensityTD
  --xgm-path-td str     path to XGM AUX data. Default: /INSTRUMENT/SA1_XTD2_XG
                        M/DOOCS/MAIN:output/data/intensityAUXTD
  --adc-root str        root path to ADC data. Default:
                        /INSTRUMENT/FXE_RR_DAQ/ADC/1:network/digitizers
  --adc-paths str [str ...]
                        Default: ['channel_1_A/apd/pulseIntegral',
                        'channel_1_B/apd/pulseIntegral',
                        'channel_1_C/apd/pulseIntegral',
                        'channel_1_D/apd/pulseIntegral']
  --temp-out-folder str
                        Default: /gpfs/exfel/data/scratch/haufs/test/tempout6
  --cluster-profile str
                        Default: noDB
  --chunk-size int      read this amount of data at once. Default: 512
  --runs str [str ...]  Default: 69-72
  --method str          method to use for evaluation of images: radial,
                        average. Default: average
  --sequences str [str ...]
                        range allowed. Default: [-1]

PNCCD

pnCCD Data Correction

Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0

The following notebook provides offset, common mode, relative gain, split events and pattern classification corrections of images acquired with the pnCCD. This notebook does not yet correct for charge transfer inefficiency.

To invoke this notebook and display help use:

xfel-calibrate PNCCD CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--in-folder str] [--out-folder str]
                      [--run int] [--sequences str [str ...]]
                      [--sequences-per-node int] [--karabo-da str]
                      [--karabo-id str] [--receiver-id str]
                      [--path-template str] [--instrument-source-template str]
                      [--commonModeAxis int]
                      [--commonModeBlockSize int [int ...]]
                      [--split-evt-primary-threshold float]
                      [--split-evt-secondary-threshold float]
                      [--saturated-threshold float]
                      [--fix-temperature-top float]
                      [--fix-temperature-bot float] [--gain int]
                      [--bias-voltage float] [--integration-time int]
                      [--photon-energy float] [--cal-db-interface str]
                      [--cal-db-timeout int] [--creation-time str]
                      [--only-offset | --no-only-offset]
                      [--common-mode | --no-common-mode]
                      [--relgain | --no-relgain]
                      [--pattern-classification | --no-pattern-classification]
                      [--chunk-size-idim int] [--limit-images int]
                      PNCCD CORRECT

# pnCCD Data Correction #
,
Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0,
,
The following notebook provides offset, common mode, relative gain, split
events and pattern classification corrections of images acquired with the
pnCCD. This notebook *does not* yet correct for charge transfer inefficiency.

positional arguments:
  PNCCD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --in-folder str       input folder. Default:
                        /gpfs/exfel/exp/SQS/202031/p900166/raw
  --out-folder str      output folder. Default: /gpfs/exfel/data/scratch/ahmed
                        k/test/remove/pnccd_correct
  --run int             which run to read data from. Default: 347
  --sequences str [str ...]
                        sequences to correct, set to -1 for all, range
                        allowed. Default: [-1]
  --sequences-per-node int
                        number of sequences running on the same slurm node..
                        Default: 1
  --karabo-da str       data aggregators. Default: PNCCD01
  --karabo-id str       karabo prefix of PNCCD devices. Default:
                        SQS_NQS_PNCCD1MP
  --receiver-id str     inset for receiver devices. Default: PNCCD_FMT-0
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{:05d}.h5
  --instrument-source-template str
                        template for data source name, will be filled with
                        karabo_id and receiver_id.. Default: {}/CAL/{}:output
  --commonModeAxis int  axis along which common mode will be calculated, 0 =
                        row, and 1 = column. Default: 0
  --commonModeBlockSize int [int ...]
                        size of the detector in pixels for common mode
                        calculations. Default: [512, 512]
  --split-evt-primary-threshold float
                        primary threshold for split event classification in
                        terms of n sigma noise. Default: 4.0
  --split-evt-secondary-threshold float
                        secondary threshold for split event classification in
                        terms of n sigma noise. Default: 3.0
  --saturated-threshold float
                        full well capacity in ADU. Default: 32000.0
  --fix-temperature-top float
                        fix temperature for top sensor in K, set to 0. to use
                        value from slow data.. Default: 0.0
  --fix-temperature-bot float
                        fix temperature for bottom senspr in K, set to 0. to
                        use value from slow data.. Default: 0.0
  --gain int            the detector's gain setting. Set to -1 to use the
                        value from the slow data.. Default: -1
  --bias-voltage float  the detector's bias voltage. set to 0. to use value
                        from slow data.. Default: 0.0
  --integration-time int
                        detector's integration time. Default: 70
  --photon-energy float
                        Al fluorescence in keV. Default: 1.6
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8015
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --creation-time str   The timestamp to use with Calibration DB. Required
                        Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
                        11:02:41. Default:
  --only-offset         Only, apply offset.. Default: False
  --no-only-offset      Opposite of --only-offset (default: False)
  --common-mode         Apply common mode correction. Default: True
  --no-common-mode      Opposite of --common-mode (default: True)
  --relgain             Apply relative gain correction. Default: True
  --no-relgain          Opposite of --relgain (default: True)
  --pattern-classification
                        classify split events. Default: True
  --no-pattern-classification
                        Opposite of --pattern-classification (default: True)
  --chunk-size-idim int
                        H5 chunking size of output data. Default: 1
  --limit-images int    this parameter is used for limiting number of images
                        to correct from a sequence file. ONLY FOR TESTING..
                        Default: 0

pnCCD Dark Characterization

Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0

The following notebook provides dark image analysis of the pnCCD detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.

On the first iteration, the offset and noise maps are generated. Initial bad pixels map is obtained based on the offset and initial noise maps. Edge pixels are also added to the bad pixels map.

On the second iteration, the noise map is corrected for common mode. A second bad pixel map is generated based on the offset map and offset-and-common-mode-corrected noise map. Then, the hole in the center of the CCD is added to the second bad pixel map.

On the third and final iteration, the pixels that show up on the abovementioned bad pixels map are masked. Possible events due to cosmic rays are found and masked. The data are then again offset and common mode corrected and a new final noise and bad pixels maps are generated.

These latter resulting maps together with the offset map are saved as .h5 files to a local path for a later use. These dark constants are not automatically sent to the database.

To invoke this notebook and display help use:

xfel-calibrate PNCCD DARK --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] --in-folder str --out-folder str
                      --run int [--karabo-da str [str ...]] [--karabo-id str]
                      [--receiver-id str] [--path-template str]
                      [--instrument-source-template str]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--db-output | --no-db-output]
                      [--local-output | --no-local-output]
                      [--creation-time str] [--fix-temperature-top float]
                      [--fix-temperature-bot float] [--temp-limits int]
                      [--gain int] [--bias-voltage float]
                      [--integration-time int] [--commonModeAxis int]
                      [--commonModeBlockSize int [int ...]]
                      [--sigmaNoise float] [--bad-pixel-offset-sigma float]
                      [--bad-pixel-noise-sigma float] [--max-trains int]
                      [--min-trains int] [--operation-mode str]
                      [--db-module str]
                      PNCCD DARK

# pnCCD Dark Characterization
,
Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0,
,
The following notebook provides dark image analysis of the pnCCD detector.
Dark characterization evaluates offset and noise of the detector and gives
information about bad pixels.
,
On the first iteration, the offset and noise maps are generated. Initial bad
pixels map is obtained based on the offset and initial noise maps. Edge
pixels are also added to the bad pixels map.,
,
On the second iteration, the noise map is corrected for common mode. A second
bad pixel map is generated based on the offset map and
offset-and-common-mode-corrected noise map. Then, the hole in the center of
the CCD is added to the second bad pixel map.,
,
On the third and final iteration, the pixels that show up on the
abovementioned bad pixels map are masked. Possible events due to cosmic rays
are found and masked. The data are then again offset and common mode
corrected and a new final noise and bad pixels maps are generated.,
,
These latter resulting maps together with the offset map are saved as .h5
files to a local path for a later use. These dark constants are not
automatically sent to the database.

positional arguments:
  PNCCD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  DARK                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --karabo-da str [str ...]
                        data aggregators. Default: ['PNCCD01']
  --karabo-id str       karabo prefix of PNCCD devices. Default:
                        SQS_NQS_PNCCD1MP
  --receiver-id str     inset for receiver devices. Default: PNCCD_FMT-0
  --path-template str   the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --instrument-source-template str
                        data source path in h5file. Template filled with
                        karabo_id and receiver_id. Default: {}/CAL/{}:output
  --use-dir-creation-date
                        use dir creation date as data production reference
                        date. Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8021
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --db-output           if True, the notebook sends dark constants to the
                        calibration database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --local-output        if True, the notebook saves dark constants locally.
                        Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --creation-time str   To overwrite the measured creation_time. Required
                        Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04
                        11:02:41.00. Default:
  --fix-temperature-top float
                        fix temperature of top pnCCD sensor in K. Set to 0, to
                        use the value from slow data. Default: 0.0
  --fix-temperature-bot float
                        fix temperature of bottom pnCCD sensor in K. Set to 0,
                        to use the value from slow data. Default: 0.0
  --temp-limits int     temperature limits in which calibration parameters are
                        considered equal. Default: 5
  --gain int            the detector's gain setting. Set to -1 to use the
                        value from the slow data.. Default: -1
  --bias-voltage float  the detector's bias voltage. set to 0. to use the
                        value from slow data.. Default: 0.0
  --integration-time int
                        detector's integration time.. Default: 70
  --commonModeAxis int  axis along which common mode will be calculated (0:
                        along rows, 1: along columns). Default: 0
  --commonModeBlockSize int [int ...]
                        size of the detector in pixels for common mode
                        calculations. Default: [512, 512]
  --sigmaNoise float    pixels whose signal value exceeds sigmaNoise*noise
                        will be considered as cosmics and are masked. Default:
                        10.0
  --bad-pixel-offset-sigma float
                        any pixel whose offset beyond this standard deviations
                        is a bad pixel. Default: 4.0
  --bad-pixel-noise-sigma float
                        any pixel whose noise beyond this standard deviations
                        is a bad pixel. Default: 4.0
  --max-trains int      Maximum number of trains to use for dark processing.
                        Set to 0 to process all trains.. Default: 500
  --min-trains int      Minimum number of trains to proceed with dark
                        processing.. Default: 1
  --operation-mode str  Detector operation mode, optional. Default:
  --db-module str       the device name for pnCCD detector. Default:

required arguments:
  --in-folder str       input folder, required. Default: None
  --out-folder str      output folder, required. Default: None
  --run int             which run to read data from, required. Default: None

pnCCD Gain Characterization

Authors: DET Group, modified by Kiana Setoodehnia on December 2020 - Version 4.0

The following notebook provides gain characterization for the pnCCD. It relies on data which are previously corrected using the Meta Data Catalog web service interface or by running the Correct_pnCCD_NBC.ipynb notebook. Prior to running this notebook, the corrections which should be applied by the web service or the aforementioned notebook are as follows:

  • offset correction
  • common mode correction
  • split pattern classification

To invoke this notebook and display help use:

xfel-calibrate PNCCD RELGAIN --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--cluster-profile str]
                      [--in-folder str] [--out-folder str] [--run int]
                      [--sequences str [str ...]] [--db-module str]
                      [--karabo-da str] [--karabo-da-control str]
                      [--karabo-id str] [--receiver-id str]
                      [--path-template str] [--path-template-ctrl str]
                      [--path-template-seqs str] [--h5path str]
                      [--h5path-ctrl str] [--cpuCores int]
                      [--use-dir-creation-date | --no-use-dir-creation-date]
                      [--sequences-per-node int] [--chunkSize int]
                      [--run-parallel | --no-run-parallel]
                      [--db-output | --no-db-output]
                      [--local-output | --no-local-output]
                      [--cal-db-interface str] [--cal-db-timeout int]
                      [--creation-time str] [--fix-temperature-top float]
                      [--fix-temperature-bot float] [--gain float]
                      [--bias-voltage float] [--integration-time int]
                      [--photon-energy float]
                      PNCCD RELGAIN

# pnCCD Gain Characterization #
,
Authors: DET Group, modified by Kiana Setoodehnia on December 2020 - Version
4.0,
,
The following notebook provides gain characterization for the pnCCD. It
relies on data which are previously  corrected using the Meta Data Catalog
web service interface or by running the Correct_pnCCD_NBC.ipynb notebook.
Prior to running this notebook, the corrections which should be applied by
the web service or the aforementioned notebook are as follows:,
,
- offset correction,
- common mode correction,
- split pattern classification

positional arguments:
  PNCCD              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  RELGAIN                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --cluster-profile str
                        ipcluster profile to use. Default: noDB
  --in-folder str       input folder for the raw data. Default:
                        /gpfs/exfel/exp/SQS/202031/p900166/raw
  --out-folder str      output folder. Default:
                        /gpfs/exfel/data/scratch/setoodeh/Test
  --run int             which run to read data from. Default: 347
  --sequences str [str ...]
                        sequences to correct, set to -1 for all, range
                        allowed. Default: [-1]
  --db-module str       Default: pnCCD_M205_M206
  --karabo-da str       data aggregators. Default: PNCCD01
  --karabo-da-control str
                        file inset for control data. Default: PNCCD02
  --karabo-id str       karabo prefix of PNCCD devices. Default:
                        SQS_NQS_PNCCD1MP
  --receiver-id str     inset for receiver devices. Default: PNCCD_FMT-0
  --path-template str   the template to use to access data. Default:
                        CORR-R{:04d}-PNCCD01-S{{:05d}}.h5
  --path-template-ctrl str
                        the template to use to access data. Default:
                        RAW-R{:04d}-{}-S{{:05d}}.h5
  --path-template-seqs str
                        Default: {}/r{:04d}/*PNCCD01-S*.h5
  --h5path str          path to data in the HDF5 file. Default:
                        /INSTRUMENT/{}/CAL/{}:output/data/
  --h5path-ctrl str     Default: /CONTROL/{}/CTRL/TCTRL
  --cpuCores int        specifies the number of running cpu cores. Default: 40
  --use-dir-creation-date
                        this is needed to obtain creation time of the run.
                        Default: True
  --no-use-dir-creation-date
                        Opposite of --use-dir-creation-date (default: True)
  --sequences-per-node int
                        Default: 1
  --chunkSize int       number of images to read per chunk. Default: 100
  --run-parallel        Default: True
  --no-run-parallel     Opposite of --run-parallel (default: True)
  --db-output           if True, the notebook injects dark constants into the
                        calibration database. Default: False
  --no-db-output        Opposite of --db-output (default: False)
  --local-output        if True, the notebook saves dark constants locally.
                        Default: True
  --no-local-output     Opposite of --local-output (default: True)
  --cal-db-interface str
                        calibration DB interface to use. Default: tcp://max-
                        exfl016:8015
  --cal-db-timeout int  timeout on caldb requests. Default: 300000
  --creation-time str   To overwrite the measured creation_time. Required
                        Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04
                        11:02:41.00. Default:
  --fix-temperature-top float
                        fix temperature of top pnCCD sensor in K, set to 0. to
                        use the value from slow data. Default: 0.0
  --fix-temperature-bot float
                        fix temperature of bottom pnCCD sensor in K, set to 0.
                        to use the value from slow data. Default: 0.0
  --gain float          the detector's gain setting. Set to 0, to use the
                        value from slow data. Default: 0.1
  --bias-voltage float  the detector's bias voltage. set to 0. to use the
                        value from slow data.. Default: 0.0
  --integration-time int
                        detector's integration time. Default: 70
  --photon-energy float
                        Al fluorescence in keV. Default: 1.6

REMI

Transformation parameters

To invoke this notebook and display help use:

xfel-calibrate REMI CORRECT --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--run int] [--in-folder str]
                      [--out-folder str] [--calib-config-path str]
                      [--cycle str] [--cal-db-timeout int]
                      [--cal-db-interface str] [--karabo-da str]
                      [--karabo-id str] [--proposal str]
                      [--out-aggregator str] [--out-seq-len int]
                      [--det-device-id str] [--det-output-key str]
                      [--save-raw-triggers | --no-save-raw-triggers]
                      [--save-raw-edges | --no-save-raw-edges]
                      [--save-rec-signals | --no-save-rec-signals]
                      [--save-rec-hits | --no-save-rec-hits]
                      [--chunks-triggers int [int ...]]
                      [--chunks-edges int [int ...]]
                      [--chunks-hits int [int ...]]
                      [--chunks-signals int [int ...]]
                      [--dataset-compression str]
                      [--dataset-compression-opts int]
                      [--quad-anode | --no-quad-anode] [--ppt-source str]
                      [--ignore-fel | --no-ignore-fel]
                      [--ignore-ppl | --no-ignore-ppl] [--ppl-offset int]
                      [--laser-ppt-mask int] [--instrument-sase int]
                      [--first-pulse-offset int] [--single-pulse-length int]
                      [--mp-find-triggers float] [--mp-find-edges float]
                      [--mt-avg-trace int] [--mp-rec-hits float]
                      REMI CORRECT

# Transformation parameters

positional arguments:
  REMI              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  CORRECT                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --run int             Run ID.. Default: 104
  --in-folder str       Partial input path appended with run ID.. Default:
                        /gpfs/exfel/exp/SQS/202101/p002535/raw
  --out-folder str      Full path to output folder.. Default:
                        /gpfs/exfel/exp/SQS/202101/p002535/scratch/cal_test
  --calib-config-path str
                        Path to correction and transform configuration.
                        Default: /gpfs/exfel/exp/SQS/202101/p002535/usr/config
                        _board2+4.yaml
  --cycle str           Proposal cycle, currently not used.. Default:
  --cal-db-timeout int  Calibration DB timeout, currently not used.. Default:
                        0
  --cal-db-interface str
                        Calibration DB interface, currently not used..
                        Default: foo
  --karabo-da str       Karabo data aggregator name, currently not used.
                        Default: bar
  --karabo-id str       Karabo device ID root for virtual output device..
                        Default: SQS_REMI_DLD6
  --proposal str        Proposal, leave empty for auto detection based on
                        in_folder. Default:
  --out-aggregator str  Aggregator name for output files.. Default: REMI01
  --out-seq-len int     Number of trains per sequence file in output..
                        Default: 5000
  --det-device-id str   Karabo device ID for virtual output device.. Default:
                        {karabo_id}/DET/{det_name}
  --det-output-key str  Pipeline name for fast data output.. Default: output
  --save-raw-triggers   Whether to save trigger position in files.. Default:
                        True
  --no-save-raw-triggers
                        Opposite of --save-raw-triggers (default: True)
  --save-raw-edges      Whether to save digitized edge positions in files..
                        Default: True
  --no-save-raw-edges   Opposite of --save-raw-edges (default: True)
  --save-rec-signals    Whether to save reconstructed signals (u1-w2, mcp) in
                        files.. Default: True
  --no-save-rec-signals
                        Opposite of --save-rec-signals (default: True)
  --save-rec-hits       Whether to save reoncstructed hits (x,y,t,m) in
                        files.. Default: True
  --no-save-rec-hits    Opposite of --save-rec-hits (default: True)
  --chunks-triggers int [int ...]
                        HDF chunk size for triggers.. Default: [500]
  --chunks-edges int [int ...]
                        HDF chunk size for edges.. Default: [500, 7, 50]
  --chunks-hits int [int ...]
                        HDF chunk size for hits.. Default: [50, 50]
  --chunks-signals int [int ...]
                        HDF chunk size for signals.. Default: [50, 50]
  --dataset-compression str
                        HDF compression method.. Default: gzip
  --dataset-compression-opts int
                        HDF GZIP compression level.. Default: 3
  --quad-anode          Reconstruction assumes a hex anode by default, change
                        for quad anodes.. Default: False
  --no-quad-anode       Opposite of --quad-anode (default: False)
  --ppt-source str      Default: SQS_RR_UTC/TSYS/TIMESERVER:outputBunchPattern
  --ignore-fel          Ignore any FEL entries in the PPT.. Default: False
  --no-ignore-fel       Opposite of --ignore-fel (default: False)
  --ignore-ppl          Ignore any PPL entries in the PPT.. Default: False
  --no-ignore-ppl       Opposite of --ignore-ppl (default: False)
  --ppl-offset int      In units of the PPT.. Default: 0
  --laser-ppt-mask int  Bit mask for used laser, negative to auto-detect from
                        instrument.. Default: -1
  --instrument-sase int
                        Default: 3
  --first-pulse-offset int
                        Default: 1000
  --single-pulse-length int
                        Default: 25000
  --mp-find-triggers float
                        Parallelization for finding triggers.. Default: 0.5
  --mp-find-edges float
                        Parallelization for digitizing analog signal..
                        Default: 0.5
  --mt-avg-trace int    Parallelization for trace averaging.. Default: 2
  --mp-rec-hits float   Parallelization for hit reconstruction.. Default: 1.0

TUTORIAL

Tutorial Calculation

Author: Astrid Muennich

Version: 0.1

A small example how to adapt a notebook to run with the offline calibration package “pycalibation”.

The first cell contains all parameters that should be exposed to the command line.

To run this notebooks with several different input parameters in parallel by submitting multiple slurm jobs, for example for various random seed we can do the following:

xfel-calibrate TUTORIAL TEST –random-seed 1,2,3,4

or

xfel-calibrate TUTORIAL TEST –random-seed 1-5

will produce 4 jobs:

Parsed input 1,2,3,4 to [1, 2, 3, 4]

Submitted job: 1169340

Submitted job: 1169341

Submitted job: 1169342

Submitted job: 1169343

Submitted the following SLURM jobs: 1169340,1169341,1169342,1169343

To invoke this notebook and display help use:

xfel-calibrate TUTORIAL TEST --help

The full parameter list of this notebook (with defaults is):

usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
                      [--report-to str] [--not-reproducible] [--skip-report]
                      [--skip-env-freeze] [--concurrency-par str]
                      [--constants-from str] [--vector-figs] [--slurm-mem int]
                      [--slurm-name str] [--slurm-scheduling int]
                      [--request-time str] [--slurm-partition str]
                      [--reservation str] [--out-folder str]
                      [--sensor-size int [int ...]]
                      [--random-seed str [str ...]] [--runs int]
                      [--cluster-profile str]
                      TUTORIAL TEST

# Tutorial Calculation #
,
Author: Astrid Muennich,
,
Version: 0.1,
,
A small example how to adapt a notebook to run with the offline calibration
package "pycalibation".,
,
The first cell contains all parameters that should be exposed to the command
line.,
,
To run this notebooks with several different input parameters in parallel by
submitting multiple slurm jobs, for example for various random seed we can do
the following:,
,
xfel-calibrate TUTORIAL TEST --random-seed 1,2,3,4,
,
or,
,
xfel-calibrate TUTORIAL TEST --random-seed 1-5,
,
will produce 4 jobs:,
,
Parsed input 1,2,3,4 to [1, 2, 3, 4],
,
Submitted job: 1169340,
,
Submitted job: 1169341,
,
Submitted job: 1169342,
,
Submitted job: 1169343,
,
Submitted the following SLURM jobs: 1169340,1169341,1169342,1169343

positional arguments:
  TUTORIAL              The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
                        TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
                        EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
  TEST                  Type of calibration.

optional arguments:
  -h, --help            show this help message and exit
  --no-cluster-job      Do not run as a cluster job (default: False)
  --prepare-only        Prepare notebooks but don't run them (default: False)
  --report-to str       Filename (and optionally path) for output report
                        (default: None)
  --not-reproducible    Disable checks to allow the processing result to not
                        be reproducible based on its metadata. (default:
                        False)
  --skip-report         Skip report generation in finalize step. (default:
                        False)
  --skip-env-freeze     Skip recording the Python environment for
                        reproducibility purposes, requires --not-reproducible
                        to run. (default: False)
  --concurrency-par str
                        Name of concurrency parameter.If not given, it is
                        taken from configuration. (default: None)
  --constants-from str  Path to a calibration-metadata.yml file. If given,
                        retrieved-constants will be copied to use for a new
                        correction. (default: None)
  --vector-figs         Use vector graphics for figures in the report.
                        (default: False)
  --slurm-mem int       Requested node RAM in GB (default: 500)
  --slurm-name str      Name of slurm job (default: xfel_calibrate)
  --slurm-scheduling int
                        Change scheduling priority for a slurm job +-
                        2147483645 (negative value increases priority)
                        (default: 0)
  --request-time str    Time of request to process notebook. Iso format
                        (default: Now)
  --slurm-partition str
                        Submit jobs in this Slurm partition (default: )
  --reservation str     Submit jobs in this Slurm reservation, overriding
                        --slurm-partition if both are set (default: )
  --out-folder str      output folder. Default:
                        /gpfs/exfel/data/scratch/amunnich/tutorial
  --sensor-size int [int ...]
                        defining the picture size. Default: [10, 30]
  --random-seed str [str ...]
                        random seed for filling of fake data array. Change it
                        to produce different results, range allowed. Default:
                        [2345]
  --runs int            how may iterations to fill histograms. Default: 500
  --cluster-profile str
                        Default: tutorial