.. _available_notebooks: Available Notebooks =================== The following notebooks are currently integrated into the European XFEL Offline Calibration tool chain. AGIPD ----- Combine Constants +++++++++++++++++ Author: S. Hauf, Version: 0.1 This notebook combines constants from various evaluations - dark image analysis, yielding offset and noise - flat field analysis, yielding X-ray gain - pulse capacitor analysis, yielding medium gain stage slopes and thresholds - charge injection analysis, yielding low gain stage slopes and thresholds into a single set of calibration constants. These constants do not include offset and noise as they need to be reevaluated more frequently. Additionally, a bad pixel mask for all gain stages is deduced from the input. The mask contains dedicated entries for all pixels and memory cells as well as all three gains stages. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate AGIPD COMBINE --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--modules str [str ...]] --out-folder str [--local-output | --no-local-output] [--db-output | --no-db-output] [--bias-voltage int] [--cal-db-interface str] [--mem-cells int] [--instrument str] [--photon-energy float] [--offset-store str] [--gain-store str] [--pc-store-base str] [--ci-store str] [--high-res-badpix-3d | --no-high-res-badpix-3d] [--db-input | --no-db-input] [--deviation-threshold float] [--max-dev-high-gain float] [--max-dev-med-gain float] [--threshold-bounds-high-med int [int ...]] [--thresholds-offset-sigma float] [--thresholds-offset-hard int [int ...]] [--thresholds-noise-sigma float] [--thresholds-noise-hard int [int ...]] [--thresholds-xraygain-hard-high float [float ...]] [--thresholds-xraygain-hard-med float [float ...]] [--thresholds-xraygain-hard-low float [float ...]] [--no-flat-fields | --no-no-flat-fields] AGIPD COMBINE # Combine Constants # , Author: S. Hauf, Version: 0.1, , This notebook combines constants from various evaluations, , * dark image analysis, yielding offset and noise, * flat field analysis, yielding X-ray gain, * pulse capacitor analysis, yielding medium gain stage slopes and thresholds, * charge injection analysis, yielding low gain stage slopes and thresholds, , into a single set of calibration constants. These constants do not include offset and noise as they need to be reevaluated more frequently., , Additionally, a bad pixel mask for all gain stages is deduced from the input. The mask contains dedicated entries for all pixels and memory cells as well as all three gains stages. positional arguments: AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS COMBINE Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --modules str [str ...] modules to work on, range allowed. Default: [-1] --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --bias-voltage int detector bias voltage. Default: 300 --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:5005 --mem-cells int number of memory cells used. Default: 64 --instrument str Default: SPB --photon-energy float the photon energy in keV. Default: 9.2 --offset-store str for file-based access. Default: /gpfs/exfel/exp/SPB/20 1830/p900019/usr/calibration0618/dark0.5MHz/agipd_offs et_store_r0852_r0853_r0854.h5 --gain-store str for file-based access. Default: /gpfs/exfel/exp/SPB/20 1830/p900019/usr/calibration0618/FF/agipd_gain_store_r 0820_modules_CHANID.h5 --pc-store-base str for file-based access, use CHANID to indicate module ranges. Default: /gpfs/exfel/exp/SPB/201831/p900039/us r/AGIPD/PC/Veto_1MHZ/agipd_pc_store_38_39_40_41_42_43_ 44_60_CHANID_CHANID.h5 --ci-store str for file based access. Default: /gpfs/exfel/data/scrat ch/haufs/gain_Data/AGIPD_ci_data_74.h5 --high-res-badpix-3d set this to True if you need high-resolution 3d bad pixel plots. Runtime: ~ 1h. Default: False --no-high-res-badpix-3d Opposite of --high-res-badpix-3d (default: False) --db-input retreive data from calibration database, setting offset-store will overwrite this. Default: False --no-db-input Opposite of --db-input (default: False) --deviation-threshold float peaks with an absolute location deviation larger than the medium are are considere bad. Default: 0.75 --max-dev-high-gain float Default: 0.2 --max-dev-med-gain float Default: 0.5 --threshold-bounds-high-med int [int ...] Default: [100, 8100] --thresholds-offset-sigma float Default: 5.0 --thresholds-offset-hard int [int ...] Default: [4000, 9000] --thresholds-noise-sigma float Default: 10.0 --thresholds-noise-hard int [int ...] Default: [1, 20] --thresholds-xraygain-hard-high float [float ...] Default: [0.75, 1.25] --thresholds-xraygain-hard-med float [float ...] Default: [0.001, 0.25] --thresholds-xraygain-hard-low float [float ...] Default: [0.0001, 0.025] --no-flat-fields Default: False --no-no-flat-fields Opposite of --no-flat-fields (default: False) required arguments: --out-folder str path to output to, required. Default: None AGIPD Offline Correction ++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 2.0 Offline Calibration for the AGIPD Detector To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate AGIPD CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--sequences str [str ...]] [--modules str [str ...]] [--train-ids str [str ...]] --run int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-template str] [--path-template str] [--instrument-source-template str] [--index-source-template str] [--ctrl-source-template str] [--karabo-id-control str] [--slopes-ff-from-files str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--creation-date-offset str] [--mem-cells int] [--bias-voltage int] [--acq-rate float] [--gain-setting int] [--gain-mode int] [--overwrite | --no-overwrite] [--max-pulses int [int ...]] [--mem-cells-db int] [--integration-time int] [--blc-noise-threshold int] [--cm-dark-fraction float] [--cm-dark-range float [float ...]] [--cm-n-itr int] [--hg-hard-threshold int] [--mg-hard-threshold int] [--noisy-adc-threshold float] [--ff-gain float] [--photon-energy float] [--only-offset | --no-only-offset] [--rel-gain | --no-rel-gain] [--xray-gain | --no-xray-gain] [--blc-noise | --no-blc-noise] [--blc-stripes | --no-blc-stripes] [--blc-hmatch | --no-blc-hmatch] [--match-asics | --no-match-asics] [--adjust-mg-baseline | --no-adjust-mg-baseline] [--zero-nans | --no-zero-nans] [--zero-orange | --no-zero-orange] [--blc-set-min | --no-blc-set-min] [--corr-asic-diag | --no-corr-asic-diag] [--force-hg-if-below | --no-force-hg-if-below] [--force-mg-if-below | --no-force-mg-if-below] [--mask-noisy-adc | --no-mask-noisy-adc] [--common-mode | --no-common-mode] [--melt-snow | --no-melt-snow] [--mask-zero-std | --no-mask-zero-std] [--low-medium-gap | --no-low-medium-gap] [--round-photons | --no-round-photons] [--use-ppu-device str] [--ppu-train-offset int] [--use-litframe-finder str] [--litframe-device-id str] [--energy-threshold int] [--use-xgm-device str] [--recast-image-data str] [--compress-fields str [str ...]] [--skip-plots | --no-skip-plots] [--cell-id-preview int] [--chunk-size int] [--n-cores-correct int] [--n-cores-files int] [--sequences-per-node int] [--max-nodes int] [--max-tasks-per-worker int] AGIPD CORRECT # AGIPD Offline Correction # , Author: European XFEL Detector Group, Version: 2.0, , Offline Calibration for the AGIPD Detector positional arguments: AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequences str [str ...] sequences to correct, set to -1 for all, range allowed. Default: [-1] --modules str [str ...] modules to correct, set to -1 for all, range allowed. Default: [-1] --train-ids str [str ...] train IDs to correct, set to -1 for all, range allowed. Default: [-1] --karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1 --karabo-da str [str ...] a list of data aggregators names, Default [-1] for selecting all data aggregators. Default: ['-1'] --receiver-template str inset for receiver devices. Default: {}CH0 --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{:05d}.h5 --instrument-source-template str path in the HDF5 file to images. Default: {}/DET/{}:xtdf --index-source-template str path in the HDF5 file to images. Default: INDEX/{}/DET/{}:xtdf/ --ctrl-source-template str path to control information. Default: {}/MDL/FPGA_COMP --karabo-id-control str karabo-id for control device. Default: MID_EXP_AGIPD1M1 --slopes-ff-from-files str Path to locally stored SlopesFF and BadPixelsFF constants, loaded in precorrection notebook. Default: --use-dir-creation-date use the creation data of the input dir for database queries. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8015#8045 --cal-db-timeout int in milliseconds. Default: 30000 --creation-date-offset str add an offset to creation date, e.g. to get different constants. Default: 00:00:00 --mem-cells int Number of memory cells used, set to 0 to automatically infer. Default: 0 --bias-voltage int bias voltage, set to 0 to use stored value in slow data.. Default: 0 --acq-rate float the detector acquisition rate, use 0 to try to auto- determine. Default: 0.0 --gain-setting int the gain setting, use -1 to use value stored in slow data.. Default: -1 --gain-mode int gain mode (0: adaptive, 1-3 fixed high/med/low, -1: read from CONTROL data). Default: -1 --overwrite set to True if existing data should be overwritten. Default: True --no-overwrite Opposite of --overwrite (default: True) --max-pulses int [int ...] range list [st, end, step] of memory cell indices to be processed within a train. 3 allowed maximum list input elements.. Default: [0, 352, 1] --mem-cells-db int set to a value different than 0 to use this value for DB queries. Default: 0 --integration-time int integration time, negative values for auto-detection.. Default: -1 --blc-noise-threshold int above this mean signal intensity now baseline correction via noise is attempted. Default: 5000 --cm-dark-fraction float threshold for fraction of empty pixels to consider module enough dark to perform CM correction. Default: 0.66 --cm-dark-range float [float ...] range for signal value ADU for pixel to be consider as a dark pixel. Default: [-50.0, 30] --cm-n-itr int number of iterations for common mode correction. Default: 4 --hg-hard-threshold int threshold to force medium gain offset subtracted pixel to high gain. Default: 1000 --mg-hard-threshold int threshold to force medium gain offset subtracted pixel from low to medium gain. Default: 1000 --noisy-adc-threshold float threshold to mask complete adc. Default: 0.25 --ff-gain float conversion gain for absolute FlatField constants, while applying xray_gain. Default: 7.2 --photon-energy float photon energy in keV, non-positive value for XGM autodetection. Default: -1.0 --only-offset Apply only Offset correction. if False, Offset is applied by Default. if True, Offset is only applied.. Default: False --no-only-offset Opposite of --only-offset (default: False) --rel-gain do relative gain correction based on PC data. Default: False --no-rel-gain Opposite of --rel-gain (default: False) --xray-gain do relative gain correction based on xray data. Default: False --no-xray-gain Opposite of --xray-gain (default: False) --blc-noise if set, baseline correction via noise peak location is attempted. Default: False --no-blc-noise Opposite of --blc-noise (default: False) --blc-stripes if set, baseline corrected via stripes. Default: False --no-blc-stripes Opposite of --blc-stripes (default: False) --blc-hmatch if set, base line correction via histogram matching is attempted. Default: False --no-blc-hmatch Opposite of --blc-hmatch (default: False) --match-asics if set, inner ASIC borders are matched to the same signal level. Default: False --no-match-asics Opposite of --match-asics (default: False) --adjust-mg-baseline adjust medium gain baseline to match highest high gain value. Default: False --no-adjust-mg-baseline Opposite of --adjust-mg-baseline (default: False) --zero-nans set NaN values in corrected data to 0. Default: False --no-zero-nans Opposite of --zero-nans (default: False) --zero-orange set to 0 very negative and very large values in corrected data. Default: False --no-zero-orange Opposite of --zero-orange (default: False) --blc-set-min Shift to 0 negative medium gain pixels after offset corr. Default: False --no-blc-set-min Opposite of --blc-set-min (default: False) --corr-asic-diag if set, diagonal drop offs on ASICs are corrected. Default: False --no-corr-asic-diag Opposite of --corr-asic-diag (default: False) --force-hg-if-below set high gain if mg offset subtracted value is below hg_hard_threshold. Default: False --no-force-hg-if-below Opposite of --force-hg-if-below (default: False) --force-mg-if-below set medium gain if mg offset subtracted value is below mg_hard_threshold. Default: False --no-force-mg-if-below Opposite of --force-mg-if-below (default: False) --mask-noisy-adc Mask entire ADC if they are noise above a relative threshold. Default: False --no-mask-noisy-adc Opposite of --mask-noisy-adc (default: False) --common-mode Common mode correction. Default: False --no-common-mode Opposite of --common-mode (default: False) --melt-snow Identify (and optionally interpolate) 'snowy' pixels. Default: False --no-melt-snow Opposite of --melt-snow (default: False) --mask-zero-std Mask pixels with zero standard deviation across train. Default: False --no-mask-zero-std Opposite of --mask-zero-std (default: False) --low-medium-gap 5 sigma separation in thresholding between low and medium gain. Default: False --no-low-medium-gap Opposite of --low-medium-gap (default: False) --round-photons Round to absolute number of photons, only use with gain corrections. Default: False --no-round-photons Opposite of --round-photons (default: False) --use-ppu-device str Device ID for a pulse picker device to only process picked trains, empty string to disable. Default: --ppu-train-offset int When using the pulse picker, offset between the PPU's sequence start and actually picked train. Default: 0 --use-litframe-finder str Process only illuminated frames: 'off' - disable, 'device' - use online device data, 'offline' - use offline algorithm, 'auto' - choose online/offline source automatically (default). Default: off --litframe-device-id str Device ID for a lit frame finder device, empty string to auto detection. Default: --energy-threshold int The low limit for the energy (uJ) exposed by frames subject to processing. If -1000, selection by pulse energy is disabled. Default: -1000 --use-xgm-device str DoocsXGM device ID to obtain actual photon energy, operating condition else.. Default: --recast-image-data str Cast data to a different dtype before saving. Default: --compress-fields str [str ...] Datasets in image group to compress.. Default: ['gain', 'mask'] --skip-plots exit after writing corrected files and metadata. Default: False --no-skip-plots Opposite of --skip-plots (default: False) --cell-id-preview int cell Id used for preview in single-shot plots. Default: 1 --chunk-size int Size of chunk for image-wise correction. Default: 1000 --n-cores-correct int Number of chunks to be processed in parallel. Default: 16 --n-cores-files int Number of files to be processed in parallel. Default: 4 --sequences-per-node int number of sequence files per cluster node if run as SLURM job, set to 0 to not run SLURM parallel. Default: 2 --max-nodes int Maximum number of SLURM jobs to split correction work into. Default: 8 --max-tasks-per-worker int the number of tasks a correction pool worker process can complete before it will exit and be replaced with a fresh worker process. Leave as -1 to keep worker alive as long as pool.. Default: 1 required arguments: --in-folder str the folder to read data from, required. Default: None --out-folder str the folder to output to, required. Default: None --run int runs to process, required. Default: None AGIPD Characterize Dark Images ++++++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 2.0 The following code analyzes a set of dark images taken with the AGIPD detector to deduce detector offsets , noise, bad-pixel maps and thresholding. All four types of constants are evaluated per-pixel and per-memory cell. Data for the detector's three gain stages needs to be present, separated into separate runs. The evaluated calibration constants are stored locally and injected in the calibration data base. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate AGIPD DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--modules str [str ...]] --run-high int --run-med int --run-low int [--operation-mode str] [--karabo-id str] [--karabo-da str [str ...]] [--receiver-template str] [--instrument-source-template str] [--ctrl-source-template str] [--karabo-id-control str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--local-output | --no-local-output] [--db-output | --no-db-output] [--mem-cells int] [--bias-voltage int] [--gain-setting int] [--gain-mode int] [--integration-time int] [--acq-rate float] [--interlaced | --no-interlaced] [--thresholds-offset-sigma float] [--thresholds-offset-hard int [int ...]] [--thresholds-offset-hard-hg int [int ...]] [--thresholds-offset-hard-mg int [int ...]] [--thresholds-offset-hard-lg int [int ...]] [--thresholds-offset-hard-hg-fixed int [int ...]] [--thresholds-offset-hard-mg-fixed int [int ...]] [--thresholds-offset-hard-lg-fixed int [int ...]] [--thresholds-noise-sigma float] [--thresholds-noise-hard int [int ...]] [--thresholds-noise-hard-hg int [int ...]] [--thresholds-noise-hard-mg int [int ...]] [--thresholds-noise-hard-lg int [int ...]] [--thresholds-gain-sigma float] [--max-trains int] [--min-trains int] [--high-res-badpix-3d | --no-high-res-badpix-3d] AGIPD DARK # AGIPD Characterize Dark Images # , Author: European XFEL Detector Group, Version: 2.0, , The following code analyzes a set of dark images taken with the AGIPD detector to deduce detector offsets noise, bad-pixel maps and thresholding. All four types of constants are evaluated per-pixel and per-memory cell. Data "for the detectors three gain stages needs to be present, separated into " separate runs., , The evaluated calibration constants are stored locally and injected in the calibration data base., positional arguments: AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --modules str [str ...] list of modules to evaluate, RANGE ALLOWED. Default: [-1] --operation-mode str Detector operation mode, optional (defaults to "ADAPTIVE_GAIN"). Default: ADAPTIVE_GAIN --karabo-id str karabo karabo_id. Default: HED_DET_AGIPD500K2G --karabo-da str [str ...] a list of data aggregators names, Default [-1] for selecting all data aggregators. Default: ['-1'] --receiver-template str inset for receiver devices. Default: {}CH0 --instrument-source-template str path in the HDF5 file to images. Default: {}/DET/{}:xtdf --ctrl-source-template str path to control information. Default: {}/MDL/FPGA_COMP --karabo-id-control str karabo-id for control device '. Default: HED_EXP_AGIPD500K2G --use-dir-creation-date use dir creation date as data production reference date. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8020 --cal-db-timeout int timeout on caldb requests". Default: 3000000 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --mem-cells int number of memory cells used, set to 0 to automatically infer. Default: 0 --bias-voltage int bias voltage, set to 0 to use stored value in slow data.. Default: 0 --gain-setting int the gain setting, use -1 to use value stored in slow data.. Default: -1 --gain-mode int gain mode, use -1 to use value stored in slow data.. Default: -1 --integration-time int integration time, negative values for auto-detection.. Default: -1 --acq-rate float the detector acquisition rate, use 0 to try to auto- determine. Default: 0.0 --interlaced assume interlaced data format, for data prior to Dec. 2017. Default: False --no-interlaced Opposite of --interlaced (default: False) --thresholds-offset-sigma float offset sigma thresholds for offset deduced bad pixels. Default: 3.0 --thresholds-offset-hard int [int ...] For setting the same threshold offset for the 3 gains. Left for backcompatability. Default [0, 0] to take the following parameters.. Default: [0, 0] --thresholds-offset-hard-hg int [int ...] High-gain thresholds in absolute ADU terms for offset deduced bad pixels. Default: [3000, 7000] --thresholds-offset-hard-mg int [int ...] Medium-gain thresholds in absolute ADU terms for offset deduced bad pixels. Default: [6000, 10000] --thresholds-offset-hard-lg int [int ...] Low-gain thresholds in absolute ADU terms for offset deduced bad pixels. Default: [6000, 10000] --thresholds-offset-hard-hg-fixed int [int ...] Same as thresholds_offset_hard_hg, but for fixed gain operation. Default: [3500, 6500] --thresholds-offset-hard-mg-fixed int [int ...] Same as thresholds_offset_hard_mg, but for fixed gain operation. Default: [3500, 6500] --thresholds-offset-hard-lg-fixed int [int ...] Same as thresholds_offset_hard_lg, but for fixed gain operation. Default: [3500, 6500] --thresholds-noise-sigma float noise sigma thresholds for offset deduced bad pixels. Default: 5.0 --thresholds-noise-hard int [int ...] For setting the same threshold noise for the 3 gains. Left for backcompatability. Default [0, 0] to take the following parameters.. Default: [0, 0] --thresholds-noise-hard-hg int [int ...] High-gain thresholds in absolute ADU terms for offset deduced bad pixels. Default: [4, 20] --thresholds-noise-hard-mg int [int ...] Medium-gain thresholds in absolute ADU terms for offset deduced bad pixels. Default: [4, 20] --thresholds-noise-hard-lg int [int ...] Low-gain thresholds in absolute ADU terms for offset deduced bad pixels. Default: [4, 20] --thresholds-gain-sigma float Gain separation sigma threshold. Default: 5.0 --max-trains int Maximum number of trains to use for processing dark. Set to 0 to process all available trains. 550 added for ~500GB nodes to temporarely avoid memory issues.. Default: 550 --min-trains int Miniumum number of trains for processing dark. If run folder has less than minimum trains, processing is stopped.. Default: 1 --high-res-badpix-3d set this to True if you need high-resolution 3d bad pixel plots. ~7mins extra time for 64 memory cells. Default: False --no-high-res-badpix-3d Opposite of --high-res-badpix-3d (default: False) required arguments: --in-folder str path to input data, required. Default: None --out-folder str path to output to, required. Default: None --run-high int run number in which high gain data was recorded, required. Default: None --run-med int run number in which medium gain data was recorded, required. Default: None --run-low int run number in which low gain data was recorded, required. Default: None Gain Characterization +++++++++++++++++++++ To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate AGIPD FF --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--hist-file-template str] [--modules str [str ...]] [--raw-folder str] [--proc-folder str] [--run int] [--karabo-id str] [--karabo-da str [str ...]] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-idx str] [--h5path-ctrl str] [--karabo-id-control str] [--karabo-da-control str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--local-output | --no-local-output] [--db-output | --no-db-output] [--peak-range int [int ...]] [--peak-width-range int [int ...]] [--peak-norm-range float [float ...]] [--peak-lim int [int ...]] [--d0-lim int [int ...]] [--peak-width-lim float [float ...]] [--chi2-lim int [int ...]] [--intensity-lim int] [--gain-lim float [float ...]] [--cell-range int [int ...]] [--pixel-range int [int ...]] [--max-bins int] [--batch-size int [int ...]] [--fit-range int [int ...]] [--n-peaks-fit int] [--fix-peaks | --no-fix-peaks] [--do-minos | --no-do-minos] [--sigma-limit float] [--mem-cells int] [--bias-voltage int] [--acq-rate float] [--gain-setting int] [--photon-energy float] [--integration-time int] AGIPD FF # Gain Characterization # positional arguments: AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS FF Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --hist-file-template str the template to use to access histograms. Default: hists_m{:02d}_sum.h5 --modules str [str ...] modules to correct, set to -1 for all, range allowed. Default: [10] --raw-folder str Path to raw image data used to create histograms. Default: /gpfs/exfel/exp/MID/202030/p900137/raw --proc-folder str Path to corrected image data used to create histograms. Default: --run int of the run of image data used to create histograms. Default: 449 --karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1 --karabo-da str [str ...] a list of data aggregators names, Default [-1] for selecting all data aggregators. Default: ['-1'] --receiver-id str inset for receiver devices. Default: {}CH0 --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{:05d}.h5 --h5path str path in the HDF5 file to images. Default: INSTRUMENT/{}/DET/{}:xtdf/ --h5path-idx str path in the HDF5 file to images. Default: INDEX/{}/DET/{}:xtdf/ --h5path-ctrl str path to control information. Default: /CONTROL/{}/MDL/FPGA_COMP --karabo-id-control str karabo-id for control device. Default: MID_IRU_AGIPD1M1 --karabo-da-control str karabo DA for control infromation. Default: AGIPD1MCTRL00 --use-dir-creation-date use the creation data of the input dir for database queries. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8015#8045 --cal-db-timeout int in milli seconds. Default: 30000 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --peak-range int [int ...] where to look for the peaks, [a0, b0, a1, b1, ...] exactly 8 elements. Default: [-30, 30, 35, 70, 95, 135, 145, 220] --peak-width-range int [int ...] fit limits on the peak widths, [a0, b0, a1, b1, ...] exactly 8 elements. Default: [0, 30, 0, 35, 0, 40, 0, 45] --peak-norm-range float [float ...] . Default: [0.0, -1, 0, -1, 0, -1, 0, -1] --peak-lim int [int ...] Limit of position of noise peak. Default: [-30, 30] --d0-lim int [int ...] hard limits for distance between noise and first peak. Default: [10, 80] --peak-width-lim float [float ...] hard limits on the peak widths for first and second peak, in units of the noise peak. 4 parameters.. Default: [0.9, 1.55, 0.95, 1.65] --chi2-lim int [int ...] Hard limit on chi2/nDOF value. Default: [0, 3.0] --intensity-lim int Threshold on standard deviation of a histogram in ADU. Contribute to BadPixel bit "No_Entry". Default: 15 --gain-lim float [float ...] Threshold on gain in relative number. Contribute to BadPixel bit "Gain_deviation". Default: [0.8, 1.2] --cell-range int [int ...] range of cell to be considered, [0,0] for all. Default: [1, 3] --pixel-range int [int ...] range of pixels x1,y1,x2,y2 to consider [0,0,512,128] for all. Default: [0, 0, 32, 32] --max-bins int Maximum number of bins to consider, 0 for all bins. Default: 0 --batch-size int [int ...] batch size: [cell,x,y]. Default: [1, 8, 8] --fit-range int [int ...] range of a histogram considered for fitting in ADU. Dynamically evaluated in case [0,0]. Default: [0, 0] --n-peaks-fit int Number of gaussian peaks to fit including noise peak. Default: 4 --fix-peaks Fix distance between photon peaks. Default: False --no-fix-peaks Opposite of --fix-peaks (default: False) --do-minos This is additional feature of minuit to evaluate errors.. Default: False --no-do-minos Opposite of --do-minos (default: False) --sigma-limit float If >0, repeat fit keeping only bins within mu +- sigma_limit*sigma. Default: 0.0 --mem-cells int number of memory cells used, negative values for auto- detection.. Default: -1 --bias-voltage int Bias voltage.. Default: 300 --acq-rate float the detector acquisition rate, use 0 to try to auto- determine.. Default: 0.0 --gain-setting int the gain setting, negative values for auto-detection.. Default: -1 --photon-energy float photon energy in keV.. Default: 8.05 --integration-time int integration time, negative values for auto-detection.. Default: -1 required arguments: --in-folder str the folder to read histograms from, required. Default: None --out-folder str the folder to output to, required. Default: None Histogramming of AGIPD FF data ++++++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 1.0 Offline Calibration for the AGIPD Detector To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate AGIPD FF_HISTS --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--sequences str [str ...]] [--modules str [str ...]] --runs int [int ...] [--cells-list str [str ...]] [--karabo-id str] [--karabo-da str [str ...]] [--receiver-id str] [--path-template str] [--h5path str] [--n-bins-adu int] [--h-range int [int ...]] [--n-cells int] [--hist-std-tresh int] [--hist-mean-tresh int] [--h-sums-tresh int] [--n-cores-hists int] [--n-cores-files int] AGIPD FF_HISTS # Histogramming of AGIPD FF data , Author: European XFEL Detector Group, Version: 1.0, , Offline Calibration for the AGIPD Detector positional arguments: AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS FF_HISTS Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequences str [str ...] module to consider, set to -1 for all, range allowed. Default: [-1] --modules str [str ...] module to consider, range allowed. Default: [5] --cells-list str [str ...] list of lists or any python expressions, which can be converted to a list. E.g. 'range(0,15,5)' 'list(range(0,5))+list(range(50,60))'. Default: ['range(0,15)'] --karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1 --karabo-da str [str ...] a list of data aggregators names, Default [-1] for selecting all data aggregators. Default: ['-1'] --receiver-id str inset for receiver devices. Default: {}CH0 --path-template str the template to use to access data. Default: CORR-R{:04d}-AGIPD{:02d}-S{}.h5 --h5path str path in the HDF5 file to images. Default: INSTRUMENT/{}/DET/{}:xtdf/image/ --n-bins-adu int number of bins per ADU. Default: 1 --h-range int [int ...] range of the histogram in ADU. Default: [-50, 450] --n-cells int total number of memory cells (used to create summary file). Default: 202 --hist-std-tresh int Threshold for histogram standard deviation. Default: 10 --hist-mean-tresh int Threshold for histogram mean. Default: 15 --h-sums-tresh int Threshold for number of entries in histograms. Default: 100 --n-cores-hists int Number of processes (cores) to create histograms. Default: 20 --n-cores-files int Number of processes (cores) to read files. Default: 10 required arguments: --in-folder str the folder to read data from, required. Default: None --out-folder str the folder to output to, required. Default: None --runs int [int ...] list of run numbers, required. Default: None Characterize AGIPD Pulse Capacitor Data +++++++++++++++++++++++++++++++++++++++ Author: Detector Operations Group, Version 1.0 The following code characterizes AGIPD gain via data take with the pulse capacitor source (PCS). The PCS allows scanning through the high and medium gains of AGIPD, by subsequently intecreasing the number of charge pulses from a on-ASIC capicitor, thus increasing the charge a pixel sees in a given integration time. Because induced charge does not originate from X-rays on the sensor, the gains evaluated here will later need to be rescaled with gains deduced from X-ray data. PCS data is organized into multiple runs, as the on-ASIC current source cannot supply all pixels of a given module with charge at the same time. Hence, only certain pixel rows will have seen charge for a given image. These rows then first need to be combined into single module images again. This script uses new style of merging, which does not support old data format. We then use a K-means clustering algorithm to identify components in the resulting per-pixel data series, matching to three general regions: - a high gain slope - a transition region, where gain switching occurs - a medium gain slope. The same regions are present in the gain-bit data and are used to deduce the switching threshold. The resulting slopes are then fitted with a linear function and a combination of a linear and exponential decay function to determine the relative gains of the pixels with respect to the module. Additionally, we deduce masks for bad pixels form the data. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate AGIPD PC --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str --runs str [str ...] [--n-sequences int] [--modules str [str ...]] [--karabo-da str [str ...]] [--karabo-da-control str] [--karabo-id-control str] [--karabo-id str] [--ctrl-source-template str] [--instrument-source-template str] [--receiver-template str] [--use-dir-creation-date | --no-use-dir-creation-date] [--delta-time int] [--cal-db-interface str] [--local-output | --no-local-output] [--db-output | --no-db-output] [--bias-voltage int] [--mem-cells int] [--acq-rate float] [--gain-setting int] [--integration-time int] [--FF-gain float] [--fit-hook | --no-fit-hook] [--high-res-badpix-3d | --no-high-res-badpix-3d] [--hg-range int [int ...]] [--mg-range int [int ...]] [--sigma-dev-cut float] AGIPD PC # Characterize AGIPD Pulse Capacitor Data # , Author: Detector Operations Group, Version 1.0, , The following code characterizes AGIPD gain via data take with the pulse capacitor source (PCS). The PCS allows scanning through the high and medium gains of AGIPD, by subsequently intecreasing the number of charge pulses from a on-ASIC capicitor, thus increasing the charge a pixel sees in a given integration time., , Because induced charge does not originate from X-rays on the sensor, the gains evaluated here will later need to be rescaled with gains deduced from X-ray data., , PCS data is organized into multiple runs, as the on-ASIC current source cannot supply all pixels of a given module with charge at the same time. Hence, only certain pixel rows will have seen charge for a given image. These rows then first need to be combined into single module images again. This script uses new style of merging, which does not support old data format., , We then use a K-means clustering algorithm to identify components in the resulting per-pixel data series, matching to three general regions:, , * a high gain slope, * a transition region, where gain switching occurs, * a medium gain slope., , The same regions are present in the gain-bit data and are used to deduce the switching threshold. , The resulting slopes are then fitted with a linear function and a combination of a linear and exponential decay function to determine the relative gains of the pixels with respect to the module. Additionally, we deduce masks for bad pixels form the data. positional arguments: AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS PC Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str The ipcluster profile to use. Default: noDB --n-sequences int number of sequence files, starting for 0 to evaluate. Default: 5 --modules str [str ...] modules to work on, required, range allowed. Default: [-1] --karabo-da str [str ...] Default: ['all'] --karabo-da-control str karabo DA for control infromation. Default: AGIPD1MCTRL00 --karabo-id-control str karabo-id for the control device e.g. "MID_EXP_AGIPD1M1", or "SPB_IRU_AGIPD1M1". Default: SPB_IRU_AGIPD1M1 --karabo-id str Default: SPB_DET_AGIPD1M-1 --ctrl-source-template str path to control information. Default: {}/MDL/FPGA_COMP --instrument-source-template str path in the HDF5 file to images. Default: {}/DET/{}:xtdf --receiver-template str inset for receiver devices. Default: {}CH0 --use-dir-creation-date Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --delta-time int offset to the creation time (e.g. useful in case we want to force the system to use diff. dark constants). Default: 0 --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8019 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --bias-voltage int detector bias voltage, negative values for auto- detection.. Default: -1 --mem-cells int number of memory cells used, negative values for auto- detection.. Default: -1 --acq-rate float the detector acquisition rate, negative values for auto-detection.. Default: -1.0 --gain-setting int gain setting can have value 0 or 1, negative values for auto-detection.. Default: -1 --integration-time int integration time, negative values for auto-detection.. Default: -1 --FF-gain float ADU/keV, gain from FF to convert ADU to keV. Default: 7.3 --fit-hook fit a hook function to medium gain slope --> run without hook. Default: False --no-fit-hook Opposite of --fit-hook (default: False) --high-res-badpix-3d set this to True if you need high-resolution 3d bad pixel plots. Runtime: ~ 1h. Default: False --no-high-res-badpix-3d Opposite of --high-res-badpix-3d (default: False) --hg-range int [int ...] [0,-150] range for linear fit. If upper edge is negative use clustering result (bound_hg - 20). Default: [30, 210] --mg-range int [int ...] [-350,600] range for linear fit. If lower edge is negative use clustering result (bound_mg + 20). Default: [-277, 600] --sigma-dev-cut float parameters outside the range median +- sigma_dev_cut*MAD are replaced with the median. Default: 5.0 required arguments: --in-folder str path to input data, required. Default: None --out-folder str path to output to, required. Default: None --runs str [str ...] runs to use, required, range allowed. Default: None DSSC ---- DSSC Offline Correction +++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 1.0 Offline Calibration for the DSSC Detector To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate DSSC CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--sequences int [int ...]] [--modules str [str ...]] --run int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-idx str] [--slow-data-pattern str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--mem-cells int] [--overwrite | --no-overwrite] [--max-pulses int] [--bias-voltage int] [--sequences-per-node int] [--chunk-size-idim int] [--mask-noisy-asic float] [--mask-cold-asic float] [--noisy-pix-threshold float] [--geo-file str] [--dinstance str] [--slow-data-aggregators int [int ...]] [--slow-data-path str] DSSC CORRECT # DSSC Offline Correction # , Author: European XFEL Detector Group, Version: 1.0, , Offline Calibration for the DSSC Detector positional arguments: DSSC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str The ipcluster profile to use. Default: noDB --sequences int [int ...] sequence files to evaluate.. Default: [-1] --modules str [str ...] modules to correct, set to -1 for all, range allowed. Default: [-1] --karabo-id str karabo karabo_id. Default: SQS_DET_DSSC1M-1 --karabo-da str [str ...] a list of data aggregators names, Default [-1] for selecting all data aggregators. Default: ['-1'] --receiver-id str inset for receiver devices. Default: {}CH0 --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{:05d}.h5 --h5path str path in the HDF5 file to images. Default: INSTRUMENT/{}/DET/{}:xtdf/image --h5path-idx str path in the HDF5 file to images. Default: /INDEX/{}/DET/{}:xtdf/image --slow-data-pattern str Default: RAW-R{}-DA{}-S00000.h5 --use-dir-creation-date use the creation data of the input dir for database queries. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8020#8025 --cal-db-timeout int in milli seconds. Default: 300000 --mem-cells int number of memory cells used, set to 0 to automatically infer. Default: 0 --overwrite set to True if existing data should be overwritten. Default: True --no-overwrite Opposite of --overwrite (default: True) --max-pulses int maximum number of pulses per train. Default: 800 --bias-voltage int detector bias voltage. Default: 100 --sequences-per-node int number of sequence files per cluster node if run as slurm job, set to 0 to not run SLURM parallel. Default: 1 --chunk-size-idim int chunking size of imaging dimension, adjust if user software is sensitive to this.. Default: 1 --mask-noisy-asic float set to a value other than 0 and below 1 to mask entire ADC if fraction of noisy pixels is above. Default: 0.25 --mask-cold-asic float mask cold ASICS if number of pixels with negligable standard deviation is larger than this fraction. Default: 0.25 --noisy-pix-threshold float threshold above which ap pixel is considered noisy.. Default: 1.0 --geo-file str detector geometry file. Default: /gpfs/exfel/data/scratch/xcal/dssc_geo_june19.h5 --dinstance str Default: DSSC1M1 --slow-data-aggregators int [int ...] quadrant/aggregator. Default: [1, 2, 3, 4] --slow-data-path str Default: SQS_NQS_DSSC/FPGA/PPT_Q required arguments: --in-folder str path to input data, required. Default: None --out-folder str path to output to, required. Default: None --run int runs to process, required. Default: None DSSC Characterize Dark Images +++++++++++++++++++++++++++++ Author: S. Hauf, Version: 0.1 The following code analyzes a set of dark images taken with the DSSC detector to deduce detector offsets and noise. Data for the detector is presented in one run and don't acquire multiple gain stages. The notebook explicitely does what pyDetLib provides in its offset calculation method for streaming data. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate DSSC DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--sequences int [int ...]] [--modules int [int ...]] --run int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-idx str] [--slow-data-pattern str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--local-output | --no-local-output] [--db-output | --no-db-output] [--mem-cells int] [--bias-voltage int] [--rawversion int] [--thresholds-offset-sigma float] [--thresholds-offset-hard int [int ...]] [--thresholds-noise-sigma float] [--thresholds-noise-hard float [float ...]] [--offset-numpy-algorithm str] [--instrument str] [--high-res-badpix-3d | --no-high-res-badpix-3d] [--slow-data-aggregators int [int ...]] [--slow-data-path str] [--operation-mode str] DSSC DARK # DSSC Characterize Dark Images # , Author: S. Hauf, Version: 0.1, , The following code analyzes a set of dark images taken with the DSSC detector to deduce detector offsets and noise. Data for the detector is presented in "one run and dont acquire multiple gain stages. ", , The notebook explicitely does what pyDetLib provides in its offset calculation method for streaming data. positional arguments: DSSC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str The ipcluster profile to use. Default: noDB --sequences int [int ...] sequence files to evaluate.. Default: [0] --modules int [int ...] modules to run for. Default: [-1] --karabo-id str karabo karabo_id. Default: SQS_DET_DSSC1M-1 --karabo-da str [str ...] a list of data aggregators names, Default [-1] for selecting all data aggregators. Default: ['-1'] --receiver-id str inset for receiver devices. Default: {}CH0 --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{:05d}.h5 --h5path str path in the HDF5 file to images. Default: /INSTRUMENT/{}/DET/{}:xtdf/image --h5path-idx str path in the HDF5 file to images. Default: /INDEX/{}/DET/{}:xtdf/image --slow-data-pattern str Default: RAW-R{}-DA{}-S00000.h5 --use-dir-creation-date use the dir creation date for determining the creation time. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8020 --cal-db-timeout int timeout on caldb requests". Default: 3000000 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --mem-cells int number of memory cells used, set to 0 to automatically infer. Default: 0 --bias-voltage int detector bias voltage. Default: 100 --rawversion int RAW file format version. Default: 2 --thresholds-offset-sigma float thresholds in terms of n sigma noise for offset deduced bad pixels. Default: 3.0 --thresholds-offset-hard int [int ...] thresholds in absolute ADU terms for offset deduced bad pixels,. Default: [4, 125] --thresholds-noise-sigma float thresholds in terms of n sigma noise for offset deduced bad pixels. Default: 3.0 --thresholds-noise-hard float [float ...] thresholds in absolute ADU terms for offset deduced bad pixels. Default: [0.001, 3] --offset-numpy-algorithm str Default: mean --instrument str the instrument. Default: SQS --high-res-badpix-3d set this to True if you need high-resolution 3d bad pixel plots. Runtime: ~ 1h. Default: False --no-high-res-badpix-3d Opposite of --high-res-badpix-3d (default: False) --slow-data-aggregators int [int ...] quadrant/aggregator. Default: [1, 2, 3, 4] --slow-data-path str Default: SQS_NQS_DSSC/FPGA/PPT_Q --operation-mode str Detector operation mode, optional. Default: required arguments: --in-folder str path to input data, required. Default: None --out-folder str path to output to, required. Default: None --run int run number in which data was recorded, required. Default: None EPIX100 ------- ePix100 Data Correction +++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 2.0 The following notebook provides data correction of images acquired with the ePix100 detector. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate EPIX100 CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--sequences str [str ...]] [--sequences-per-node int] --run int [--karabo-id str] [--karabo-da str] [--db-module str] [--receiver-template str] [--path-template str] [--instrument-source-template str] [--chunk-size-idim int] [--limit-images int] [--cal-db-interface str] [--cal-db-timeout int] [--creation-time str] [--bias-voltage int] [--in-vacuum | --no-in-vacuum] [--integration-time int] [--fix-temperature int] [--gain-photon-energy float] [--photon-energy float] [--pattern-classification | --no-pattern-classification] [--relative-gain | --no-relative-gain] [--absolute-gain | --no-absolute-gain] [--common-mode | --no-common-mode] [--cm-min-frac float] [--cm-noise-sigma float] [--split-evt-primary-threshold float] [--split-evt-secondary-threshold float] [--split-evt-mip-threshold float] EPIX100 CORRECT # ePix100 Data Correction , Author: European XFEL Detector Group, Version: 2.0, , The following notebook provides data correction of images acquired with the ePix100 detector. positional arguments: EPIX100 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequences str [str ...] sequences to correct, set to -1 for all, range allowed. Default: [-1] --sequences-per-node int number of sequence files per cluster node if run as slurm job, set to 0 to not run SLURM parallel. Default: 1 --karabo-id str karabo karabo_id. Default: MID_EXP_EPIX-1 --karabo-da str data aggregators. Default: EPIX01 --db-module str module id in the database. Default: --receiver-template str detector receiver template for accessing raw data files. Default: RECEIVER --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --instrument-source-template str instrument detector data source in h5files. Default: {}/DET/{}:daqOutput --chunk-size-idim int H5 chunking size of output data. Default: 1 --limit-images int ONLY FOR TESTING. process only first N images, 0 - process all.. Default: 0 --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8015#8025 --cal-db-timeout int timeout on caldb requests. Default: 300000 --creation-time str The timestamp to use with Calibration DBe. Required Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04 11:02:41. Default: --bias-voltage int bias voltage. Default: 200 --in-vacuum detector operated in vacuum. Default: False --no-in-vacuum Opposite of --in-vacuum (default: False) --integration-time int Detector integration time, Default value -1 to use the value from the slow data.. Default: -1 --fix-temperature int fixed temperature value in Kelvin, Default value -1 to use the value from files.. Default: -1 --gain-photon-energy float Photon energy used for gain calibration. Default: 9.0 --photon-energy float Photon energy to calibrate in number of photons, 0 for calibration in keV. Default: 0.0 --pattern-classification do clustering.. Default: True --no-pattern-classification Opposite of --pattern-classification (default: True) --relative-gain Apply relative gain correction.. Default: True --no-relative-gain Opposite of --relative-gain (default: True) --absolute-gain Apply absolute gain correction (implies relative gain).. Default: True --no-absolute-gain Opposite of --absolute-gain (default: True) --common-mode Apply common mode correction.. Default: True --no-common-mode Opposite of --common-mode (default: True) --cm-min-frac float No CM correction is performed if after masking the ratio of good pixels falls below this. Default: 0.25 --cm-noise-sigma float CM correction noise standard deviation. Default: 5.0 --split-evt-primary-threshold float primary threshold for split event correction. Default: 7.0 --split-evt-secondary-threshold float secondary threshold for split event correction. Default: 5.0 --split-evt-mip-threshold float minimum ionizing particle threshold. Default: 1000.0 required arguments: --in-folder str input folder, required. Default: None --out-folder str output folder, required. Default: None --run int which run to read data from, required. Default: None ePix100 Dark Characterization +++++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 2.0 The following notebook provides dark image analysis and calibration constants of the ePix100 detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Noise and bad pixels maps are calculated independently for each of the 4 ASICs of ePix100, since their noise behaviour can be significantly different. Common mode correction can be applied to increase sensitivity to noise related bad pixels. Common mode correction is achieved by subtracting the median of all pixels that are read out at the same time along a row/column. This is done in an iterative process, by which a new bad pixels map is calculated and used to mask data as the common mode values across the rows/columns is updated. Resulting maps are saved as .h5 files for a later use and injected to calibration DB. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate EPIX100 DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--sequence int] --run int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-template str] [--path-template str] [--instrument-source-template str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--db-output | --no-db-output] [--local-output | --no-local-output] [--bias-voltage int] [--in-vacuum | --no-in-vacuum] [--fix-integration-time int] [--fix-temperature int] [--temp-limits int] [--badpixel-threshold-sigma float] [--CM-N-iterations int] [--min-trains int] [--max-trains int] [--operation-mode str] [--db-module str] EPIX100 DARK # ePix100 Dark Characterization , Author: European XFEL Detector Group, Version: 2.0, , The following notebook provides dark image analysis and calibration constants of the ePix100 detector., , Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. , Noise and bad pixels maps are calculated independently for each of the 4 ASICs of ePix100, since their noise behaviour can be significantly different., , Common mode correction can be applied to increase sensitivity to noise related bad pixels. Common mode correction is achieved by subtracting the median of all pixels that are read out at the same time along a row/column. This is done in an iterative process, by which a new bad pixels map is calculated and used to mask data as the common mode values across the rows/columns is updated., , Resulting maps are saved as .h5 files for a later use and injected to calibration DB. positional arguments: EPIX100 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequence int sequence file to use. Default: 0 --karabo-id str karabo karabo_id. Default: HED_IA1_EPX100-1 --karabo-da str [str ...] data aggregators. Default: ['EPIX01'] --receiver-template str detector receiver template for accessing raw data files. Default: RECEIVER --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --instrument-source-template str instrument detector data source in h5files. Default: {}/DET/{}:daqOutput --use-dir-creation-date Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8020 --cal-db-timeout int timeout on caldb requests. Default: 300000 --db-output Output constants to the calibration database. Default: False --no-db-output Opposite of --db-output (default: False) --local-output Output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --bias-voltage int Bias voltage. Default: 200 --in-vacuum Detector operated in vacuum. Default: False --no-in-vacuum Opposite of --in-vacuum (default: False) --fix-integration-time int Integration time. Set to -1 to read from .h5 file. Default: -1 --fix-temperature int Fixed temperature in Kelvin. Set to -1 to read from .h5 file. Default: -1 --temp-limits int Limit for parameter Operational temperature. Default: 5 --badpixel-threshold-sigma float Bad pixels defined by values outside n times this std from median. Default: 5. Default: 5.0 --CM-N-iterations int Number of iterations for common mode correction. Set to 0 to skip it. Default: 2 --min-trains int Minimum number of trains that should be available to process dark constants. Default 1.. Default: 1 --max-trains int Maximum number of trains to use for processing dark constants. Set to 0 to use all available trains.. Default: 1000 --operation-mode str Detector operation mode, optional. Default: --db-module str ID of module in calibration database, this parameter is ignore in the notebook. TODO: remove from calibration_configurations.. Default: required arguments: --in-folder str input folder, required. Default: None --out-folder str output folder, required. Default: None --run int which run to read data from, required. Default: None EPIX10K ------- ePIX10K Data Correction +++++++++++++++++++++++ Authors: M. Karnevskiy, S. Hauf, Version 1.0 The following notebook provides Offset correction of images acquired with the ePix10K detector. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate EPIX10K CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--sequences str [str ...]] --run int [--karabo-id str] [--karabo-da str] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-t str] [--h5path-cntrl str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--cpuCores int] [--chunk-size-idim int] [--overwrite | --no-overwrite] [--limit-images int] [--db-module str] [--sequences-per-node int] [--bias-voltage int] [--in-vacuum | --no-in-vacuum] [--fix-temperature float] [--split-evt-primary-threshold float] [--split-evt-secondary-threshold float] [--split-evt-mip-threshold float] EPIX10K CORRECT # ePIX10K Data Correction # , Authors: M. Karnevskiy, S. Hauf, Version 1.0, , The following notebook provides Offset correction of images acquired with the ePix10K detector. positional arguments: EPIX10K The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str ipcluster profile to use. Default: noDB --sequences str [str ...] sequences to correct, set to -1 for all, range allowed. Default: [-1] --karabo-id str karabo karabo_id. Default: HED_IA1_EPIX10K-1 --karabo-da str data aggregators. Default: EPIX03 --receiver-id str inset for receiver devices. Default: RECEIVER --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --h5path str path in the HDF5 file to images. Default: /INSTRUMENT/{}/DET/{}:daqOutput/data/image --h5path-t str path to find temperature at. Default: /INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp --h5path-cntrl str path to control data. Default: /CONTROL/{}/DET --use-dir-creation-date date constants injected before directory creation time. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8015#8025 --cal-db-timeout int timeout on caldb requests. Default: 30000000 --cpuCores int Specifies the number of running cpu cores. Default: 4 --chunk-size-idim int H5 chunking size of output data. Default: 1 --overwrite overwrite output folder. Default: True --no-overwrite Opposite of --overwrite (default: True) --limit-images int process only first N images, 0 - process all. Default: 0 --db-module str module id in the database. Default: ePix10K_M43 --sequences-per-node int number of sequence files per cluster node if run as slurm job, set to 0 to not run SLURM parallel. Default: 1 --bias-voltage int bias voltage. Default: 200 --in-vacuum detector operated in vacuum. Default: False --no-in-vacuum Opposite of --in-vacuum (default: False) --fix-temperature float fix temperature to this value. Default: 290.0 --split-evt-primary-threshold float primary threshold for split event correction. Default: 7.0 --split-evt-secondary-threshold float secondary threshold for split event correction. Default: 5.0 --split-evt-mip-threshold float minimum ionizing particle threshold. Default: 1000.0 required arguments: --in-folder str input folder, required. Default: None --out-folder str output folder, required. Default: None --run int which run to read data from, required. Default: None ePix10K Dark Characterization +++++++++++++++++++++++++++++ Author: M. Karnevskiy, Version 1.0 The following notebook provides dark image analysis of the ePix10K detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Resulting maps are saved as .h5 files for a latter use and injected to the calibration DB. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate EPIX10K DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--sequence int] --run int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-t str] [--h5path-cntrl str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--db-output | --no-db-output] [--local-output | --no-local-output] [--temp-limits int] [--number-dark-frames int] [--db-module str] [--bias-voltage int] [--in-vacuum | --no-in-vacuum] [--fix-temperature float] [--operation-mode str] EPIX10K DARK # ePix10K Dark Characterization # , Author: M. Karnevskiy, Version 1.0, , The following notebook provides dark image analysis of the ePix10K detector., , Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Resulting maps are saved as .h5 files for a latter use and injected to the calibration DB. positional arguments: EPIX10K The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str ipcluster profile to use. Default: noDB --sequence int sequence file to use. Default: 0 --karabo-id str karabo karabo_id. Default: HED_IA1_EPIX10K-1 --karabo-da str [str ...] data aggregators. Default: ['EPIX03'] --receiver-id str inset for receiver devices. Default: RECEIVER --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --h5path str path in the HDF5 file to images. Default: /INSTRUMENT/{}/DET/{}:daqOutput/data/image/pixels --h5path-t str path to find temperature at. Default: /INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp --h5path-cntrl str path to control data. Default: /CONTROL/{}/DET --use-dir-creation-date Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8020 --cal-db-timeout int timeout on caldb requests. Default: 300000 --db-output Output constants to the calibration database. Default: False --no-db-output Opposite of --db-output (default: False) --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --temp-limits int limit for parameter Operational temperature. Default: 5 --number-dark-frames int number of images to be used, if set to 0 all available images are used. Default: 0 --db-module str detector karabo_id. Default: ePix10K_M43 --bias-voltage int bias voltage. Default: 200 --in-vacuum detector operated in vacuum. Default: False --no-in-vacuum Opposite of --in-vacuum (default: False) --fix-temperature float fix temperature to this value. Default: 290.0 --operation-mode str Detector operation mode, optional. Default: required arguments: --in-folder str input folder, required. Default: None --out-folder str output folder, required. Default: None --run int which run to read data from, required. Default: None FASTCCD ------- FastCCD Data Correction +++++++++++++++++++++++ Authors: I. Klačková, S. Hauf, Version 1.0 The following notebook provides correction of images acquired with the FastCCD. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate FASTCCD CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--sequences str [str ...]] [--run int] [--karabo-da str] [--karabo-id str] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-t str] [--h5path-cntrl str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--cpuCores int] [--operation-mode str] [--split-evt-primary-threshold float] [--split-evt-secondary-threshold float] [--split-evt-mip-threshold float] [--chunk-size-idim int] [--overwrite | --no-overwrite] [--sequences-per-node int] [--limit-images int] [--time-offset-days int] [--photon-energy-gain-map float] [--fix-temperature float] [--flipped-between str [str ...]] [--temp-limits int] [--commonModeAxis int] [--only-offset | --no-only-offset] [--cti-corr | --no-cti-corr] [--relgain-corr | --no-relgain-corr] [--common-mode-corr | --no-common-mode-corr] [--correct-offset-drift | --no-correct-offset-drift] [--do-pattern-classification | --no-do-pattern-classification] FASTCCD CORRECT # FastCCD Data Correction , Authors: I. Klačková, S. Hauf, Version 1.0, , The following notebook provides correction of images acquired with the FastCCD. positional arguments: FASTCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str ipcluster profile to use. Default: noDB --sequences str [str ...] sequences to correct, set to -1 for all, range allowed. Default: [-1] --run int run number. Default: 277 --karabo-da str data aggregators. Default: DA05 --karabo-id str karabo prefix of PNCCD devices. Default: SCS_CDIDET_FCCD2M --receiver-id str inset for receiver devices. Default: FCCD --path-template str path template in hdf5 file. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --h5path str path in HDF5 file. Default: /INSTRUMENT/{}/DAQ/{}:daqOutput/data/image --h5path-t str temperature path in HDF5 file. Default: /CONTROL/{}/CTRL/LSLAN/inputA/crdg/value --h5path-cntrl str path to control data. Default: /RUN/{}/DET/FCCD --use-dir-creation-date use dir creation data for calDB queries. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8015#8025 --cal-db-timeout int timeout on caldb requests. Default: 300000000 --cpuCores int Specifies the number of running cpu cores. Default: 16 --operation-mode str FS stands for frame-store and FF for full-frame opeartion. Default: FF --split-evt-primary-threshold float primary threshold for split event classification in terms of n sigma noise. Default: 7.0 --split-evt-secondary-threshold float secondary threshold for split event classification in terms of n sigma noise. Default: 4.0 --split-evt-mip-threshold float MIP threshold for event classification. Default: 1000.0 --chunk-size-idim int H5 chunking size of output data. Default: 1 --overwrite overwrite existing files. Default: True --no-overwrite Opposite of --overwrite (default: True) --sequences-per-node int sequences to correct per node. Default: 1 --limit-images int limit images per file. Default: 0 --time-offset-days int offset in days for calibration parameters. Default: 0 --photon-energy-gain-map float energy in keV. Default: 5.9 --fix-temperature float fix temperature to this value, set to 0 to use slow control value. Default: 0.0 --flipped-between str [str ...] detector was flipped during this timespan. Default: ['2019-02-01', '2019-04-02'] --temp-limits int limits within which temperature is considered the same. Default: 5 --commonModeAxis int Axis along which common mode will be calculated (0: along rows, 1: along columns). Default: 1 --only-offset Only apply offset correction. Default: False --no-only-offset Opposite of --only-offset (default: False) --cti-corr Apply CTI correction. Default: False --no-cti-corr Opposite of --cti-corr (default: False) --relgain-corr Apply relative gain correction. Default: True --no-relgain-corr Opposite of --relgain-corr (default: True) --common-mode-corr Apply commonMode correction. Default: False --no-common-mode-corr Opposite of --common-mode-corr (default: False) --correct-offset-drift correct for offset drifts. Default: False --no-correct-offset-drift Opposite of --correct-offset-drift (default: False) --do-pattern-classification classify split events. Default: True --no-do-pattern-classification Opposite of --do-pattern-classification (default: True) required arguments: --in-folder str input folder, required. Default: None --out-folder str output folder, required. Default: None FastCCD Dark Characterization +++++++++++++++++++++++++++++ Author: I. Klačková, S. Hauf, K. Setoodehnia and M. Cascella The following notebook provides dark image analysis of the FastCCD detector. Dark characterization evaluates offset and noise of the FastCCD detector, corrects the noise for Common Mode (CM), and defines bad pixels relative to offset and CM corrected noise. Bad pixels are then excluded and CM corrected noise is recalculated excluding the bad pixels. Resulting offset and CM corrected noise maps, as well as the bad pixel map are sent to the calibration database. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate FASTCCD DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--sequence int] --run int [--karabo-da str [str ...]] [--karabo-id str] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-t str] [--h5path-cntrl str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--db-output | --no-db-output] [--local-output | --no-local-output] [--number-dark-frames int] [--operation-mode str] [--temp-limits int] [--bad-pixel-offset-sigma float] [--bad-pixel-noise-sigma float] [--sigmaNoise float] [--fix-temperature float] [--chunkSize int] [--cpuCores int] [--commonModeAxis int] [--run-parallel | --no-run-parallel] [--ADU-to-electron-upper-hg float] [--ADU-to-electron-lower-hg float] [--ADU-to-electron-upper-mg float] [--ADU-to-electron-lower-mg float] [--ADU-to-electron-upper-lg float] [--ADU-to-electron-lower-lg float] FASTCCD DARK # FastCCD Dark Characterization , Author: I. Klačková, S. Hauf, K. Setoodehnia and M. Cascella, , The following notebook provides dark image analysis of the FastCCD detector., , Dark characterization evaluates offset and noise of the FastCCD detector, corrects the noise for Common Mode (CM), and defines bad pixels relative to offset and CM corrected noise. Bad pixels are then excluded and CM corrected noise is recalculated excluding the bad pixels. Resulting offset and CM corrected noise maps, as well as the bad pixel map are sent to the calibration database. positional arguments: FASTCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str ipcluster profile to use. Default: noDB --sequence int sequence file to use. Default: 0 --karabo-da str [str ...] data aggregators. Default: ['DA05'] --karabo-id str karabo prefix of PNCCD devices. Default: SCS_CDIDET_FCCD2M --receiver-id str inset for receiver devices. Default: FCCD --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --h5path str path to the data in the HDF5 file. Default: /INSTRUMENT/{}/DAQ/{}:daqOutput/data/image/pixels --h5path-t str path to find temperature. Default: /CONTROL/{}/CTRL/LSLAN/inputA/crdg/value --h5path-cntrl str path to find control data. Default: /RUN/{}/DET/FCCD --use-dir-creation-date To be used to retrieve calibration constants later on (for database time derivation). Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the calibration database interface to use. Default: tcp://max-exfl016:8020 --cal-db-timeout int timeout on calibration database requests. Default: 300000 --db-output Output constants to the calibration database. Default: False --no-db-output Opposite of --db-output (default: False) --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --number-dark-frames int number of images to be used, if set to 0 all available images are used. Default: 0 --operation-mode str FS stands for frame-store and FF for full-frame operation.. Default: FF --temp-limits int to find calibration constants later on, the sensor temperature is allowed to vary by 5 units. Default: 5 --bad-pixel-offset-sigma float Any pixel whose offset is beyond 5 standard deviations, is a bad pixel. Default: 5.0 --bad-pixel-noise-sigma float Any pixel whose noise is beyond 5 standard deviations, is a bad pixel. Default: 5.0 --sigmaNoise float Any pixel whose signal exceeds 'sigmaNoise'*noiseCM (common mode corrected noise) will be masked. Default: 5.0 --fix-temperature float Fixed operation temperature in Kelvins. If set to 0, mean value of the data file's temperature is. Default: 0.0 --chunkSize int Number of images to read per chunk. Default: 100 --cpuCores int Specifies the number of running cpu cores. Default: 40 --commonModeAxis int Axis along which common mode will be calculated (0: along rows, 1: along columns). Default: 1 --run-parallel For parallel computation. Default: True --no-run-parallel Opposite of --run-parallel (default: True) --ADU-to-electron-upper-hg float for upper hemisphere and high gain. Default: 6.3 --ADU-to-electron-lower-hg float for lower hemisphere and high gain. Default: 6.4 --ADU-to-electron-upper-mg float for upper hemisphere and medium gain. Default: 23.4 --ADU-to-electron-lower-mg float for lower hemisphere and medium gain. Default: 23.4 --ADU-to-electron-upper-lg float for upper hemisphere and low gain. Default: 49.3 --ADU-to-electron-lower-lg float for lower hemisphere and low gain. Default: 47.3 required arguments: --in-folder str input folder, required. Default: None --out-folder str output folder, required. Default: None --run int which run to read data from, required. Default: None GENERIC ------- Constants from DB to HDF5 +++++++++++++++++++++++++ Version 0.1, Author: S. Hauf Currently available instances are LPD1M1 and AGIPD1M1 To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate GENERIC DB_TO_H5 --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --detector-instance str --out-file str [--valid-at str] [--cal-db-interface str] [--modules str [str ...]] GENERIC DB_TO_H5 # Constants from DB to HDF5 # , Version 0.1, Author: S. Hauf, , Currently available instances are LPD1M1 and AGIPD1M1 positional arguments: GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DB_TO_H5 Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --valid-at str ISO formatted date for which constants shoudl be valid. Leave empty to get most current ones. Default: --cal-db-interface str Default: tcp://max-exfl015:5005 --modules str [str ...] modules to get data from, in terms of numerical quadrant indices, range allowed. Default: [-1] required arguments: --detector-instance str the detector instance to get constants for e.g. LPD1M1, required. Default: None --out-file str HDF5 file to output constants into, required. Default: None Statistical analysis of calibration factors +++++++++++++++++++++++++++++++++++++++++++ Author: Mikhail Karnevskiy, Version 0.2 Plot calibration constants retrieved from the cal. DB. To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant. Values shown in plots are saved in h5 files. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate GENERIC STATS_FROM_DB --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--start-date str] [--end-date str] [--dclass str] [--modules str [str ...]] [--submodules str [str ...]] [--constants str [str ...]] [--nconstants int] [--max-time int] [--nMemToShow int] [--gain-setting int [int ...]] [--bias-voltage int [int ...]] [--temperature int [int ...]] [--integration-time float [float ...]] [--pixels-x int [int ...]] [--pixels-y int [int ...]] [--in-vacuum int [int ...]] [--memory-cells int [int ...]] [--acquisition-rate float [float ...]] [--parameter-names str [str ...]] [--separate-plot str [str ...]] [--x-labels str [str ...]] [--photon-energy float] [--out-folder str] [--use-existing str] [--cal-db-interface str] [--cal-db-timeout int] [--plot-range int] [--spShape int [int ...]] [--sp-name str] [--gain-titles str [str ...]] GENERIC STATS_FROM_DB # Statistical analysis of calibration factors# , Author: Mikhail Karnevskiy, Version 0.2, , Plot calibration constants retrieved from the cal. DB., , To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant., , Values shown in plots are saved in h5 files. positional arguments: GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS STATS_FROM_DB Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --start-date str date to start investigation interval from. Default: 2019-01-01 --end-date str date to end investigation interval at, can be "now". Default: NOW --dclass str Detector class. Default: jungfrau --modules str [str ...] detector entry in the DB to investigate. Default: ['Jungfrau_M035'] --submodules str [str ...] module index of a modular detector (1 for Q1M1 of AGIPD), range allowed. Default: [2] --constants str [str ...] constants to plot. Default: ['RelativeGain'] --nconstants int Number of time stamps to plot. If not 0, overcome start_date.. Default: 20 --max-time int max time margin in minutes to match bad pixels. Default: 15 --nMemToShow int Number of memory cells to be shown in plots. Default: 16 --gain-setting int [int ...] gain stages. Default: [0, 1, 2] --bias-voltage int [int ...] Bias voltage. Default: [90, 180] --temperature int [int ...] Operation temperature. Default: [291] --integration-time float [float ...] Integration time. Default: [4.96, 10, 50, 250] --pixels-x int [int ...] number of pixels along X axis. Default: [1024] --pixels-y int [int ...] number of pixels along Y axis. Default: [512] --in-vacuum int [int ...] 0 if detector is operated in room pressure. Default: [0] --memory-cells int [int ...] number of memory cells. Default: [1, 16] --acquisition-rate float [float ...] aquisition rate. Default: [1.1] --parameter-names str [str ...] names of parameters. Default: ['bias_voltage', 'integration_time', 'pixels_x', 'pixels_y', 'temperature', 'memory_cells'] --separate-plot str [str ...] Plot on separate plots. Default: ['gain_setting', 'memory_cells', 'integration_time'] --x-labels str [str ...] parameters to be shown on X axis: Acquisition rate, Memory cells, Sensor Temperature, Integration Time. Default: ['Sensor Temperature', 'Integration Time'] --photon-energy float Photon energy of the beam. Default: 9.2 --out-folder str output folder. Default: /gpfs/exfel/data/scratch/karnem/test_bla4/ --use-existing str If not empty, constants stored in given folder will be used. Default: --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8016 --cal-db-timeout int timeout on caldb requests",. Default: 180000 --plot-range int range for plotting in units of median absolute deviations. Default: 3 --spShape int [int ...] Shape of superpixel. Default: [256, 64] --sp-name str name of superpixel. Default: ASIC IDs --gain-titles str [str ...] Title inset related to gain. Default: ['High gain', 'Medium gain', 'Low gain'] Statistical analysis of calibration factors +++++++++++++++++++++++++++++++++++++++++++ Author: Mikhail Karnevskiy, Version 0.2 Plot calibration constants retrieved from the cal. DB. To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant. Values shown in plots are saved in h5 files. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate GENERIC STATS_FROM_DB2 --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--start-date str] [--end-date str] [--dclass str] [--modules str [str ...]] [--submodules str [str ...]] [--constants str [str ...]] [--nconstants int] [--max-time int] [--nMemToShow int] [--gain-setting int [int ...]] [--bias-voltage int [int ...]] [--temperature int [int ...]] [--integration-time float [float ...]] [--pixels-x int [int ...]] [--pixels-y int [int ...]] [--in-vacuum int [int ...]] [--memory-cells int [int ...]] [--acquisition-rate float [float ...]] [--parameter-names str [str ...]] [--separate-plot str [str ...]] [--x-labels str [str ...]] [--photon-energy float] [--out-folder str] [--use-existing str] [--cal-db-interface str] [--cal-db-timeout int] [--plot-range int] [--spShape int [int ...]] [--sp-name str] [--gain-titles str [str ...]] GENERIC STATS_FROM_DB2 # Statistical analysis of calibration factors# , Author: Mikhail Karnevskiy, Version 0.2, , Plot calibration constants retrieved from the cal. DB., , To be visualized, calibration constants are averaged per group of pixels. Plots shows calibration constant over time for each constant., , Values shown in plots are saved in h5 files. positional arguments: GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS STATS_FROM_DB2 Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --start-date str date to start investigation interval from. Default: 2019-01-01 --end-date str date to end investigation interval at, can be "now". Default: NOW --dclass str Detector class. Default: jungfrau --modules str [str ...] detector entry in the DB to investigate. Default: ['Jungfrau_M035'] --submodules str [str ...] module index of a modular detector (1 for Q1M1 of AGIPD), range allowed. Default: [2] --constants str [str ...] constants to plot. Default: ['RelativeGain'] --nconstants int Number of time stamps to plot. If not 0, overcome start_date.. Default: 20 --max-time int max time margin in minutes to match bad pixels. Default: 15 --nMemToShow int Number of memory cells to be shown in plots. Default: 16 --gain-setting int [int ...] gain stages. Default: [0, 1, 2] --bias-voltage int [int ...] Bias voltage. Default: [90, 180] --temperature int [int ...] Operation temperature. Default: [291] --integration-time float [float ...] Integration time. Default: [4.96, 10, 50, 250] --pixels-x int [int ...] number of pixels along X axis. Default: [1024] --pixels-y int [int ...] number of pixels along Y axis. Default: [512] --in-vacuum int [int ...] 0 if detector is operated in room pressure. Default: [0] --memory-cells int [int ...] number of memory cells. Default: [1, 16] --acquisition-rate float [float ...] aquisition rate. Default: [1.1] --parameter-names str [str ...] names of parameters. Default: ['bias_voltage', 'integration_time', 'pixels_x', 'pixels_y', 'temperature', 'memory_cells'] --separate-plot str [str ...] Plot on separate plots. Default: ['gain_setting', 'memory_cells', 'integration_time'] --x-labels str [str ...] parameters to be shown on X axis: Acquisition rate, Memory cells, Sensor Temperature, Integration Time. Default: ['Sensor Temperature', 'Integration Time'] --photon-energy float Photon energy of the beam. Default: 9.2 --out-folder str output folder. Default: /gpfs/exfel/data/scratch/karnem/test_bla4/ --use-existing str If not empty, constants stored in given folder will be used. Default: --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8016 --cal-db-timeout int timeout on caldb requests",. Default: 180000 --plot-range int range for plotting in units of median absolute deviations. Default: 3 --spShape int [int ...] Shape of superpixel. Default: [256, 64] --sp-name str name of superpixel. Default: ASIC IDs --gain-titles str [str ...] Title inset related to gain. Default: ['High gain', 'Medium gain', 'Low gain'] GOTTHARD2 --------- Gotthard2 Offline Correction ++++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 1.0 Offline Calibration for the Gothard2 Detector To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate GOTTHARD2 CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str --run int [--sequences str [str ...]] [--sequences-per-node int] [--karabo-id str] [--karabo-da str [str ...]] [--receiver-template str] [--control-template str] [--instrument-source-template str] [--ctrl-source-template str] [--karabo-id-control str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--overwrite-creation-time str] [--constants-file str] [--offset-correction | --no-offset-correction] [--gain-correction | --no-gain-correction] [--bias-voltage int] [--exposure-time float] [--exposure-period float] [--acquisition-rate float] [--single-photon int] [--skip-plots | --no-skip-plots] [--pulse-idx-preview int] GOTTHARD2 CORRECT # Gotthard2 Offline Correction # , Author: European XFEL Detector Group, Version: 1.0, , Offline Calibration for the Gothard2 Detector positional arguments: GOTTHARD2 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequences str [str ...] sequences to correct, set to [-1] for all, range allowed. Default: [-1] --sequences-per-node int number of sequence files per node if notebook executed through xfel-calibrate, set to 0 to not run SLURM parallel. Default: 1 --karabo-id str karabo prefix of Gotthard-II devices. Default: FXE_XAD_G2XES --karabo-da str [str ...] data aggregators. Default: ['GH201'] --receiver-template str receiver template used to read INSTRUMENT keys.. Default: RECEIVER --control-template str control template used to read CONTROL keys.. Default: CONTROL --instrument-source-template str template for source name (filled with karabo_id & receiver_id). e.g. 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default: {}/DET/{}:daqOutput --ctrl-source-template str template for control source name (filled with karabo_id_control). Default: {}/DET/{} --karabo-id-control str Control karabo ID. Set to empty string to use the karabo-id. Default: --use-dir-creation-date use the creation data of the input dir for database queries.. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use.. Default: tcp://max- exfl016:8016#8025 --cal-db-timeout int timeout on caldb requests.. Default: 180000 --overwrite-creation-time str To overwrite the measured creation_time. Required Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28 13:00:00.00". Default: --constants-file str Use constants in given constant file path. /gpfs/exfel /data/scratch/ahmedk/dont_remove/gotthard2/constants/c alibration_constants_GH2.h5. Default: --offset-correction apply offset correction. This can be disabled to only apply LUT or apply LUT and gain correction for non- linear differential results.. Default: True --no-offset-correction Opposite of --offset-correction (default: True) --gain-correction apply gain correction.. Default: True --no-gain-correction Opposite of --gain-correction (default: True) --bias-voltage int Detector bias voltage, set to -1 to use value in raw file.. Default: -1 --exposure-time float Detector exposure time, set to -1 to use value in raw file.. Default: -1.0 --exposure-period float Detector exposure period, set to -1 to use value in raw file.. Default: -1.0 --acquisition-rate float Detector acquisition rate (1.1/4.5), set to -1 to use value in raw file.. Default: -1.0 --single-photon int Detector single photon mode (High/Low CDS), set to -1 to use value in raw file.. Default: -1 --skip-plots exit after writing corrected files. Default: False --no-skip-plots Opposite of --skip-plots (default: False) --pulse-idx-preview int pulse index to preview. The following even/odd pulse index is used for preview. TODO: update to pulseId preview.. Default: 3 required arguments: --in-folder str the folder to read data from, required. Default: None --out-folder str the folder to output to, required. Default: None --run int run to process, required. Default: None Gotthard2 Dark Image Characterization +++++++++++++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 1.0 The following is a processing for offset, noise, and Badpixels maps using dark images taken with Gotthard2 detector. All constants are evaluated per strip, per pulse, and per memory cell. The maps are calculated for each gain stage that is acquired in 3 separate runs. The three maps (calibration constants) can be injected to the database and stored locally. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate GOTTHARD2 DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str --run-high int --run-med int --run-low int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-template str] [--control-template str] [--instrument-source-template str] [--ctrl-source-template str] [--karabo-id-control str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--overwrite-creation-time str] [--db-output | --no-db-output] [--local-output | --no-local-output] [--bias-voltage int] [--exposure-time float] [--exposure-period float] [--acquisition-rate float] [--single-photon int] [--min-trains int] [--max-trains int] [--badpixel-threshold-sigma float] [--operation-mode str] GOTTHARD2 DARK # Gotthard2 Dark Image Characterization # , Author: European XFEL Detector Group, Version: 1.0, , The following is a processing for offset, noise, and Badpixels maps using dark images taken with Gotthard2 detector., All constants are evaluated per strip, per pulse, and per memory cell. The maps are calculated for each gain stage that is acquired in 3 separate runs., , The three maps (calibration constants) can be injected to the database and stored locally. positional arguments: GOTTHARD2 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --karabo-id str karabo prefix of Gotthard-II devices. Default: FXE_XAD_G2XES --karabo-da str [str ...] data aggregators. Default: ['GH201'] --receiver-template str receiver template used to read INSTRUMENT keys.. Default: RECEIVER --control-template str control template used to read CONTROL keys.. Default: CONTROL --instrument-source-template str template for source name (filled with karabo_id & receiver_id). e.g. 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput' noqa. Default: {}/DET/{}:daqOutput --ctrl-source-template str template for control source name (filled with karabo_id_control). Default: {}/DET/{} --karabo-id-control str Control karabo ID. Set to empty string to use the karabo-id. Default: --use-dir-creation-date Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8020 --cal-db-timeout int timeout on caldb requests. Default: 300000 --overwrite-creation-time str To overwrite the measured creation_time. Required Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28 13:00:00.00". Default: --db-output Output constants to the calibration database. Default: False --no-db-output Opposite of --db-output (default: False) --local-output Output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --bias-voltage int Detector bias voltage, set to -1 to use value in raw file.. Default: -1 --exposure-time float Detector exposure time, set to -1 to use value in raw file.. Default: -1.0 --exposure-period float Detector exposure period, set to -1 to use value in raw file.. Default: -1.0 --acquisition-rate float Detector acquisition rate (1.1/4.5), set to -1 to use value in raw file.. Default: -1.0 --single-photon int Detector single photon mode (High/Low CDS), set to -1 to use value in raw file.. Default: -1 --min-trains int Minimum number of trains that should be available to process dark constants. Default 1.. Default: 1 --max-trains int Maximum number of trains to use for processing dark constants. Set to 0 to use all available trains.. Default: 1000 --badpixel-threshold-sigma float bad pixels defined by values outside n times this std from median. Default: 5.0 --operation-mode str Detector dark run acquiring operation mode, optional. Default: required arguments: --in-folder str the folder to read data from, required. Default: None --out-folder str the folder to output to, required. Default: None --run-high int run number for G0 dark run, required. Default: None --run-med int run number for G1 dark run, required. Default: None --run-low int run number for G2 dark run, required. Default: None JUNGFRAU -------- Jungfrau Offline Correction +++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 2.0 Offline Calibration for the Jungfrau Detector To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate JUNGFRAU CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str --run int [--sequences str [str ...]] [--sequences-per-node int] [--karabo-id str] [--karabo-da str [str ...]] [--receiver-template str] [--instrument-source-template str] [--ctrl-source-template str] [--karabo-id-control str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--relative-gain | --no-relative-gain] [--strixel-sensor | --no-strixel-sensor] [--strixel-double-norm float] [--limit-trains int] [--chunks-ids int] [--chunks-data int] [--manual-slow-data | --no-manual-slow-data] [--integration-time float] [--gain-setting int] [--gain-mode int] [--mem-cells int] [--bias-voltage int] [--skip-plots | --no-skip-plots] [--plot-trains int] [--cell-id-preview int] [--roi-definitions int [int ...]] JUNGFRAU CORRECT # Jungfrau Offline Correction # , Author: European XFEL Detector Group, Version: 2.0, , Offline Calibration for the Jungfrau Detector positional arguments: JUNGFRAU The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequences str [str ...] sequences to correct, set to [-1] for all, range allowed. Default: [-1] --sequences-per-node int number of sequence files per cluster node if run as slurm job, set to 0 to not run SLURM parallel. Default: 1 --karabo-id str karabo prefix of Jungfrau devices. Default: SPB_IRDA_JF4M --karabo-da str [str ...] data aggregators. Default: ['JNGFR01', 'JNGFR02', 'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07', 'JNGFR08'] --receiver-template str Detector receiver template for accessing raw data files. e.g. "JNGFR{:02d}". Default: JNGFR{:02d} --instrument-source-template str template for source name (filled with karabo_id & receiver_id). e.g. 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default: {}/DET/{}:daqOutput --ctrl-source-template str template for control source name (filled with karabo_id_control). Default: {}/DET/CONTROL --karabo-id-control str if control is on a different ID, set to empty string if it is the same a karabo-id. Default: --use-dir-creation-date use the creation data of the input dir for database queries. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8017#8025 --cal-db-timeout int timeout on caldb requests. Default: 180000 --relative-gain do relative gain correction.. Default: True --no-relative-gain Opposite of --relative-gain (default: True) --strixel-sensor reordering for strixel detector layout.. Default: False --no-strixel-sensor Opposite of --strixel-sensor (default: False) --strixel-double-norm float normalization to use for double-size pixels, only applied for strixel sensors.. Default: 2.0 --limit-trains int ONLY FOR TESTING. process only first N trains, Use 0 to process all.. Default: 0 --chunks-ids int HDF chunk size for memoryCell and frameNumber.. Default: 32 --chunks-data int HDF chunk size for pixel data in number of frames.. Default: 1 --manual-slow-data if true, use manually entered bias_voltage, integration_time, gain_setting, and gain_mode values. Default: False --no-manual-slow-data Opposite of --manual-slow-data (default: False) --integration-time float integration time in us, will be overwritten by value in file. Default: 4.96 --gain-setting int 0 for dynamic gain, 1 for dynamic HG0, will be overwritten by value in file. Default: 0 --gain-mode int 0 for runs with dynamic gain setting, 1 for fixgain. It will be overwritten by value in file, if manual_slow_data is set to True.. Default: 0 --mem-cells int Set mem_cells to -1 to automatically use the value stored in RAW data.. Default: -1 --bias-voltage int will be overwritten by value in file. Default: 180 --skip-plots exit after writing corrected files. Default: False --no-skip-plots Opposite of --skip-plots (default: False) --plot-trains int Number of trains to plot for RAW and CORRECTED plots. Set to -1 to automatically plot all trains.. Default: 500 --cell-id-preview int cell Id used for preview in single-shot plots. Default: 15 --roi-definitions int [int ...] List with groups of 6 values defining ROIs, e.g. [3, 120, 180, 200, 550, -2] for module 3 (JNGFR03), slice 120:180, 200:550, average along axis -2 (slow scan, or -1 for fast scan). Default: [-1] required arguments: --in-folder str the folder to read data from, required. Default: None --out-folder str the folder to output to, required. Default: None --run int run to process, required. Default: None Jungfrau Dark Image Characterization ++++++++++++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 2.0 Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad pixel maps To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate JUNGFRAU DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str --run-high int --run-med int --run-low int [--karabo-da str [str ...]] [--karabo-id str] [--karabo-id-control str] [--receiver-template str] [--instrument-source-template str] [--ctrl-source-template str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--local-output | --no-local-output] [--db-output | --no-db-output] [--badpixel-threshold-sigma float] [--offset-abs-threshold-low int [int ...]] [--offset-abs-threshold-high int [int ...]] [--max-trains int] [--min-trains int] [--manual-slow-data | --no-manual-slow-data] [--time-limits float] [--integration-time int] [--gain-setting int] [--gain-mode int] [--bias-voltage int] [--memory-cells int] [--detailed-report | --no-detailed-report] [--operation-mode str] JUNGFRAU DARK # Jungfrau Dark Image Characterization # , Author: European XFEL Detector Group, Version: 2.0, , Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad pixel maps positional arguments: JUNGFRAU The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --karabo-da str [str ...] list of data aggregators, which corresponds to different JF modules. Default: ['JNGFR01', 'JNGFR02', 'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07', 'JNGFR08'] --karabo-id str karabo_id (detector identifier) prefix of Jungfrau detector to process.. Default: SPB_IRDA_JF4M --karabo-id-control str if control is on a different ID, set to empty string if it is the same a karabo-id. Default: --receiver-template str inset for receiver devices. Default: JNGFR{:02} --instrument-source-template str template for instrument source name (filled with karabo_id & receiver_id). e.g. 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default: {}/DET/{}:daqOutput --ctrl-source-template str template for control source name (filled with karabo_id_control). Default: {}/DET/CONTROL --use-dir-creation-date use dir creation date. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str calibrate db interface to connect to. Default: tcp://max-exfl016:8016#8045 --cal-db-timeout int timeout on caldb requests. Default: 300000 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --badpixel-threshold-sigma float bad pixels defined by values outside n times this std from median. Default: 5.0 --offset-abs-threshold-low int [int ...] absolute bad pixel threshold in terms of offset, lower values. Default: [1000, 10000, 10000] --offset-abs-threshold-high int [int ...] absolute bad pixel threshold in terms of offset, upper values. Default: [8000, 15000, 15000] --max-trains int Maximum trains to process darks. Set to 0 to process all available train images.. Default: 0 --min-trains int Minimum number of trains that should be available to process dark constants. Default 1.. Default: 1 --manual-slow-data if true, use manually entered bias_voltage and integration_time values. Default: False --no-manual-slow-data Opposite of --manual-slow-data (default: False) --time-limits float to find calibration constants later on, the integration time is allowed to vary by 0.5 us. Default: 0.025 --integration-time int integration time in us, will be overwritten by value in file. Default: 1000 --gain-setting int 0 for dynamic, forceswitchg1, forceswitchg2, 1 for dynamichg0, fixgain1, fixgain2. Will be overwritten by value in file. Default: 0 --gain-mode int 1 if medium and low runs are fixgain1 and fixgain2, otherwise 0. It will be overwritten by value in file, if manual_slow_data. Default: 0 --bias-voltage int sensor bias voltage in V, will be overwritten by value in file. Default: 90 --memory-cells int number of memory cells. Default: 16 --detailed-report Default: False --no-detailed-report Opposite of --detailed-report (default: False) --operation-mode str Detector operation mode, optional. Default: ADAPTIVE_GAIN required arguments: --in-folder str folder under which runs are located, required. Default: None --out-folder str path to place reports at, required. Default: None --run-high int run number for G0 dark run, required. Default: None --run-med int run number for G1 dark run, required. Default: None --run-low int run number for G2 dark run, required. Default: None LPD --- LPD Offline Correction ++++++++++++++++++++++ Author: European XFEL Data Analysis Group To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate LPD CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--sequences int [int ...]] [--modules int [int ...]] [--karabo-da str [str ...]] --run int [--karabo-id str] [--input-source str] [--output-source str] [--creation-time str] [--cal-db-interface str] [--cal-db-timeout int] [--cal-db-root str] [--mem-cells int] [--bias-voltage float] [--capacitor str] [--photon-energy float] [--category int] [--offset-corr | --no-offset-corr] [--rel-gain | --no-rel-gain] [--ff-map | --no-ff-map] [--gain-amp-map | --no-gain-amp-map] [--overwrite | --no-overwrite] [--chunks-data int] [--chunks-ids int] [--create-virtual-cxi-in str] [--sequences-per-node int] [--max-nodes int] [--num-workers int] [--num-threads-per-worker int] LPD CORRECT # LPD Offline Correction # Author: European XFEL Data Analysis Group positional arguments: LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequences int [int ...] Sequences to correct, use [-1] for all. Default: [-1] --modules int [int ...] Modules indices to correct, use [-1] for all, only used when karabo_da is empty. Default: [-1] --karabo-da str [str ...] Data aggregators names to correct, use [''] for all. Default: [''] --karabo-id str Karabo domain for detector.. Default: FXE_DET_LPD1M-1 --input-source str Input fast data source.. Default: {karabo_id}/DET/{module_index}CH0:xtdf --output-source str Output fast data source, empty to use same as input.. Default: --creation-time str The timestamp to use with Calibration DB. Required Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04 11:02:41. Default: --cal-db-interface str Not needed, compatibility with current webservice.. Default: --cal-db-timeout int Not needed, compatbility with current webservice.. Default: 0 --cal-db-root str Default: /gpfs/exfel/d/cal/caldb_store --mem-cells int Memory cells, LPD constants are always taken with 512 cells.. Default: 512 --bias-voltage float Detector bias voltage.. Default: 250.0 --capacitor str Capacitor setting: 5pF or 50pF. Default: 5pF --photon-energy float Photon energy in keV.. Default: 9.2 --category int Whom to blame.. Default: 0 --offset-corr Offset correction.. Default: True --no-offset-corr Opposite of --offset-corr (default: True) --rel-gain Gain correction based on RelativeGain constant.. Default: True --no-rel-gain Opposite of --rel-gain (default: True) --ff-map Gain correction based on FFMap constant.. Default: True --no-ff-map Opposite of --ff-map (default: True) --gain-amp-map Gain correction based on GainAmpMap constant.. Default: True --no-gain-amp-map Opposite of --gain-amp-map (default: True) --overwrite set to True if existing data should be overwritten. Default: True --no-overwrite Opposite of --overwrite (default: True) --chunks-data int HDF chunk size for pixel data in number of frames.. Default: 1 --chunks-ids int HDF chunk size for cellId and pulseId datasets.. Default: 32 --create-virtual-cxi-in str Folder to create virtual CXI files in (for each sequence).. Default: --sequences-per-node int Sequence files to process per node. Default: 1 --max-nodes int Maximum number of SLURM jobs to split correction work into. Default: 8 --num-workers int Worker processes per node, 8 is safe on 768G nodes but won't work on 512G.. Default: 8 --num-threads-per-worker int Number of threads per worker.. Default: 32 required arguments: --in-folder str the folder to read data from, required. Default: None --out-folder str the folder to output to, required. Default: None --run int run to process, required. Default: None LPD Offset, Noise and Dead Pixels Characterization ++++++++++++++++++++++++++++++++++++++++++++++++++ Author: M. Karnevskiy, S. Hauf This notebook performs re-characterize of dark images to derive offset, noise and bad-pixel maps. All three types of constants are evaluated per-pixel and per-memory cell. The notebook will correctly handle veto settings, but note that if you veto cells you will not be able to use these offsets for runs with different veto settings - vetoed cells will have zero offset. The evaluated calibration constants are stored locally and injected in the calibration data base. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate LPD DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--sequence int] [--modules str [str ...]] --run-high int --run-med int --run-low int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-id str] [--path-template str] [--h5path str] [--h5path-idx str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--local-output | --no-local-output] [--db-output | --no-db-output] [--capacitor-setting int] [--mem-cells int] [--bias-voltage int] [--thresholds-offset-sigma float] [--thresholds-offset-hard int [int ...]] [--thresholds-noise-sigma float] [--thresholds-noise-hard int [int ...]] [--skip-first-ntrains int] [--instrument str] [--ntrains int] [--high-res-badpix-3d | --no-high-res-badpix-3d] [--test-for-normality | --no-test-for-normality] [--operation-mode str] LPD DARK # LPD Offset, Noise and Dead Pixels Characterization # , Author: M. Karnevskiy, S. Hauf, , This notebook performs re-characterize of dark images to derive offset, noise and bad-pixel maps. All three types of constants are evaluated per-pixel and per-memory cell., , The notebook will correctly handle veto settings, but note that if you veto cells you will not be able to use these offsets for runs with different veto settings - vetoed cells will have zero offset., , The evaluated calibration constants are stored locally and injected in the calibration data base., , positional arguments: LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str The ipcluster profile to use. Default: noDB --sequence int sequence files to evaluate. Default: 0 --modules str [str ...] list of modules to evaluate, RANGE ALLOWED. Default: [-1] --karabo-id str karabo karabo_id. Default: FXE_DET_LPD1M-1 --karabo-da str [str ...] a list of data aggregators names, Default [-1] for selecting all data aggregators. Default: ['-1'] --receiver-id str inset for receiver devices. Default: {}CH0 --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{:05d}.h5 --h5path str path in the HDF5 file to images. Default: /INSTRUMENT/{}/DET/{}:xtdf/image --h5path-idx str path in the HDF5 file to images. Default: /INDEX/{}/DET/{}:xtdf/image --use-dir-creation-date use the creation date of the directory for database time derivation. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8015#8025 --cal-db-timeout int timeout on caldb requests". Default: 300000 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --capacitor-setting int capacitor_setting for which data was taken. Default: 5 --mem-cells int number of memory cells used. Default: 512 --bias-voltage int detector bias voltage. Default: 250 --thresholds-offset-sigma float bad pixel relative threshold in terms of n sigma offset. Default: 3.0 --thresholds-offset-hard int [int ...] bad pixel hard threshold. Default: [400, 1500] --thresholds-noise-sigma float bad pixel relative threshold in terms of n sigma noise. Default: 7.0 --thresholds-noise-hard int [int ...] bad pixel hard threshold. Default: [1, 35] --skip-first-ntrains int Number of first trains to skip. Default: 10 --instrument str instrument name. Default: FXE --ntrains int number of trains to use. Default: 100 --high-res-badpix-3d plot bad-pixel summary in high resolution. Default: False --no-high-res-badpix-3d Opposite of --high-res-badpix-3d (default: False) --test-for-normality permorm normality test. Default: False --no-test-for-normality Opposite of --test-for-normality (default: False) --operation-mode str Detector operation mode, optional. Default: required arguments: --in-folder str path to input data, required. Default: None --out-folder str path to output to, required. Default: None --run-high int run number in which high gain data was recorded, required. Default: None --run-med int run number in which medium gain data was recorded, required. Default: None --run-low int run number in which low gain data was recorded, required. Default: None LPD Radial X-ray Gain Evaluation ++++++++++++++++++++++++++++++++ Author: S. Hauf, Version 0.5 Taking proper flat field for LPD can be difficult, as air scattering will always be present. Additionally, the large detector mandates a large distance to the source, in order to reduce :math:`1/r` effects. Because of this a radial evaluation method is used, which assumes that pixels a the same radial distance :math:`r` should on average have the same signal :math:`S(r)`. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate LPD FF --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str --run int [--sequences int [int ...]] [--capacitor-setting int] [--mem-cells int] [--local-output | --no-local-output] [--db-output | --no-db-output] [--bias-voltage int] [--cal-db-interface str] [--use-dir-creation-date | --no-use-dir-creation-date] [--instrument str] [--limit-trains int] [--geometry-file str] [--beam-center-offset float [float ...]] [--allowed-gain-thresholds float [float ...]] [--badpix-entire-asic-threshold float] [--photon-energy float] [--max-images int] LPD FF # LPD Radial X-ray Gain Evaluation # , Author: S. Hauf, Version 0.5, , Taking proper flat field for LPD can be difficult, as air scattering will always be present. Additionally, the large detector mandates a large distance to the source, in order to reduce $1/r$ effects. , Because of this a radial evaluation method is used, which assumes that pixels a the same radial distance $r$ should on average have the same signal $S(r)$. positional arguments: LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS FF Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --sequences int [int ...] which sequence files to use. Default: [0] --capacitor-setting int capacitor_setting for which data was taken. Default: 5 --mem-cells int number of memory cells used. Default: 512 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --bias-voltage int detector bias voltage. Default: 250 --cal-db-interface str the database interface to use. Default: tcp://max- exfl015:5005 --use-dir-creation-date use the creation date of the directory for database time derivation. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --instrument str Default: FXE --limit-trains int limit the number of train for the evaluation. Default: 10 --geometry-file str the geometry file to use, MAR 2018. Default: /gpfs/exfel/d/cal/exchange/lpdMF_00.h5 --beam-center-offset float [float ...] offset from the beam center, MAR 2018. Default: [1.5, 1] --allowed-gain-thresholds float [float ...] relative gain values within these bounds are valid. Default: [0.5, 1.5] --badpix-entire-asic-threshold float if more than this fraction of pixels on an ASIC are "bad" the entire ASIC is flagged. Default: 0.25 --photon-energy float photon enery in keV. Default: 9.2 --max-images int maximum number of images to use in evaluation. Default: 1024 required arguments: --in-folder str path to already corrected input data, required. Default: None --out-folder str path to output to, required. Default: None --run int runs to process, required. Default: None Injecting calibration constant data to the database +++++++++++++++++++++++++++++++++++++++++++++++++++ Author: European XFEL Detector Group, Version: 1.0 Reading h5files of calibration constants to inject them to the database. Used for LPD To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate LPD INJECT_CONSTANTS --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --constant-names str [str ...] --in-folder str --out-folder str [--proposal str] [--runs str [str ...]] [--karabo-id str] [--karabo-da str [str ...]] [--cal-db-interface str] [--memory-cells int] [--bias-voltage int] [--capacitor int] [--category int] [--photon-energy float] --creation-time str LPD INJECT_CONSTANTS # Injecting calibration constant data to the database # , Author: European XFEL Detector Group, Version: 1.0, , Reading h5files of calibration constants to inject them to the database. Used for LPD positional arguments: LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS INJECT_CONSTANTS Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --proposal str Add proposal number to be sent to the database as a part of Raw data location.. Default: --runs str [str ...] Add list of runs to be sent to the database as a part of Raw data location.. Default: [''] --karabo-id str detector identifier.. Default: FXE_DET_LPD1M-1 --karabo-da str [str ...] karabo data aggregators. default "all" for all 16 karabo data aggregator names.. Default: ['all'] --cal-db-interface str calibration DB zmq address.. Default: tcp://max- exfl016:8015#8045 --memory-cells int Number of memory cells. Used for constant conditions.. Default: 512 --bias-voltage int bias voltage value. Used for constant conditions.. Default: 250 --capacitor int capacitor value. Used for constant conditions.. Default: 5 --category int calibration constant source category, 0 for European XFEl and 1 for RAL. Used for constant conditions.. Default: 0 --photon-energy float calibration constant photon energy. Used for constant conditions.. Default: 9.2 required arguments: --constant-names str [str ...] calibration constant name, required.. Default: None --in-folder str calibration constants folder, required.. Default: None --out-folder str output folder to store report path in case the notebook is executed by CLI, required.. Default: None --creation-time str creation time for the injected constants. required format '2019-01-20T14:12:06'. Default: None LPD Gain Characterization (Charge Injection) ++++++++++++++++++++++++++++++++++++++++++++ Author: S. Hauf, Version: 0.5 The following code characterizes the gain of the LPD detector from charge injection data, i.e. data with charge injected into the amplifiers, bypassing the sensor. The data needs to fulfil the following requirements: - each file should represent one scan point for one mudle, defined by detector gain setting and charge injections setting - settings need to overlap at at least one point for two neighboring gain ranges - 100 samples or more per pixel and memory cell should be present for each setting. The data is then analyzed by calcualting the per-pixel, per memory cell mean of the samples for each setting. These means are then normalized to the median peak position of a all means of the first module. Overlapping settings in neighboring gain ranges are used to deduce the slopes of the different gains with respect to the high gain setting. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate LPD PC --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] --in-folder str --out-folder str [--offset-store str] [--modules int [int ...]] [--capacitor-setting int] [--mem-cells int] [--local-output | --no-local-output] [--db-output | --no-db-output] [--db-input | --no-db-input] [--bias-voltage int] [--cal-db-interface str] [--use-dir-creation-date | --no-use-dir-creation-date] [--instrument str] [--high-res-badpix-3d | --no-high-res-badpix-3d] LPD PC # LPD Gain Characterization (Charge Injection) # , Author: S. Hauf, Version: 0.5, , The following code characterizes the gain of the LPD detector from charge injection data, i.e. data with charge injected into the amplifiers, bypassing the sensor. The data needs to fulfil the following requirements:, , * each file should represent one scan point for one mudle, defined by detector gain setting and charge injections setting, * settings need to overlap at at least one point for two neighboring gain ranges, * 100 samples or more per pixel and memory cell should be present for each setting., , The data is then analyzed by calcualting the per-pixel, per memory cell mean of the samples for each setting. These means are then normalized to the median peak position of a all means of the first module. Overlapping settings in neighboring gain ranges are used to deduce the slopes of the different gains with respect to the high gain setting. positional arguments: LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS PC Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str The ipcluster profile to use. Default: noDB --offset-store str path to offset store, overwrite use_db_input. Default: --modules int [int ...] Default: [-1] --capacitor-setting int capacitor_setting for which data was taken. Default: 5 --mem-cells int number of memory cells used. Default: 512 --local-output output constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --db-output output constants to database. Default: False --no-db-output Opposite of --db-output (default: False) --db-input Default: True --no-db-input Opposite of --db-input (default: True) --bias-voltage int detector bias voltage. Default: 300 --cal-db-interface str the database interface to use. Default: tcp://max- exfl016:8015 --use-dir-creation-date use the creation date of the directory for database time derivation. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --instrument str Default: FXE --high-res-badpix-3d Default: False --no-high-res-badpix-3d Opposite of --high-res-badpix-3d (default: False) required arguments: --in-folder str path to input data, required. Default: None --out-folder str path to output to, required. Default: None Mine XGM Data and LPD Profile +++++++++++++++++++++++++++++ Author: S. Hauf, Version: 0.1 To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate LPD XGM_MINE --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--dc float [float ...]] [--geometry-file str] [--proposal-folder str] [--xgm-folder str] [--adc-folder str] [--file-prefix str] [--raw-prefix str] [--xgm-prefix str] [--adc-prefix str] [--data-path str] [--gain-path str] [--xgm-path str] [--xgm-path-td str] [--adc-root str] [--adc-paths str [str ...]] [--temp-out-folder str] [--cluster-profile str] [--chunk-size int] [--runs str [str ...]] [--method str] [--sequences str [str ...]] LPD XGM_MINE # Mine XGM Data and LPD Profile # Author: S. Hauf, Version: 0.1 positional arguments: LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS XGM_MINE Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --dc float [float ...] center offset. Default: [1.5, 1] --geometry-file str the geometry file to use, MAR 2018. Default: /gpfs/exfel/d/cal/exchange/lpdMF_00.h5 --proposal-folder str folder under which runs can be found. Default: /gpfs/exfel/exp/FXE/201701/p002045/proc/ --xgm-folder str folder under which runs can be found. Default: /gpfs/exfel/exp/FXE/201701/p002045/raw/ --adc-folder str folder under which runs can be found. Default: /gpfs/exfel/exp/FXE/201701/p002045/raw/ --file-prefix str prefix of each data file - should be a template to replace run number. Default: CORR-R{:04d}-LPD --raw-prefix str prefix of each data file - should be a template to replace run number. Default: RAW-R{:04d}-LPD --xgm-prefix str prefix for XGM file - should be a template to replace run number. Default: RAW-R{:04d}-DA03 --adc-prefix str prefix for ADC file - should be a template to replace run number. Default: RAW-R{:04d}-DA01 --data-path str path to data in file. Default: INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/data --gain-path str path to data in file. Default: INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/gain --xgm-path str path to XGM data. Default: /INSTRUMENT/SA1_XTD2_XGM/DO OCS/MAIN:output/data/intensityTD --xgm-path-td str path to XGM AUX data. Default: /INSTRUMENT/SA1_XTD2_XG M/DOOCS/MAIN:output/data/intensityAUXTD --adc-root str root path to ADC data. Default: /INSTRUMENT/FXE_RR_DAQ/ADC/1:network/digitizers --adc-paths str [str ...] Default: ['channel_1_A/apd/pulseIntegral', 'channel_1_B/apd/pulseIntegral', 'channel_1_C/apd/pulseIntegral', 'channel_1_D/apd/pulseIntegral'] --temp-out-folder str Default: /gpfs/exfel/data/scratch/haufs/test/tempout6 --cluster-profile str Default: noDB --chunk-size int read this amount of data at once. Default: 512 --runs str [str ...] Default: 69-72 --method str method to use for evaluation of images: radial, average. Default: average --sequences str [str ...] range allowed. Default: [-1] PNCCD ----- pnCCD Data Correction +++++++++++++++++++++ Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0 The following notebook provides offset, common mode, relative gain, split events and pattern classification corrections of images acquired with the pnCCD. This notebook *does not* yet correct for charge transfer inefficiency. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate PNCCD CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--in-folder str] [--out-folder str] [--run int] [--sequences str [str ...]] [--sequences-per-node int] [--karabo-da str] [--karabo-id str] [--receiver-id str] [--path-template str] [--instrument-source-template str] [--commonModeAxis int] [--commonModeBlockSize int [int ...]] [--split-evt-primary-threshold float] [--split-evt-secondary-threshold float] [--saturated-threshold float] [--fix-temperature-top float] [--fix-temperature-bot float] [--gain int] [--bias-voltage float] [--integration-time int] [--photon-energy float] [--cal-db-interface str] [--cal-db-timeout int] [--creation-time str] [--only-offset | --no-only-offset] [--common-mode | --no-common-mode] [--relgain | --no-relgain] [--pattern-classification | --no-pattern-classification] [--chunk-size-idim int] [--limit-images int] PNCCD CORRECT # pnCCD Data Correction # , Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0, , The following notebook provides offset, common mode, relative gain, split events and pattern classification corrections of images acquired with the pnCCD. This notebook *does not* yet correct for charge transfer inefficiency. positional arguments: PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --in-folder str input folder. Default: /gpfs/exfel/exp/SQS/202031/p900166/raw --out-folder str output folder. Default: /gpfs/exfel/data/scratch/ahmed k/test/remove/pnccd_correct --run int which run to read data from. Default: 347 --sequences str [str ...] sequences to correct, set to -1 for all, range allowed. Default: [-1] --sequences-per-node int number of sequences running on the same slurm node.. Default: 1 --karabo-da str data aggregators. Default: PNCCD01 --karabo-id str karabo prefix of PNCCD devices. Default: SQS_NQS_PNCCD1MP --receiver-id str inset for receiver devices. Default: PNCCD_FMT-0 --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{:05d}.h5 --instrument-source-template str template for data source name, will be filled with karabo_id and receiver_id.. Default: {}/CAL/{}:output --commonModeAxis int axis along which common mode will be calculated, 0 = row, and 1 = column. Default: 0 --commonModeBlockSize int [int ...] size of the detector in pixels for common mode calculations. Default: [512, 512] --split-evt-primary-threshold float primary threshold for split event classification in terms of n sigma noise. Default: 4.0 --split-evt-secondary-threshold float secondary threshold for split event classification in terms of n sigma noise. Default: 3.0 --saturated-threshold float full well capacity in ADU. Default: 32000.0 --fix-temperature-top float fix temperature for top sensor in K, set to 0. to use value from slow data.. Default: 0.0 --fix-temperature-bot float fix temperature for bottom senspr in K, set to 0. to use value from slow data.. Default: 0.0 --gain int the detector's gain setting. Set to -1 to use the value from the slow data.. Default: -1 --bias-voltage float the detector's bias voltage. set to 0. to use value from slow data.. Default: 0.0 --integration-time int detector's integration time. Default: 70 --photon-energy float Al fluorescence in keV. Default: 1.6 --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8015 --cal-db-timeout int timeout on caldb requests. Default: 300000 --creation-time str The timestamp to use with Calibration DB. Required Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04 11:02:41. Default: --only-offset Only, apply offset.. Default: False --no-only-offset Opposite of --only-offset (default: False) --common-mode Apply common mode correction. Default: True --no-common-mode Opposite of --common-mode (default: True) --relgain Apply relative gain correction. Default: True --no-relgain Opposite of --relgain (default: True) --pattern-classification classify split events. Default: True --no-pattern-classification Opposite of --pattern-classification (default: True) --chunk-size-idim int H5 chunking size of output data. Default: 1 --limit-images int this parameter is used for limiting number of images to correct from a sequence file. ONLY FOR TESTING.. Default: 0 pnCCD Dark Characterization +++++++++++++++++++++++++++ Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0 The following notebook provides dark image analysis of the pnCCD detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. On the first iteration, the offset and noise maps are generated. Initial bad pixels map is obtained based on the offset and initial noise maps. Edge pixels are also added to the bad pixels map. On the second iteration, the noise map is corrected for common mode. A second bad pixel map is generated based on the offset map and offset-and-common-mode-corrected noise map. Then, the hole in the center of the CCD is added to the second bad pixel map. On the third and final iteration, the pixels that show up on the abovementioned bad pixels map are masked. Possible events due to cosmic rays are found and masked. The data are then again offset and common mode corrected and a new final noise and bad pixels maps are generated. These latter resulting maps together with the offset map are saved as .h5 files to a local path for a later use. These dark constants are not automatically sent to the database. To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate PNCCD DARK --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] --in-folder str --out-folder str --run int [--karabo-da str [str ...]] [--karabo-id str] [--receiver-id str] [--path-template str] [--instrument-source-template str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int] [--db-output | --no-db-output] [--local-output | --no-local-output] [--creation-time str] [--fix-temperature-top float] [--fix-temperature-bot float] [--temp-limits int] [--gain int] [--bias-voltage float] [--integration-time int] [--commonModeAxis int] [--commonModeBlockSize int [int ...]] [--sigmaNoise float] [--bad-pixel-offset-sigma float] [--bad-pixel-noise-sigma float] [--max-trains int] [--min-trains int] [--operation-mode str] [--db-module str] PNCCD DARK # pnCCD Dark Characterization , Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0, , The following notebook provides dark image analysis of the pnCCD detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. , On the first iteration, the offset and noise maps are generated. Initial bad pixels map is obtained based on the offset and initial noise maps. Edge pixels are also added to the bad pixels map., , On the second iteration, the noise map is corrected for common mode. A second bad pixel map is generated based on the offset map and offset-and-common-mode-corrected noise map. Then, the hole in the center of the CCD is added to the second bad pixel map., , On the third and final iteration, the pixels that show up on the abovementioned bad pixels map are masked. Possible events due to cosmic rays are found and masked. The data are then again offset and common mode corrected and a new final noise and bad pixels maps are generated., , These latter resulting maps together with the offset map are saved as .h5 files to a local path for a later use. These dark constants are not automatically sent to the database. positional arguments: PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS DARK Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --karabo-da str [str ...] data aggregators. Default: ['PNCCD01'] --karabo-id str karabo prefix of PNCCD devices. Default: SQS_NQS_PNCCD1MP --receiver-id str inset for receiver devices. Default: PNCCD_FMT-0 --path-template str the template to use to access data. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --instrument-source-template str data source path in h5file. Template filled with karabo_id and receiver_id. Default: {}/CAL/{}:output --use-dir-creation-date use dir creation date as data production reference date. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8021 --cal-db-timeout int timeout on caldb requests. Default: 300000 --db-output if True, the notebook sends dark constants to the calibration database. Default: False --no-db-output Opposite of --db-output (default: False) --local-output if True, the notebook saves dark constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --creation-time str To overwrite the measured creation_time. Required Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04 11:02:41.00. Default: --fix-temperature-top float fix temperature of top pnCCD sensor in K. Set to 0, to use the value from slow data. Default: 0.0 --fix-temperature-bot float fix temperature of bottom pnCCD sensor in K. Set to 0, to use the value from slow data. Default: 0.0 --temp-limits int temperature limits in which calibration parameters are considered equal. Default: 5 --gain int the detector's gain setting. Set to -1 to use the value from the slow data.. Default: -1 --bias-voltage float the detector's bias voltage. set to 0. to use the value from slow data.. Default: 0.0 --integration-time int detector's integration time.. Default: 70 --commonModeAxis int axis along which common mode will be calculated (0: along rows, 1: along columns). Default: 0 --commonModeBlockSize int [int ...] size of the detector in pixels for common mode calculations. Default: [512, 512] --sigmaNoise float pixels whose signal value exceeds sigmaNoise*noise will be considered as cosmics and are masked. Default: 10.0 --bad-pixel-offset-sigma float any pixel whose offset beyond this standard deviations is a bad pixel. Default: 4.0 --bad-pixel-noise-sigma float any pixel whose noise beyond this standard deviations is a bad pixel. Default: 4.0 --max-trains int Maximum number of trains to use for dark processing. Set to 0 to process all trains.. Default: 500 --min-trains int Minimum number of trains to proceed with dark processing.. Default: 1 --operation-mode str Detector operation mode, optional. Default: --db-module str the device name for pnCCD detector. Default: required arguments: --in-folder str input folder, required. Default: None --out-folder str output folder, required. Default: None --run int which run to read data from, required. Default: None pnCCD Gain Characterization +++++++++++++++++++++++++++ Authors: DET Group, modified by Kiana Setoodehnia on December 2020 - Version 4.0 The following notebook provides gain characterization for the pnCCD. It relies on data which are previously corrected using the Meta Data Catalog web service interface or by running the Correct\_pnCCD\_NBC.ipynb notebook. Prior to running this notebook, the corrections which should be applied by the web service or the aforementioned notebook are as follows: - offset correction - common mode correction - split pattern classification To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate PNCCD RELGAIN --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--cluster-profile str] [--in-folder str] [--out-folder str] [--run int] [--sequences str [str ...]] [--db-module str] [--karabo-da str] [--karabo-da-control str] [--karabo-id str] [--receiver-id str] [--path-template str] [--path-template-ctrl str] [--path-template-seqs str] [--h5path str] [--h5path-ctrl str] [--cpuCores int] [--use-dir-creation-date | --no-use-dir-creation-date] [--sequences-per-node int] [--chunkSize int] [--run-parallel | --no-run-parallel] [--db-output | --no-db-output] [--local-output | --no-local-output] [--cal-db-interface str] [--cal-db-timeout int] [--creation-time str] [--fix-temperature-top float] [--fix-temperature-bot float] [--gain float] [--bias-voltage float] [--integration-time int] [--photon-energy float] PNCCD RELGAIN # pnCCD Gain Characterization # , Authors: DET Group, modified by Kiana Setoodehnia on December 2020 - Version 4.0, , The following notebook provides gain characterization for the pnCCD. It relies on data which are previously corrected using the Meta Data Catalog web service interface or by running the Correct_pnCCD_NBC.ipynb notebook. Prior to running this notebook, the corrections which should be applied by the web service or the aforementioned notebook are as follows:, , - offset correction, - common mode correction, - split pattern classification positional arguments: PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS RELGAIN Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --cluster-profile str ipcluster profile to use. Default: noDB --in-folder str input folder for the raw data. Default: /gpfs/exfel/exp/SQS/202031/p900166/raw --out-folder str output folder. Default: /gpfs/exfel/data/scratch/setoodeh/Test --run int which run to read data from. Default: 347 --sequences str [str ...] sequences to correct, set to -1 for all, range allowed. Default: [-1] --db-module str Default: pnCCD_M205_M206 --karabo-da str data aggregators. Default: PNCCD01 --karabo-da-control str file inset for control data. Default: PNCCD02 --karabo-id str karabo prefix of PNCCD devices. Default: SQS_NQS_PNCCD1MP --receiver-id str inset for receiver devices. Default: PNCCD_FMT-0 --path-template str the template to use to access data. Default: CORR-R{:04d}-PNCCD01-S{{:05d}}.h5 --path-template-ctrl str the template to use to access data. Default: RAW-R{:04d}-{}-S{{:05d}}.h5 --path-template-seqs str Default: {}/r{:04d}/*PNCCD01-S*.h5 --h5path str path to data in the HDF5 file. Default: /INSTRUMENT/{}/CAL/{}:output/data/ --h5path-ctrl str Default: /CONTROL/{}/CTRL/TCTRL --cpuCores int specifies the number of running cpu cores. Default: 40 --use-dir-creation-date this is needed to obtain creation time of the run. Default: True --no-use-dir-creation-date Opposite of --use-dir-creation-date (default: True) --sequences-per-node int Default: 1 --chunkSize int number of images to read per chunk. Default: 100 --run-parallel Default: True --no-run-parallel Opposite of --run-parallel (default: True) --db-output if True, the notebook injects dark constants into the calibration database. Default: False --no-db-output Opposite of --db-output (default: False) --local-output if True, the notebook saves dark constants locally. Default: True --no-local-output Opposite of --local-output (default: True) --cal-db-interface str calibration DB interface to use. Default: tcp://max- exfl016:8015 --cal-db-timeout int timeout on caldb requests. Default: 300000 --creation-time str To overwrite the measured creation_time. Required Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04 11:02:41.00. Default: --fix-temperature-top float fix temperature of top pnCCD sensor in K, set to 0. to use the value from slow data. Default: 0.0 --fix-temperature-bot float fix temperature of bottom pnCCD sensor in K, set to 0. to use the value from slow data. Default: 0.0 --gain float the detector's gain setting. Set to 0, to use the value from slow data. Default: 0.1 --bias-voltage float the detector's bias voltage. set to 0. to use the value from slow data.. Default: 0.0 --integration-time int detector's integration time. Default: 70 --photon-energy float Al fluorescence in keV. Default: 1.6 REMI ---- Transformation parameters +++++++++++++++++++++++++ To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate REMI CORRECT --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--run int] [--in-folder str] [--out-folder str] [--calib-config-path str] [--cycle str] [--cal-db-timeout int] [--cal-db-interface str] [--karabo-da str] [--karabo-id str] [--proposal str] [--out-aggregator str] [--out-seq-len int] [--det-device-id str] [--det-output-key str] [--save-raw-triggers | --no-save-raw-triggers] [--save-raw-edges | --no-save-raw-edges] [--save-rec-signals | --no-save-rec-signals] [--save-rec-hits | --no-save-rec-hits] [--chunks-triggers int [int ...]] [--chunks-edges int [int ...]] [--chunks-hits int [int ...]] [--chunks-signals int [int ...]] [--dataset-compression str] [--dataset-compression-opts int] [--quad-anode | --no-quad-anode] [--ppt-source str] [--ignore-fel | --no-ignore-fel] [--ignore-ppl | --no-ignore-ppl] [--ppl-offset int] [--laser-ppt-mask int] [--instrument-sase int] [--first-pulse-offset int] [--single-pulse-length int] [--mp-find-triggers float] [--mp-find-edges float] [--mt-avg-trace int] [--mp-rec-hits float] REMI CORRECT # Transformation parameters positional arguments: REMI The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS CORRECT Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --run int Run ID.. Default: 104 --in-folder str Partial input path appended with run ID.. Default: /gpfs/exfel/exp/SQS/202101/p002535/raw --out-folder str Full path to output folder.. Default: /gpfs/exfel/exp/SQS/202101/p002535/scratch/cal_test --calib-config-path str Path to correction and transform configuration. Default: /gpfs/exfel/exp/SQS/202101/p002535/usr/config _board2+4.yaml --cycle str Proposal cycle, currently not used.. Default: --cal-db-timeout int Calibration DB timeout, currently not used.. Default: 0 --cal-db-interface str Calibration DB interface, currently not used.. Default: foo --karabo-da str Karabo data aggregator name, currently not used. Default: bar --karabo-id str Karabo device ID root for virtual output device.. Default: SQS_REMI_DLD6 --proposal str Proposal, leave empty for auto detection based on in_folder. Default: --out-aggregator str Aggregator name for output files.. Default: REMI01 --out-seq-len int Number of trains per sequence file in output.. Default: 5000 --det-device-id str Karabo device ID for virtual output device.. Default: {karabo_id}/DET/{det_name} --det-output-key str Pipeline name for fast data output.. Default: output --save-raw-triggers Whether to save trigger position in files.. Default: True --no-save-raw-triggers Opposite of --save-raw-triggers (default: True) --save-raw-edges Whether to save digitized edge positions in files.. Default: True --no-save-raw-edges Opposite of --save-raw-edges (default: True) --save-rec-signals Whether to save reconstructed signals (u1-w2, mcp) in files.. Default: True --no-save-rec-signals Opposite of --save-rec-signals (default: True) --save-rec-hits Whether to save reoncstructed hits (x,y,t,m) in files.. Default: True --no-save-rec-hits Opposite of --save-rec-hits (default: True) --chunks-triggers int [int ...] HDF chunk size for triggers.. Default: [500] --chunks-edges int [int ...] HDF chunk size for edges.. Default: [500, 7, 50] --chunks-hits int [int ...] HDF chunk size for hits.. Default: [50, 50] --chunks-signals int [int ...] HDF chunk size for signals.. Default: [50, 50] --dataset-compression str HDF compression method.. Default: gzip --dataset-compression-opts int HDF GZIP compression level.. Default: 3 --quad-anode Reconstruction assumes a hex anode by default, change for quad anodes.. Default: False --no-quad-anode Opposite of --quad-anode (default: False) --ppt-source str Default: SQS_RR_UTC/TSYS/TIMESERVER:outputBunchPattern --ignore-fel Ignore any FEL entries in the PPT.. Default: False --no-ignore-fel Opposite of --ignore-fel (default: False) --ignore-ppl Ignore any PPL entries in the PPT.. Default: False --no-ignore-ppl Opposite of --ignore-ppl (default: False) --ppl-offset int In units of the PPT.. Default: 0 --laser-ppt-mask int Bit mask for used laser, negative to auto-detect from instrument.. Default: -1 --instrument-sase int Default: 3 --first-pulse-offset int Default: 1000 --single-pulse-length int Default: 25000 --mp-find-triggers float Parallelization for finding triggers.. Default: 0.5 --mp-find-edges float Parallelization for digitizing analog signal.. Default: 0.5 --mt-avg-trace int Parallelization for trace averaging.. Default: 2 --mp-rec-hits float Parallelization for hit reconstruction.. Default: 1.0 TUTORIAL -------- Tutorial Calculation ++++++++++++++++++++ Author: Astrid Muennich Version: 0.1 A small example how to adapt a notebook to run with the offline calibration package "pycalibation". The first cell contains all parameters that should be exposed to the command line. To run this notebooks with several different input parameters in parallel by submitting multiple slurm jobs, for example for various random seed we can do the following: xfel-calibrate TUTORIAL TEST --random-seed 1,2,3,4 or xfel-calibrate TUTORIAL TEST --random-seed 1-5 will produce 4 jobs: Parsed input 1,2,3,4 to [1, 2, 3, 4] Submitted job: 1169340 Submitted job: 1169341 Submitted job: 1169342 Submitted job: 1169343 Submitted the following SLURM jobs: 1169340,1169341,1169342,1169343 To invoke this notebook and display help use: .. code-block:: bash xfel-calibrate TUTORIAL TEST --help The full parameter list of this notebook (with defaults is): .. code-block:: bash usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str] [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str] [--slurm-partition str] [--reservation str] [--out-folder str] [--sensor-size int [int ...]] [--random-seed str [str ...]] [--runs int] [--cluster-profile str] TUTORIAL TEST # Tutorial Calculation # , Author: Astrid Muennich, , Version: 0.1, , A small example how to adapt a notebook to run with the offline calibration package "pycalibation"., , The first cell contains all parameters that should be exposed to the command line., , To run this notebooks with several different input parameters in parallel by submitting multiple slurm jobs, for example for various random seed we can do the following:, , xfel-calibrate TUTORIAL TEST --random-seed 1,2,3,4, , or, , xfel-calibrate TUTORIAL TEST --random-seed 1-5, , will produce 4 jobs:, , Parsed input 1,2,3,4 to [1, 2, 3, 4], , Submitted job: 1169340, , Submitted job: 1169341, , Submitted job: 1169342, , Submitted job: 1169343, , Submitted the following SLURM jobs: 1169340,1169341,1169342,1169343 positional arguments: TUTORIAL The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS TEST Type of calibration. optional arguments: -h, --help show this help message and exit --no-cluster-job Do not run as a cluster job (default: False) --prepare-only Prepare notebooks but don't run them (default: False) --report-to str Filename (and optionally path) for output report (default: None) --not-reproducible Disable checks to allow the processing result to not be reproducible based on its metadata. (default: False) --skip-report Skip report generation in finalize step. (default: False) --skip-env-freeze Skip recording the Python environment for reproducibility purposes, requires --not-reproducible to run. (default: False) --concurrency-par str Name of concurrency parameter.If not given, it is taken from configuration. (default: None) --constants-from str Path to a calibration-metadata.yml file. If given, retrieved-constants will be copied to use for a new correction. (default: None) --vector-figs Use vector graphics for figures in the report. (default: False) --slurm-mem int Requested node RAM in GB (default: 500) --slurm-name str Name of slurm job (default: xfel_calibrate) --slurm-scheduling int Change scheduling priority for a slurm job +- 2147483645 (negative value increases priority) (default: 0) --request-time str Time of request to process notebook. Iso format (default: Now) --slurm-partition str Submit jobs in this Slurm partition (default: ) --reservation str Submit jobs in this Slurm reservation, overriding --slurm-partition if both are set (default: ) --out-folder str output folder. Default: /gpfs/exfel/data/scratch/amunnich/tutorial --sensor-size int [int ...] defining the picture size. Default: [10, 30] --random-seed str [str ...] random seed for filling of fake data array. Change it to produce different results, range allowed. Default: [2345] --runs int how may iterations to fill histograms. Default: 500 --cluster-profile str Default: tutorial