Calibration Configurations¶
Calibration configurations is a separate project that is used by the offline calibration webservice to input the needed arguments for the triggered calibrations (e.g. correct or dark) per proposal.
These configurations are collected in YAML files. Each proposal can has it's own configuration, otherwise a default YAML file consists of the configurations for all calibrations, instruments, and detectors in case a proposal didn't have a dedicated YAML file.
Configuration project structure¶
The default configuration is located in the default.yaml file on the top hierarchy level.
The proposal specific configurations are optional (if none is found) the default is used, and are organized by facility cycles:
--<CYCLE A>
|--<PROPOSAL 1>.yaml
|--<PROPOSAL 2>.yaml
--<CYCLE B>
|--<PROPOSAL 3>.yaml
|--<PROPOSAL 4>.yaml
Where cycles and proposals are identified by their numerical number, without any prefixes or suffixes.
Configuration file contents¶
Each proposal YAML file (and the default) is structured in the following way
<CALIBRATION>:
<INSTRUMENT>:
<DETECTOR-IDENTIFIER>:
<cal-parameter-name>: <cal-parameter-value>
<cal-parameter-name>: <cal-parameter-value>
<DETECTOR-IDENTIFIER>:
<cal-parameter-name>: <cal-parameter-value>
<cal-parameter-name>: <cal-parameter-value>
<data-mapping>:
<DETECTOR-IDENTIFIER>:
detector-type: <DET-TYPE>
karabo-da:
- <Aggregator-1>
- <Aggregator-2>
<xfel-calibrate-setting F>: XXX
<parameter-name>: <parameter-value>
Multiple instruments and detector types per YAML are possible. The data-mapping key must have all detectors added in any of the available calibrations (<CALIBRATION>). It consists of the parameter names and values that are used to access data and that are expected to be the same for all calibrations.
Each calibration (e.g. dark or correct) can have the cal-parameter-names and cal-parameter-values that correlate to those expected in their calibration notebooks, with _ replaced with -.
Below is an example for the AGIPD detector at SPB:
correct:
SPB:
SPB_DET_AGIPD1M-1:
adjust-mg-baseline: true
force-hg-if-below: true
force-mg-if-below: true
hg-hard-threshold: 1000
low-medium-gap: true
mg-hard-threshold: 1000
rel-gain: true
dark:
SPB_DET_AGIPD1M-1:
thresholds-offset-hard:
- 0
- 0
thresholds-offset-hard-hg:
- 3500
- 6000
thresholds-offset-hard-lg:
- 6000
- 9000
thresholds-offset-hard-mg:
- 6000
- 9000
data-mapping:
SPB_DET_AGIPD1M-1:
ctrl-source-template: '{}/MDL/FPGA_COMP'
detector-type: agipd
karabo-da:
- AGIPD00
- AGIPD01
- AGIPD02
- AGIPD03
- AGIPD04
- AGIPD05
- AGIPD06
- AGIPD07
- AGIPD08
- AGIPD09
- AGIPD10
- AGIPD11
- AGIPD12
- AGIPD13
- AGIPD14
- AGIPD15
karabo-id-control: SPB_IRU_AGIPD1M1
receiver-template: '{}CH0'
Note how Boolean flags are indicated by
Updating configuration through the webservice (update_config)¶
update_config.py is a python script that can be used to update the calibration configurations directly in the production environment without having access to the production node. Through command line interface the user can apply the desired modifications to the proposal YAML files at https://git.xfel.eu/detectors/calibration_configurations
This script is expected to be available in a location which is easily accessed by any user to modify the configurations of specific proposal. Not all detectors or parameters can be modified through the script.
Changing the configuration:¶
- SSH to maxwell node
- run
module load anaconda3
At this moment you are ready to start interacting with the update_config script at: /gpfs/exfel/sw/calsoft/update_config.py
The available detectors and parameters to modify for a proposal can be checked using --help. python /gpfs/exfel/sw/calsoft/update_config.py --help
Below is the expected output. As it can be seen from the karabo-id argument, the only available detectors at the moment are: SPB_DET_AGIPD1M-1, MID_DET_AGIPD1M-, SQS_REMI_DLD6
usage: update_config.py [-h]
[--karabo-id {SPB_DET_AGIPD1M-1,MID_DET_AGIPD1M-1,SQS_REMI_DLD6}]
[--proposal PROPOSAL] [--cycle CYCLE]
[--correct | --dark] [--apply]
[--webservice-address WEBSERVICE_ADDRESS]
[--instrument {CALLAB}]
Request update of configuration
optional arguments:
-h, --help show this help message and
exit
--apply Apply and push the
requested configuration
update to the git.
--webservice-address WEBSERVICE_ADDRESS The port of the webservice
to update calibration
configurations repository.
--instrument {CALLAB} This is only used for
testing purposes.
required arguments:
--karabo-id {SPB_DET_AGIPD1M-1,MID_DET_AGIPD1M-1,SQS_REMI_DLD6}
--proposal PROPOSAL The proposal number,
without leading p, but
with leading zeros.
--cycle CYCLE The facility cycle.
--correct, -c
--dark, -d
To check the available parameters that can be modified one can run: python /gpfs/exfel/sw/calsoft/update_config.py --karabo-id SPB_DET_AGIPD1M-1 --help
Below is a part of the output of the CL. As can be seen under the optional arguments are the exposed parameters by update_config for SPB_DET_AGIPD1M-1.
optional arguments:
-h, --help show this help message and
exit
--apply Apply and push the
requested configuration
update to the git.
--webservice-address WEBSERVICE_ADDRESS The port of the webservice
to update calibration
configurations repository.
--instrument {CALLAB} This is only used for
testing purposes.
--force-hg-if-below FORCE_HG_IF_BELOW TYPE: INT
--rel-gain REL_GAIN TYPE: BOOL
--no-rel-gain NO_REL_GAIN TYPE: BOOL
--xray-gain XRAY_GAIN TYPE: BOOL
--no-xray-gain NO_XRAY_GAIN TYPE: BOOL
--blc-noise BLC_NOISE TYPE: BOOL
--no-blc-noise NO_BLC_NOISE TYPE: BOOL
--blc-set-min BLC_SET_MIN TYPE: BOOL
--no-blc-set-min NO_BLC_SET_MIN TYPE: BOOL
--dont-zero-nans DONT_ZERO_NANS TYPE: BOOL
--no-dont-zero-nans NO_DONT_ZERO_NANS TYPE: BOOL
--dont-zero-orange DONT_ZERO_ORANGE TYPE: BOOL
--no-dont-zero-orange NO_DONT_ZERO_ORANGE TYPE: BOOL
--max-pulses MAX_PULSES [MAX_PULSES ...] Range list of maximum
pulse indices (--max-
pulses start end step). 3
max input elements. TYPE:
LIST
--use-litframe-finder USE_LITFRAME_FINDER TYPE: STR
--litframe-device-id LITFRAME_DEVICE_ID TYPE: STR
--energy-threshold ENERGY_THRESHOLD TYPE: INT
--karabo-da KARABO-DA [KARABO-DA ...] Choices: [AGIPD00 ...
AGIPD15]. TYPE: LIST
required arguments:
--karabo-id {SPB_DET_AGIPD1M-1,MID_DET_AGIPD1M-1,SQS_REMI_DLD6}
--proposal PROPOSAL The proposal number,
without leading p, but
with leading zeros.
--cycle CYCLE The facility cycle.
--correct, -c
--dark, -d
Every exposed parameter has its type available beside the name.
Note: The boolean parameters can not be set to false. For example to set xray-gain to false, one should set no-xray-gain to true.
An example running the CL:
python /gpfs/exfel/sw/calsoft/update_config.py --cycle 202031 --proposal 900146 --karabo-id SPB_DET_AGIPD1M-1 --rel-gain true --no-xray-gain true --max-pulses 1 20 1
The output can be something like this:
--------------------------------------------------------
THIS IS A DRY RUN ONLY, NO CHANGES ARE MADE
---------------------------------------------------------
# Sending the following update:
correct:
SPB:
SPB_DET_AGIPD1M-1:
max-pulses:
- '1'
- '20'
- '1'
rel-gain: true
xray-gain: false
---------------------------------------------------------
# Configuration now in place is:
correct:
SPB:
SPB_DET_AGIPD1M-1:
adjust-mg-baseline: false
blc-noise: true
blc-set-min: false
blc-stripes: false
cm-dark-fraction: 0.5
cm-dark-range:
- -50
- 30
cm-n-itr: 4
common-mode: false
force-hg-if-below: true
force-mg-if-below: true
hg-hard-threshold: 1000
low-medium-gap: false
max-pulses:
- '1'
- '20'
- '1'
mg-hard-threshold: 1000
rel-gain: true
xray-gain: false
As the output shows, this is a dry run only. That means that the changes are not applied. This is just a display of what the changes would look like.
To run the changes and apply it to the calibration configurations, you should execute the same command with --apply argument. e.g.:
python /gpfs/exfel/sw/calsoft/update_config.py --cycle 202031 --proposal 900146 --karabo-id SPB_DET_AGIPD1M-1 --rel-gain true --no-xray-gain true --max-pulses 1 20 1 --apply
This should update the production with your changes. To validate the added changes into production, you can simply check the gitlab calibration_configurations repository and look for the latest commit to master. This should have your applied changes.