Calibration database¶
This is the main database where all calibration constants are stored. The calibration database is based on CALCAT as the main web interface and the GPFS file system where all calibration constant files are stored.
It is responsible for storing and managing all calibration detector constants. It maintains the history of calibration constants and manage different versions for the calibration database.
The Calibration database is expected to be queried using: - Calibration constant name - Conditions - Detector module name
Calibration constant¶
CALCAT¶
CALCAT can be accessed through https://in.xfel.eu/calibration. It consists of multiple definitions and metadata related to the stored calibration constants. the definitions for the available detector types (AGIPD, JUNGFRAU, ePix100, ...), the available instruments, the available detectors in all instrument, the mapping of each detector to it's connected physical detector units, and all calibration constants.
All of these content ensure retrieving the correct calibration constant any point in time.
Through CALCAT page one can check all stored calibration constants, where the files are stored, and their metadata. Additionally, there are different ways to edit various parameters like the detector to PDU mapping \<TODO: add a definition link>, adding new calibrations, and new parameter conditions.
In order to proceed with the relevant parts of CALCAT, it is critical to first establish clear definitions for the terms used in both CALCAT and pyCalibration within the context of the Calibration database.
- Calibrations: These are the types of calibration available that can be applied to RAW data. For example Offset, Noise, and BadPixelsDark are some calibrations available in CALCAT.
- Calibration Constants (CC): Calibration Constants are a list of calibration constant versions with the same parameter conditions. For example 3 Offsets injected in with 3 different creation time for AGIPD and the same parameter conditions are in the same CC list.
- Calibration Constant Version (CCV): CCVs are injected through time into the database. Each CCV must have a list of parameter conditions, an attached timestamp of when it was created, the physical detector unit, the file path, and the dataset name to access its data.
- Parameters conditions: The CC parameter conditions are decided by detector experts because the CC\'s data would differ based on it. Some examples are memory cells, integration time, and bias voltage.
- PDU: Physical detector unit is the physical name given by the manufacturer or detector experts to the detector module's hardware. This representation is very important to calibration, as it is only trace of the physical movement for the detector modules.
- Detector: A detector for CALCAT is based on a detector identifier name, which was based on the Karabo ID give for detector karabo devices. A detector can consist of one module like HED_IA1_JF500K1, or multiple modules like MID_DET_AGIPD1M-1, FXE_DET_LPD1M-1, and SPB_IRDA_JF4M.
- Reports: pyCalibration's PDF reports are the main data visualization source for any calibration processing, either correcting raw data or generating calibration constants. For CALCAT the only relevant reports are the reports for generating new calibration constants. The paths for these reports are one injected metadata along with the calibration constant.
Detector Mapping¶
Detector mapping is the operation of associating physical detector units (modules) with its data sources (raw data, data aggregators and calibration constants files)
PDUs can be transferred between detectors and instruments or replaced. To correct raw data, the calibration pipeline produces calibration constants, which are specific to the module they were generated from. As a result, it's crucial to map the hardware to the corresponding software modules.
It's possible to update a detector mapping through CalCat in case the PDUs are already available. i.e. moving a PDU from one detector to another. Otherwise, only users with admin access can add entirely new PDUs.
Modifying detector mapping¶
The detector experts are the most up-to date with the PDU changes. They are able to update and track the PDU names to software module using CALCAT.
Updating the mapping immediately after making any hardware modifications is crucial before acquiring RAW data for calibration. This is because during offline calibration, the timestamp of the acquired RAW data serves as a reference point to retrieve the correct mapping from the database.
History of Detector mapping¶
In the beginning, detector mapping was designed to be hard coded in iCalibrationDB. It is an interface package to CalibrationDBRemote and CALCAT.
This screenshot shows the last updated hard coded physical detector units for AGIPD detector at SPB.
The of course led with error-prone to not be updated in time after there were changes reported when PDUs where replaced or moved.
Database snapshot¶
A database snapshot is a point-in-time copy of a database that provides a read-only, consistent view of the data as it existed at the time the snapshot was created. It's a static picture of the database and includes all the data that was in the database at the time of the snapshot's creation. Database snapshots are useful for performing data analysis or creating reports without interfering with ongoing database transactions. They can also be used for backup and recovery purposes or to provide a consistent view of the data for testing and development purposes.
CALCAT snapshot feature is used for obtaining the correct detector mapping. This is done by using the RAW data timestamp (creation_time) as the snapshot timestamp.
This is why it is crucial to update the detector mapping immediately after any hardware changes.