Offline calibration overview

TODO: HERE I PLAN TO GIVE DIAGRAMS OVER THE OFFLINE CALIBRATION SERVICES AND PIECES

Quote

European XFEL aims to provide facility users with a fully corrected and calibrated dataset as the primary data product. - SRN 27.4, 35 (2014)

Offline calibration consists of multiple algorithms and processes to achieve this role. These steps can differ across detectors. Offline calibration process big amount of data, and it is essential to have the calibration pipeline compatible with RAW data format across time.

pyCalibration is the main pipeline package for offline calibration at European XFEL. It utilizes MAXWELL HPC to run Jupyter notebooks concurrently using SLURM nodes to process RAW data. Before correcting detector RAW data, and after the detector experts characterize the detector

The pieces we have are:

• Calibration Database The calibration database (CALCAT)
• Metadata Catalog myMDC for requesting run corrects and dark constants' generation.
• Calibration Notebooks All available detector calibration notebooks
• xfel-calibrate The machinery for running calibration notebooks using SLURM
• Calibration webservice The main entry for offline calibration pipeline in production