Skip to content

Calibration Webservice

As the heart of the calibration data pipeline. The webservice receives requests from myMDC to correct data or generate dark calibration constants.

The webservice is a calibration service that is deployed on a Maxwell node as a part of pyCalibration's new releases.

This service hanlde requests from myMDC via ZMQ interface to create the needed xfel-calibrate CL for the selected detector (i.e dark request), or for the available detectors in the RAW data (i.e correction request). The CL arguments are given from different configurations in the production environment, e.g. calibration configuration

Beside forming and executing the CL for the corresponding detector and calibration, the webservice monitor the states for the calibration Slurm nodes to report periodically it's state to myMDC and to conclude the last state for the whole calibration request with a response that can help the user to reach the successfully calibration data or to have an indication on why the Slurm job was not completed at the end.

Job database

The webservice uses SQLite database to store and keep track of the requests, executions, and calibration Slurm jobs.

job database

As can be seen, there are three tables. Executions, Slurm jobs, Requests.

Handling dark request

Users can generate calibration constants using myMDC. At the moment only dark calibration constants can be generated through myMDC and the webservice. Via ZMQ, the webservice start handling a dark request and update the Job DB with the new request ID. Similar to the correction, the configurations are used for adding the needed configurations for xfel-calibrate CLI. The launch will not go through as long as the dark runs are migrated and it will either wait for the transfer to finish or timeout. The webservice is expected to reply to myMDC with the status for the dark request and send a successful message or an error message with the reason for the error.

Handling correction request

Users can trigger offline calibration through myMDC. The webservice would handle this remote request via ZMQ and start with assigning this request into the Job DB. Next step would be reading configurations for the correction xfel-calibrate CLI and launch the correction after confirming that the RAW data is migrated, otherwise the correction waits until the transfer is complete. By default the corrections are disabled and skipped for dark run types. The webservice replies to myMDC with a success message that the correction is launched or with an error message if the correction was not launched.

upex calibration partitions

As the webservice is the main entry for all calibration requests during operation and user experiments. It is essential to be able to manage resource acquirement to different proposals and calibration actions.

Previously, there was a several tried options to provide the urgent proposals with available nodes and avoid delaying a crucial dark processing for an experiment because other correction jobs and users are allocating the current maxwell nodes. Node reservations were used as well as having scripts from ITDM side to increase slurm priority for jobs running with xcal user.

The current solution for sometime has been having two upex partitions with a priority given to xcal user, and the possibility to take over nodes from different users if no other node is available in the used partition.

The two partitions are upex-middle and upex-high, which are used for correction and dark calibrations respectively. The idea is to launch slurm jobs using one of these partitions in case the calibrated run belongs to an ACTIVE proposal.

As shown below this is a screenshot showing proposal 5438 when it had a Beamline status as Active Active

job monitor

The Job DB is regularly monitored by a separate and a dedicated service. Below is a screenshot for it's logs through the webservice overview page.

webservice job monitor

Webservice Overview webpage

This is the main webservice webpage max-exfl-cal001.desy.de:8008. Through it one can have an overview of the latest webservice activity:

  • Running calibration jobs and access.
  • The latest dark processing reports. It is possible to access all dark processing reports for all detectors.
  • The latest correction reports.
  • The main webservice logs.
  • The job monitor logs.

This webpage can be accessed only on Maxwell. And it is a separate service which runs along the webservice.