Metadata Catalouge

European XFEL Data Portal is provided by myMdC, allowing users to access the metadata of scientific data taken at the European XFEL facility. Datasets are linked to experiment proposals, annotated by data collection types, samples and are assigned DOIs to be referenced in publications. myMdC helps data management activities and allows users to find open and own proposals data based on the metadata.

myMDC stands for Metadata Catalog. It arranges the facility's data into proposals. Each proposal consists of a number of acquired runs. Offline calibration only works with runs set to Good Run Quality, which means it was migrated from the online cluster to gpfs.

Through myMDC users can request to correct migrated runs. Additionally, users can trigger dark calibration constants generation detector data.

Proposals

Proposals are created for commissioning or user experiments. There can be a proposal per instrument. Every proposal has a number of team members who have access to interact with the proposal's runs.

proposal's RAW data location can be found at: /gpfs/exfel/exp/<instrument>/<cycle>/<proposal>/raw

Runs

Runs have multiple properties to modify through myMDC.

The most relevant properties are:

• Run Quality: Currently only migrated runs are allowed to be offline calibrated.
• Run Types: A run type can be relevant in accepting a correction request or not. e.g. Correcting dark runs can be skipped on purpose.

Triggering offline correction

Through the Runs page, one can trigger offline correction with the right proposal access. This screenshot shows a sample from the Runs page for the test proposal 900203.

The main important columns in the Runs page are: - Run number: The run number - Run type: This is an assigned type for the run. - Start date: The data when the run was acquired. - Run status: {TODO} - Data Assessment: If data was migrated or not. - Calibration: A symbol showing the calibration availability.

Dark runs are skipped.

Dark runs are not calibrated by default. In case a manual request was done. The calibration status will be in error state.

To check more details about a specific run like its ID, size, number of files, first and last train Ids, clicking on the run number will convert you the run page.

This page is very useful in checking the correction status, as both Calibrated Process data status and Calibration Pipeline auto Reply keys are updated regularly by the calibration service until a correction is finished.

For example if a run correction request failed. Calibration Pipeline auto Reply can have very useful information on the issue and ideally how to solve it.

Additionally, Processing Reports tab have all generated PDF reports for every calibration attempt.

---

Generating calibration constants

Currently, only dark calibration constants generation can be triggered through myMDC. the below screenshot shows the Calibration Constants page for proposal 900203. As noted before the current available option is to request dark calibration. To go through:

• Detector: First you need to select the desired detector from the available detectors of the proposal.
• Detector Units: By default, all the detector's modules are selected. But you can deselect the undesired modules. e.g. if a module is known not to have data for this run.
• Operation Mode: This can be left with the default value. It is meant to be used if there is a detector that should generate constants in different operation modes. But this is not used at the moment.
• Run Number(s): At the end you need to include the number of dark runs to create the calibration constants. This can be one or more runs depending on the detector.
• Description [optional]: You can optionally add a description for this particular dark constant request.

Calibration Constants requests

Note

The screenshots are from a special test proposal, therefore there are multiple detectors from different instruments in the same proposal.

As it can be seen in the screenshot, this section has the latest dark requests with their status, last update, PDF report link, CALCAT report link, and two blue buttons to clone the request for another submit and to check the dark request details, as can be seen in the below screenshot.

Dark runs' request details

This page is very useful to check more details regarding the dark request. For example the report path, the output path, who triggered request and when, and the calibration pipeline feedback.

The calibration pipeline feedback is crucial in case the dark request is in error state. As it can have helpful message over what was the cause of the failure.

Dark runs' request details doesn't auto-refresh.

This dark runs' request details page doesn't update automatically. Therefore, you need to close and reopen to see new updates.