Skip to content

Release Notes

3.14.1

[AGIPD][CORRECT][DARK] Break wrong assumption for availability for 1st module. [AGIPD][CS] new current source calibration constant. [AGIPD][CS] Correction using Current source slopes. [Shimadzu][CORRECT] Fix name of index group in output files. [JUNGFRAU][FF] Feat: new notebook for producing gain constants. Feat/ci new runners python311. Create an unreadable temp directory for testing PermissionError.

3.14.0

  • [shimadzu] Add support ofr PCA-based dynamic flat-field corrections of Shimadzu HPVX2 data.
  • [AGIPD] Fix PC processing to allow for a single module.
  • [AGIPD] Remove default value for caldb_root.
  • Add staging version of new CalCat API to retrieve calibration data (to be superseded by extra.calibration).
  • Add API to create calibration constant data files and inject directly into CalCat.
  • Configure number of required characterization runs from CalCat based on operation mode.

3.13.0

  • [AGIPD][PC] Fixing ValueError.
  • [Gotthard2][Dark] Sort dark runs.
  • [ePix100][Correct] Correcting one train for epix100 and storing a list of one pulseId.
  • [Doc] Change log update with latest releases.
  • [Gotthard2][Correct] Add idx to da_to_pdu to use in picking correct data_source.
  • [LPD][Dark] Sort Dark runs by Gain.
  • [Webservice] Use AW status on myMdC for warnings that prevent launching correction.
  • [Webservice] Move list of three gain/run detectors into webservice config yaml.
  • [Webservice] add environment bin/ to PATH if not already there.
  • Convert request time to local timezone in report.
  • Don't keep DEBUG level logging from requests_oauthlib.
  • Avoid using run_metadata() by default, to not fail for EXDF-v0.5 files.

3.12.6

  • [Jungfrau][Correct] Avoid NAN disturbance for correction plots scale
  • [Timepix] Updating Timepix3 Calibration Pipeline for compatibility with new Karabo Implementation + some general improvements
  • [Timepix] Allow pipelines to be part of nodes in DataFile API
  • [Timepix] Throw pasha at it
  • [Timepix] Store labels assigned to pixels

3.12.5

  • SPARTA AGIPD Single Module integration

3.12.4

  • [Gotthard2][Correct] Store mask data properly for gotthard2 25um
  • Make the run type regex for skipping JF darks less aggressive

3.12.3

  • [webservice] Update Kafka config to use EuXFEL broker

3.12.2

  • [GH2] Temporary workaround for processing HIREX detector's dark runs using the correct data sources
  • [GH2][Correct] Show number of available trains and raise ValueError for runs with no trains
  • [GH2][Correct] New paramater to reverse second GH2 25um module
  • [Jungfrau][Correct] Fix not returning after finding a dark fixed gain constants
  • [Jungfrau][Dark][Correct] Improve Jungfrau plots
  • [Webservice] Add new detector HIREX to 3 dark runs detectors

3.12.1

  • [AGIPD][PC] Summary notebook for PC processing
  • [GH2][Correct][Dark] Break assumptions on receiver names
  • [Webservice] Create correction reports in MyMdC when jobs have finished
  • [Webservice] Use leading zeros in usr/Reports folders
  • Make serve_overview resistant against missing karabo-da in data-mapping
  • Add links to DESY grafana for running jobs in serve_overview

3.12.0

  • [AGIPD][Correct] Bug for using drop with a list in AGIPD CORR NB
  • [AGIPD][Correct] Try to simplify & speed up file reading code
  • [GH2][Correct][DARK] Feat/add support for gh2 25um
  • [Jungfrau][Correct] Update rows A1256 LUT after rotation
  • [Jungfrau][Dark] Account for dark runs not taken by the MDL device
  • [JUNGFRAU][Dark] Fix bad pixel constant from darks taken in burst mode
  • [REMI] Add and enhance plots, more documentation, clean-up and some deprecation
  • [TEST] Update AGIPD tests with xray-gain, add DSSC, ePix100 tests and update reference folder path
  • [TEST] clearer comparison of HDF5 files
  • DataFile: Don't write creationDate & updateDate unless specified

3.11.5

  • Update CalParrot==0.3 and EXtra-data==1.15.1 dependencies
  • [DSSC][Dark][Correct] No longer restrict memory cells to a multiple of 100s and add lower deviation for memory cells parameter conditions.
  • [Webservice] Catch errors on failure to launch dark processing
  • Add script to update dark run status in myMdC
  • [Epix100][Correct] Calcat error when no gain is retrieved
  • [REMI] Disable trailing trigger by default

3.11.4

  • [Jungfrau][Correct] Force fixed gain for JF data in burst mode

  • [Jungfrau][Correct] Force replacement for gain value in Jungfrau correction

  • [DSSC] Allow 900 memory cells for DSSC darks to workaround appearance of cell 810

  • [Jungfrau][Dark] Reflect WRONG_GAIN_VALUE over a pixel in all gain for badpixels map

  • [AGIPD][Dark] Add timings

  • [Jungfrau][Correct] New A1256 JF Strixel

  • [webservice] Add JUNGF and PEP 8 on the line

3.11.3

  • [AGIPD][LPD][DARK] Show table for bad pixels bitmaps
  • [AGIPD][CORRECT] Process all AGIPD trains if the PPU device is missing or if no trigger
  • [AGIPD][DARK] Sort dark runs
  • [AGIPD][DARK] Improvements for reading conditions by creating a new data class for multiple runs

  • [AGIPD][FF] Fixing FF summary performance plots

  • [Jungfrau][Correct] Add thresholding for ROI projections

  • [Jungfrau][Correct][Dark] Fix manual edit for operating conditions
  • [Jungfrau][DARK] Validate and reorder dark runs before processing

  • [EPIX][FF] ePixFF characterization

  • [REMI] Add support for virtual trailing trigger

  • Fix manually submitting the confirmation

  • move some logs to DEBUG and extend the report sleep

  • Expose --blc-stripes to update_config.py and fix old parameter names

3.11.2

  • Operational release for SPB to support configurable rounding thresholds.

3.11.1

  • [AGIPD][CORRECT] Use calcat_interface and remove precorrection notebook
  • [EPIX100] Feat: Compliance with update to receiver device
  • [REMI] Various fixes and improvements for quad DLDs
  • [REMI] Fix missing re-allocation of trigger array with neither FEL nor PPL
  • [Tests] Fix: Accept uppercase calibration type
  • [Test] Find difference by default
  • Clearer error when xfel-calibrate would run no jobs
  • Detect cycle automatically in update_config script
  • Fix link to CalCat

3.11.0

  • [AGIPD][Correct] Handle selecting multiple trains per PPU trigger
  • [AGIPD][Dark] Fix: Skip corrupted frame from dark processing
  • [LPD1M] Automatically decide whether to inject & use memory cell order
  • [LPD1M][Dark] Use EXtra-data to create darks from >1 sequence file
  • [LPD1M][Correct] Use parameter names instead of IDs to find constants
  • [LPD1M][Correct] Using CALCAT interface
  • [LPD1M][Correct] Fix: Constant type conversion
  • [LPD1M][Correct] Use the fragment file and remove the precorrection notebook
  • [LPD-Mini] Rework cell order condition to match LPD-1M again
  • [JUNGFRAU][pnCCD][ePix100] Feat: new method to display CCV metadata in reports
  • [JUNGFRAU][CORRECT] Add fragment file and remove precorrection notebook
  • [EPIX][DARK] Mark dead pixels as Bad Pixels
  • [EPIX][CORR] Optimize histograms and plots
  • [GH2][Correct] Move false warning and disable gain correction as printed.
  • [GH2][Correct] Remove the precorrection notebook and add fragment
  • [TIMEPIX] Add select parameters to update_config
  • [TIMEPIX] Fix types of notebook arguments
  • [xfel-calibrate] Fix: Break the line properly into latex when the next line starts with _
  • [Webservice] Don't mark jobs as finished just because they disappear from squeue output
  • [Webservice] Use status AW in myMdC if correction failed for some detectors in a run
  • Add a pytest to run a dict of CALLAB test runs before releases
  • Look up CCVs using parameter_name in place of parameter_id
  • Replace max-exfl016 and max-exfl017 into max-exfl-cal001 and max-exfl-cal002, respectively.
  • Make metadata directory name match report filename
  • Add reorder_axes function

3.10.3

  • [LPD][Correct] Harden against empty sequencee sets with train-on-demand
  • [JF][correct] Add missing gain mode parameter
  • [Timepix3] Add centroiding notebook

3.10.2

  • [PNCCD][CORRECT] Fix: Skip error for missing gain

  • [PNCCD][CORRECT] Fix: Hack to wrong ctrl bias voltage values p002857

  • [LPD][Correct] Fix axis order for LPD-1M RelativeGain constant

  • [LPDMini][Dark] Add only number of available data trains into data_samples

  • [LPDMini][Dark] Fix first notebook cell to execute CL through the webservice

  • [LPDMini] Feat: Inject gain constants notebook

  • [Jungfrau] Workaround for SPB_CFEL_JF1M as the modules start with 09 not 01

  • Fix update_config to work with non-AGIPD and add REMI

3.10.1

  • [JUNGFRAU][CORRECT] Using calcat interface

  • [JUNGFRAU][CORRECT][DARK] Extend accepted detectors based on substrings of karabo_id

  • [LPD Mini][CORRECT][DARK] Initial work on LPD Mini notebooks

  • [PNCCD][CORRECT] Avoid raising a CalCat error while retrieving metadata for missing gain constant from DB

3.10.0

  • [ePix100][Correct] Avoid including histogram calculator with empty array

  • [ePix100][Correct] Remove pre notebook

  • [pnCCD][Correct] Using calcat interface

  • [pnCCD][Correct] Record fragment file and remove pre correction notebook

  • [ePix100][pnCCD][Correct] Display creation time for retrieved constants

  • [AGIPD][Correct] error out only if all sources are empty

  • [AGIPD][Correct] exit notebook if no correction files are found for the selected sequence

  • [DSSC][DARK] Group all slow data to the same aggregator

  • [JUNGFRAU][Correct] Use DataCollection.from_paths for reading JF CORR files for plots

  • Support for saving metadata fragments & merging into calibration_metadata.yml

  • [REMI] Save pulse amplitudes during discrimination

  • [REMI] Make plots robust against no edges or no hits in data

  • [Webservice] Add script to check run in webservice DB

  • [Webservice] Fix database lock timeouts

3.9.2

  • [AGIPD] Adding back overwrite parameter
  • [AGIPD] Another hotfix for experimenting DAQ filtering at SPB

3.9.1

  • Hotfix version for 3.9.0
  • [AGIPD] LitFrameFinder bug fix for experimenting DAQ filtering at SPB

3.9.0

  • [GH2][CORRECT]CALCAT interface.
  • [ePix100][CORRECT]CALCAT interface.
  • [pnCCD][CORRECT]Use DataFile to store aligned corrected data.
  • [GH2][CORRECT]Use DataFile to store aligned corrected data.

  • [ePix100][CORRECT]Use DataFile to store aligned corrected data.

  • [AGIPD][SlopesFF][CORRECT]Add the deviation for all possible memory cells and enable correction using these FF constants with more memory cells.

  • https://git.xfel.eu/calibration/pycalibration/-/merge_requests/806
  • https://git.xfel.eu/calibration/pycalibration/-/merge_requests/613
  • [AGIPD][CORRECT]Fix checking ccv_variant condition for AGIPD.
  • [DSSC][Dark]Remove unused instrument parameter from DSSC dark notebook
  • Update nbparameterise to 0.6.
  • Update EXtra-redu to version 0.0.7

3.8.1

  • Hotfix version for 3.8.0 with
  • update extra_redu version

3.8.0

  • [AGIPD][CORRECT] Clean up before AGIPD calcalt_interface changes
  • [AGIPD][FF] Styling modification for both FF notebooks:1st nb cell, and removing unneeded imports
  • [LPD] uses memory cell order as a condition for constants
  • [LPD][Correct] Don't pass default snapshot_at=None to calibration_client
  • [LPD][JF][DataFile] Add parallel dataset compression
  • [JF] Replace strixel cython code by NumPy implementation
  • [EPIX100][CORRECT] Include gain and charge sharing to corrected data.
  • New CALCAT interface
  • Store CalCat requests/responses for reproducibility
  • Update to calibration_client 11.2.0

3.7.6

  • [AGIPD][CORRECT] Fix common mode correction: array reshaping and hardcoded 256 trains per file.
  • [AGIPD][LFF] Improve lit frame selection
  • [AGIPD][CORRECT] Fix common mode correction
  • Array reshaping.

  • Hardcoded 256 trains per file.

  • [AGIPD][LFF] Improve lit frame selection:

  • fix the misalignment of selection by trains.

  • introduce super pattern selection.
  • guess missed patterns for patterns that repeat with constant step.
  • shows patterns that repeat with a constant step as one line in the report.

3.7.5

16-11-2022

3.7.4

15-11-2022

  • JUNGFRAU Only emit a warning if all sequences of a job are empty
  • JUNGFRAU Fix storing multiple ROIs for a single module
  • JUNGFRAU Replicate a raw dataset in processed data for legacy analysis code

3.7.3

10-11-2022

  • DataFile Add support for older INDEX and METADATA versions.
  • LPD Fix histogram plotting if selected cell's data is identical for trains.
  • JUNGFRAU Avoid unclear error messages when the run has no trains.
  • AGIPD Change notebook-default setting of max-task-per-worker to 1.
  • REMI Allow pulse separation in REMI_DLD reconstruction to ignore FEL pulses.
  • webservice Fix timestamps in the overview page.
  • webservice Check run type more directly in myMdC response.

3.7.2

28-10-2022

  • JUNGFRAU Add support for strixel sensors
  • JUNGFRAU Shorter summary notebook for darks by default
  • JUNGFRAU Always align corrected data properly
  • JUNGFRAU Harden against 0 adc data in dark characterization
  • REMI Add support for quad nodes and various pulse separation improvements
  • AGIPD Allow manual photon energy input for photonization and add consistency plots
  • Grant priority partitions for set-up time

3.7.1

14-10-2022

  • EPIX100 Add Common mode corrections to dark characterization
  • EPIX100 Use a single value for temperature
  • JUNGFRAU Fix bad pixel masking
  • JUNGFRAU Limit number of plotted trains to 500 during corrections
  • AGIPD Don't fail when there are no trains in sequence file
  • REMI Use DataFile API and support asymmetric pump-probe patterns
  • Skip corrections for runs which match specific types

3.7.0

07-09-2022

3.6.4

20-08-2022

  • AGIPD Significant improvements to AGIPG LitFrameFinder implementation, including offline support.
  • AGIPD Fix misalignment of data and gain/mask when common mode is used with manual pulse slicing.

3.6.3

23-08-2022

3.6.2

15-08-2022

3.6.1

26-07-2022

3.6.0

20-07-2022

3.5.5

15-06-2022

  • AGIPD - CORRECT Expose max tasks per pool worker.

3.5.4

13-06-2022

  • AGIPD Convert bias_voltage parameter condition to integer in cal_tools.
  • LPD Fix correcting a single pulse.
  • LPD VCXI require 4 modules.

3.5.3

19-05-2022

  • LPD - CORRECT Optionally create virtual CXI files
  • LPD - CORRECT Expose max-nodes parameter
  • AGIPD Replace gain_choose_int by fused types
  • Fix missing install of restful_config.yaml
  • Fix use of xfel-calibrate --skip-report

3.5.2

16.05.2022

  • LPD - CORRECT New correction notebook for LPD
  • New [files] module to write European XFEL HDF5 corrected data files.

3.5.1

05-04-2022

3.5.0

01-03-2022

3.4.3

20-10-2021

  • Update pyDetLib tag.
  • Add explicit dependencies on matplotlib, scipy.
  • Remove outdated matplotlib rcParams setting.
  • Update EXtra-geom to 1.6.
  • Remove cluster_profile parameter from notebooks which don't use it.
  • Fix checking availability for the concurrency parameter.
  • Fix launching work directly (not via Slurm).
  • Fix sphinx-rep temp folder recreation, if sphinx-rep already existed.
  • Fix missing string conversion for slurm-scheduling argument.
  • Fix title reports for multiple detectors per run folder.
  • Append to .out files for preemptable finalize job.
  • AGIPD CORRECT Reuse previously found constants.
  • AGIPD Fix missing memory cell index in SlopesPC constant sanitization.
  • AGIPD Only use bad pixels from darks in agipdutils.baseline_correct_via_stripes.
  • AGIPD DARK Use function to get list of karabo_da from run for making Slurm jobs.
  • EPIX100 - CORRECT Set absolute_gain to false if relative gain was not retrieved.
  • JUNGFRAU Fix running for multiple modules and flip logic for do_relative_gain.
  • JUNGFRAU Style changes for Dark and Correct notebooks.
  • REMI Add notebook to reconstruct detector hits from raw data.
  • webservice Check run migration status using MyMDC.
  • Resolve \"Skip ZMQ tests if zmq connection for calibration DB not available\".
  • Reproducibility, step 1.

3.4.2

17-09-2021

  • Remove driver=core from all notebook
  • webservice Make use of Dynaconf for managing secrets.
  • webservice Make use of dedicated slurm partitions.
  • webservice Handle missing migration information (missing user.status fattr).
  • webservice Implement, raise, and catch, migration errors to send mdc messages.
  • webservice Simplify handling of user notebook paths.
  • webservice Update princess to 0.4 (use Unix sockets).
  • webservice Update MyMDC with begin and end times.
  • webservice create output folder before copying slow data.
  • AGIPD CORRECT read acq_rate from slow data.
  • AGIPD - CORRECT Set default memory cells to 352.
  • AGIPD CORRECT Set maximum pulses to correct based on file content.
  • AGIPD - FF Correctly label legends in figures.
  • AGIPD - FF Add HIBEF AGIPD500K and fix some issue with retrieval of conditions.
  • JUNGFRAU Add Gain setting to Jungfrau notebooks.
  • JUNGFRAU Fix max gain plot in LPD correct notebook
  • JUNGFRAU DARK Clearer error message for Jungfrau Dark notebooks no suitable files are found
  • LPD CORRECT Fix max gain plot.
  • EPIX100 CORRECT Solve conflict between gain correction and clustering

3.4.1

16-07-2021

  • Update h5py to 3.3
  • Stop execution on notebook errors
  • AGIPD Add integration time as operating condition to all notebooks
  • webservice Add blocklist pattern when copying untouched files in webservice.
  • webservice Expose dark configurations in update_config.py
  • Fix MetadataClient.get_proposal_runs arguments call.
  • Fix Use snapshot for injecting constants for old PDU mappings
  • Fix the old time-summary (creation time for retrieved constants)
  • Update documentation notes on venv installation
  • Ignore all .so files in gitignore

3.4.0

28-06-2021

  • Update to Python 3.8.
  • Bump numpy to 1.20.3 and remove fabio.
  • remove PyQT dependency.
  • Disable dark requests from serve overview.
  • Update report upload parameter key.
  • Override locale to always use UTF-8.
  • Assorted cleanup of xfel-calibrate.
  • Fix pre-commit.
  • Use argparse only if name is main, call main with args dict.
  • webservice Use full hostname for webservice overview.
  • webservice Show clearer messages when running webservice in sim mode.
  • webservice Fix filename lineno and typos in webservice logs.
  • webservice Fix creating an extra run folder in run output folder.
  • AGIPD Parallelize gain/mask compression for writing corrected AGIPD files.
  • AGIPD - DARK Fix processing empty sequence files.
  • AGIPD - PC - FF Update notebooks with new CALCAT mapping.
  • AGIPD - JUNGFRAU Use all available sequences for processing darks for AGIPD and Jungfrau.
  • AGIPD - LPD - DSSC Fix retrieve old constants for comparison for modular detectors.
  • LPD Fix data paths in LPD notebook.
  • REMI Fix user notebook path for REMI correct notebook provisionally.
  • EPIX100 - CORRECT Add Common mode correction.
  • Fix plotting-related warnings.
  • Test update config.
  • Test get_from_db and send_to_db.