Release Notes ============= 3.5.5 ----- 15-06-2022 - [AGIPD][CORRECT] Expose max tasks per pool worker. 3.5.4 ----- 13-06-2022 - [AGIPD] Convert bias_voltage parameter condition to integer in cal_tools. - [LPD] Fix correcting a single pulse. - [LPD] VCXI require 4 modules. 3.5.3 ----- 19-05-2022 - [LPD][CORRECT] Optionally create virtual CXI files - [LPD][CORRECT] Expose max-nodes parameter - [AGIPD] Replace gain_choose_int by fused types - Fix missing install of restful_config.yaml - Fix use of xfel-calibrate --skip-report 3.5.2 ----- 16.05.2022 - [LPD][CORRECT] New correction notebook for LPD - New `files` module to write European XFEL HDF5 corrected data files. 3.5.1 ----- 05-04-2022 - Calibration Constant version's new `Variant` file attribute. To indicate method of handling the constant post retrieval. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/619 - Epix100 dark Badpixels Map. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/637 - `skip-plots` flag to finish correction before plotting. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/635 - First trainId's timestamp as RAW data creation_time, if there is myMDC connection. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/630 - AGIPD correction can correct one cellId without plotting errors. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/642 - Fixed mode relative gain constants in Jungfrau can be retrieved. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/639 - Only instrument source is selected to check number of trains to dark process. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/636 - AGIPD trains for dark processing is selected for each module individually. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/653 - Produce report after trying to correct AGIPD run with no images. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/641 - AGIPD's bias voltage for AGIPD1M is read from slow data. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/647 - Removed psutil dependency. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/653 - Update Pasha to 0.1.1 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/638 3.5.0 ----- 01-03-2022 - Updating Correction and dark notebooks for JUNGFRAU: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/518 - Updating Correction and dark notebooks for AGIPD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535 - Updating Correction and dark notebooks for PnCCD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/559 - Updating Correction and dark notebooks for ePix100: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/500 * EXtra-data is integrated to read files in pycalibration for AGIPD, JUNGFRAU, ePix100, and PnCCD. Dark and Correction notebooks. * Pasha is now used for processing data for JUNGFRAU, ePix100 and PnCCD. * pyDetLib correction functions were removed (except for common-mode correction). * `db-module` is useless now for JUNGFRAU, ePix100 and PnCCD. Some parameters were updated in dark and correction notebooks for the mentioned detectors. - `gain_mode` and burst mode are now available for JUNGFRAU. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/591 - JUNGFRAU has now a new badpixel value, `WRONG_GAIN_VALUE`. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/591 - Pass through available for testing in-progress ORCA service. https://git.xfel.eu/detectors/pycalibration/-/merge_requests?scope=all&state=merged&search=orca - Non-calibrated RAW h5files are no longer copied. - High priority partitions (`upex-high`and `upex-middle`) are used for runs from ACTIVE and READY proposals, only. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/628 - Supporting to disable LPD Correction through the webservice. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/629 - Compatibility for old DAQ files for REMI is added. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/607 - server-overview refactors. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/593 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/589 - AGIPD correction notebook support AgipdLitFrameFinder device. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/603 - Parsing code arguments in xfel-calibrate is refactored. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/575 - skip-plots option for AGIPD. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/581 - Native implementation for transposition of constants AGIPD. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/580 - Trains for AGIPD can be selected for correction. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/574 - Skip report flag in xfel-calibrate. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/579 - Fix ReadTheDocs. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/448 - Fix error reporting for re-injecting the same CCV. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/627 - Fix AGIPD for legacy runs without `gain_mode`. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/617 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/626 - Pinning markupsafe version 2.0.1 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/631 - Pinning psutil 5.9.0 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535 - Updating Extra-data to 1.9.1 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535 - Updating h5py to 3.5.0 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/602 3.4.3 ----- 20-10-2021 - Update pyDetLib tag. - Add explicit dependencies on matplotlib, scipy. - Remove outdated matplotlib rcParams setting. - Update EXtra-geom to 1.6. - Remove cluster_profile parameter from notebooks which don't use it. - Fix checking availability for the concurrency parameter. - Fix launching work directly (not via Slurm). - Fix `sphinx-rep` temp folder recreation, if sphinx-rep already existed. - Fix missing string conversion for slurm-scheduling argument. - Fix title reports for multiple detectors per run folder. - Append to .out files for preemptable finalize job. - [AGIPD] [Correct] Reuse previously found constants. - [AGIPD] Fix missing memory cell index in SlopesPC constant sanitization. - [AGIPD] Only use bad pixels from darks in agipdutils.baseline_correct_via_stripes. - [AGIPD] [Dark] Use function to get list of karabo_da from run for making Slurm jobs. - [EPIX100][CORRECT] Set absolute_gain to false if relative gain was not retrieved. - [JUNGFRAU] Fix running for multiple modules and flip logic for do_relative_gain. - [JUNGFRAU] Style changes for Dark and Correct notebooks. - [REMI] Add notebook to reconstruct detector hits from raw data. - [webservice] Check run migration status using MyMDC. - Resolve "Skip ZMQ tests if zmq connection for calibration DB not available". - Reproducibility, step 1. 3.4.2 ----- 17-09-2021 - Remove driver=core from all notebook - [webservice] Make use of Dynaconf for managing secrets. - [webservice] Make use of dedicated slurm partitions. - [webservice] Handle missing migration information (missing user.status fattr). - [webservice] Implement, raise, and catch, migration errors to send mdc messages. - [webservice] Simplify handling of user notebook paths. - [webservice] Update princess to 0.4 (use Unix sockets). - [webservice] Update MyMDC with begin and end times. - [webservice] create output folder before copying slow data. - [AGIPD] [CORRECT] read acq_rate from slow data. - [AGIPD][CORRECT] Set default memory cells to 352. - [AGIPD] [CORRECT] Set maximum pulses to correct based on file content. - [AGIPD] [FF] Correctly label legends in figures. - [AGIPD] [FF] Add HIBEF AGIPD500K and fix some issue with retrieval of conditions. - [Jungfrau] Add Gain setting to Jungfrau notebooks. - [Jungfrau] Fix max gain plot in LPD correct notebook - [JUNGFRAU] [DARK] Clearer error message for Jungfrau Dark notebooks no suitable files are found - [LPD] [CORRECT] Fix max gain plot. - [EPIX100] [CORRECT] Solve conflict between gain correction and clustering 3.4.1 ----- 16-07-2021 - Update h5py to 3.3 - Stop execution on notebook errors - [AGIPD] Add integration time as operating condition to all notebooks - [webservice] Add blocklist pattern when copying untouched files in webservice. - [webservice] Expose dark configurations in update_config.py - Fix MetadataClient.get_proposal_runs arguments call. - Fix Use snapshot for injecting constants for old PDU mappings - Fix the old time-summary (creation time for retrieved constants) - Update documentation notes on venv installation - Ignore all .so files in gitignore 3.4.0 ----- 28-06-2021 - Update to Python 3.8. - Bump numpy to 1.20.3 and remove fabio. - remove PyQT dependency. - Disable dark requests from serve overview. - Update report upload parameter key. - Override locale to always use UTF-8. - Assorted cleanup of xfel-calibrate. - Fix pre-commit. - Use argparse only if name is main, call main with args dict. - [webservice] Use full hostname for webservice overview. - [webservice] Show clearer messages when running webservice in sim mode. - [webservice] Fix filename lineno and typos in webservice logs. - [webservice] Fix creating an extra run folder in run output folder. - [AGIPD] Parallelize gain/mask compression for writing corrected AGIPD files. - [AGIPD][DARK] Fix processing empty sequence files. - [AGIPD][PC][FF] Update notebooks with new CALCAT mapping. - [AGIPD][JUNGFRAU] Use all available sequences for processing darks for AGIPD and Jungfrau. - [AGIPD][LPD][DSSC] Fix retrieve old constants for comparison for modular detectors. - [LPD] Fix data paths in LPD notebook. - [REMI] Fix user notebook path for REMI correct notebook provisionally. - [EPIX][CORRECT] Add Common mode correction. - Fix plotting-related warnings. - Test update config. - Test get_from_db and send_to_db.