How to’s¶

Loading run data¶

Reading the bunch pattern¶

bunch pattern decoding.

Extracting peaks from digitizers¶

How to extract peaks from digitizer traces.

Determining the FEL or OL beam size and the fluence¶

Knife-edge scan and fluence calculation.

Finding time overlap by transient reflectivity¶

Transient reflectivity of the optical laser measured on a large bandgap material pumped by the FEL is often used at SCS to find the time overlap between the two beams. The example notebook

Transient reflectivity measurement

shows how to analyze such data, including correcting the delay by the bunch arrival monitor (BAM).

DSSC¶

DSSC data binning¶

In scattering experiment one typically wants to bin DSSC image data versus time delay between pump and probe or versus photon energy. After this first data reduction steps, one can do azimuthal integration on a much smaller amount of data.

The DSSC data binning procedure is based on the notebook Dask DSSC module binning. It performs DSSC data binning against a coordinate specified by xaxis which can be nrj for the photon energy, delay in which case the delay stage position will be converted in picoseconds and corrected but the BAM, or another slow data channel. Specific pulse pattern can be defined, such as:

['pumped', 'unpumped']

which will be repeated. XGM data will also be binned similarly to the DSSC data.

Since this data reduction step can be quite time consuming for large datasets, it is recommended to launch the notebook via a SLURM script. The script can be downloaded from scripts/bin_dssc_module_job.sh and reads as:

#!/bin/bash
#SBATCH -N 1
#SBATCH --partition=exfel
#SBATCH --time=12:00:00
#SBATCH --mail-type=END,FAIL
#SBATCH --output=logs/%j-%x.out

while getopts ":p:d:r:k:m:x:b:" option
do
    case $option in
        p) PROPOSAL="$OPTARG";;
        d) DARK="$OPTARG";;
        r) RUN="$OPTARG";;
        k) KERNEL="$OPTARG";;
        m) MODULE_GROUP="$OPTARG";;
        x) XAXIS="$OPTARG";;
        b) BINWIDTH="$OPTARG";;
        \?) echo "Unknown option"
            exit 1;;
        :) echo "Missing option for input flag"
           exit 1;;
    esac
done

# Load xfel environment
source /etc/profile.d/modules.sh
module load exfel exfel-python

echo processing run $RUN
PDIR=$(findxfel $PROPOSAL)
PPROPOSAL="p$(printf '%06d' $PROPOSAL)"
RDIR="$PDIR/usr/processed_runs/r$(printf '%04d' $RUN)"
mkdir $RDIR

NB='Dask DSSC module binning.ipynb'

# kernel list can be seen from 'jupyter kernelspec list'
if [ -z "${KERNEL}" ]; then
	  KERNEL="toolbox_$PPROPOSAL"
fi

python -c "import papermill as pm; pm.execute_notebook(\
  '$NB', \
  '$RDIR/output$MODULE_GROUP.ipynb', \
  kernel_name='$KERNEL', \
  parameters=dict(proposalNB=int('$PROPOSAL'), \
                  dark_runNB=int('$DARK'), \
                  runNB=int('$RUN'), \
                  module_group=int('$MODULE_GROUP'), \
                  path='$RDIR/', \
                  xaxis='$XAXIS', \
                  bin_width=float('$BINWIDTH')))"

It is launched with the following:

sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 0 -x delay -b 0.1
sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 1 -x delay -b 0.1
sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 2 -x delay -b 0.1
sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 3 -x delay -b 0.1

where 2719 is the proposal number, 180 is the dark run number, 179 is the run nummber and 0, 1, 2 and 3 are the 4 module group, each job processing a set of 4 DSSC module, delay is the bin axis and 0.1 is the bin width.

The result will be 16 *.h5 files, one per module, saved in the folder specified in the script, a copy of which can be found in the scripts folder in the toolbox source. This files can then be loaded and combined with:

import xarray as xr
data = xr.open_mfdataset(path + '/*.h5', parallel=True, join='inner')

DSSC azimuthal integration¶

Azimuthal integration can be performed with pyFAI which can utilize the hexagonal pixel shape information from the DSSC geometry to split the intensity in a pixel in the bins covered by it. An example notebook Azimuthal integration of DSSC with pyFAI.ipynb is available.

A second example notebook DSSC scattering time-delay.ipynb demonstrates how to:

refine the geometry such that the scattering pattern is centered before azimuthal integration
perform azimuthal integration on a time delay dataset with xr.apply_ufunc for multiprocessing.
plot a two-dimensional map of the scattering change as function of scattering vector and time delay
integrate certain scattering vector range and plot a time trace

DSSC fine timing¶

When DSSC is reused after a period of inactivity or when the DSSC gain setting use a different operation frequency the DSSC fine trigger delay needs to be checked. To analysis runs recorded with different fine delay, one can use the notebook DSSC fine delay with SCS toolbox.ipynb.

DSSC quadrant geometry¶

To check or refined the DSSC geometry or quadrants position, the following notebook can be used DSSC create geometry.ipynb.

Legacy DSSC binning procedure¶

Most of the functions within toolbox_scs.detectors can be accessed directly. This is useful during development, or when working in a non-standardized way, which is often neccessary during data evaluation. For frequent routines there is the possibility to use dssc objects that guarantee consistent data structure, and reduce the amount of recurring code within the notebook.

bin data using toolbox_scs.tbdet -> to be documented.
bin data using the DSSCBinner.
post processing, data analysis -> to be documented

Photo-Electron Spectrometer (PES)¶

Basic analysis of PES spectra.

BOZ: Beam-Splitting Off-axis Zone plate analysis¶

The BOZ analysis consists of 4 notebooks and a script. The first notebook BOZ analysis part I.a Correction determination is used to determine all the necessary correction, that is the flat field correction from the zone plate optics and the non-linearity correction from the DSSC gain. The inputs are a dark run and a run with X-rays on three broken or empty membranes. For the latter, an alternative is to use pre-edge data on an actual sample. The result is a JSON file that contains the flat field and non-linearity correction as well as the parameters used for their determination such that this can be reproduced and investigated in case of issues. The determination of the flat field correction is rather quick, few minutes and is the most important correction for the change in XAS computed from the -1st and +1st order. For quick correction of the online preview one can bypass the non-linearity calculation by taking the JSON file as soon as it appears. The determination of the non-linearity correction is a lot longer and can take some 2 to 8 hours depending on the number of pulses in the train. For this reason it is possible to use a script that can be downloaded from scripts/boz_parameters_job.sh and reads as:

#!/bin/bash
#SBATCH -N 1
#SBATCH --partition=allgpu
#SBATCH --constraint=V100
#SBATCH --time=2:00:00
#SBATCH --mail-type=END,FAIL
#SBATCH --output=logs/%j-%x.out

ROISTH='1'
SATLEVEL='500'
MODULE='15'

while getopts ":p:d:r:k:g:t:s:m:" option
do
    case $option in
        p) PROPOSAL="$OPTARG";;
        d) DARK="$OPTARG";;
        r) RUN="$OPTARG";;
        k) KERNEL="$OPTARG";;
        g) GAIN="$OPTARG";;
        t) ROISTH="$OPTARG";;
        s) SATLEVEL="$OPTARG";;
        m) MODULE="$OPTARG";;
        \?) echo "Unknown option"
            exit 1;;
        :) echo "Missing option for input flag"
           exit 1;;
    esac
done

# Load xfel environment
source /etc/profile.d/modules.sh
module load exfel exfel-python

echo processing run $RUN
PDIR=$(findxfel $PROPOSAL)
PPROPOSAL="p$(printf '%06d' $PROPOSAL)"
RDIR="$PDIR/usr/processed_runs/r$(printf '%04d' $RUN)"
mkdir $RDIR

NB='BOZ analysis part I.a Correction determination.ipynb'

# kernel list can be seen from 'jupyter kernelspec list'
if [ -z "${KERNEL}" ]; then
	KERNEL="toolbox_$PPROPOSAL"
fi

python -c "import papermill as pm; pm.execute_notebook(\
  '$NB', \
  '$RDIR/output.ipynb', \
  kernel_name='$KERNEL', \
  parameters=dict(proposal=int('$PROPOSAL'), \
                  darkrun=int('$DARK'), \
                  run=int('$RUN'), \
                  module=int('$MODULE'), \
                  gain=float('$GAIN'), \
                  rois_th=float('$ROISTH'), \
                  sat_level=int('$SATLEVEL')))"

It uses the first notebook and is launched via slurm:

sbatch ./boz_parameters_job.sh -p 2937 -d 615 -r 614 -g 3

where 2937 is the proposal run number, where 615 is the dark run number, 614 is the run on 3 broken membranes and 3 is the DSSC gain in photon per bin. The proposal run number is defined inside the script file.

The second notebook BOZ analysis part I.b Correction validation can be used to check how well the calculated correction still work on a characterization run recorded later, i.e. on 3 broken membrane or empty membranes.

The third notebook BOZ analysis part II.1 Small data then use the JSON correction file to load all needed corrections and process an run, saving the rois extracted DSSC as well as aligning them to photon energy and delay stage in a small data h5 file.

That small data h5 file can then be loaded and the data binned to compute a spectrum or a time resolved XAS scan using the fourth and final notebook BOZ analysis part II.2 Binning

Point detectors¶

Detectors that produce one point per pulse, or 0D detectors, are all handled in a similar way. Such detectors are, for instance, the X-ray Gas Monitor (XGM), the Transmitted Intensity Monitor (TIM), the electron Bunch Arrival Monitor (BAM) or the photo diodes monitoring the PP laser.

HRIXS¶

Analyzing HRIXS data

Viking spectrometer¶

Analysis of Viking spectrometer data

SLURM, sbatch, partition, reservation¶

Scripts launched by sbatch command can employ magic cookie with #SBATCH to pass options SLURM, such as which partition to run on. To work, the magic cookie has to be at the beginning of the line. This means that:

to comment out a magic cookie, adding another “#” before it is sufficient
to comment a line to detail what the option does, it is best practice to put the comment on the line before

Reserved partition are of the form “upex_003333” where 3333 is the proposal number. To check what reserved partition are existing, their start and end date, one can ssh to max-display and use the command sview.

To use a reserved partition with sbatch, one can use the magic cookie

#SBATCH --reservation=upex_003333

instead of the usual

#SBATCH --partition=upex