Output formats

Output axis order

By default, calng correction devices output image data axes (frames, pixels X, and pixels Y) in the same order as they are received from the detector / DAQ. In case online analysis requires a different order, output axis order parameter can be used to specify an arbitrary different order. On a technical level, this order is achieved via numpy.transpose or equivalent before data is sent on the main output.

Keep in mind that optional stacking happens downstream on a (group or full) matcher. The module axis added by stacking is determined by the stackingAxis parameter in the stacking configuration.

Stacking

By default, a TrainMatcher sends matched data per source. Some software may need data to be stacked (or interleaved): for instance, image data for multiple modules arranged within a single contiguous array. To this end, the ShmemTrainMatcher extends the regular TrainMatcher by allowing for stacking of arbitrary keys from arbitrary sources. Stacking can either work across sources - stacking arrays with the same name from multiple sources - or keys - stacking arrays multiple keys from one source.

In the array stacking table, each row specifies a stacking rule with the following fields:

Select: Whether or not to apply this rule.
Source pattern: In sources mode: regular expression used to select the subset of sources from which to take the desired key for stacking. This regular expression is matched against every source listed in the source table. The order in which the contents from individual sources appear in the stacked output array follows the order from the source table.; In keys mode: concrete name of the source to perform stacking on.
Key pattern: In sources mode: concrete name of key holding array to stack across sources. Note that the data type and shape of the array under this key should be consistent for all sources in the stacking pattern.; In keys mode: regular expression used to select the subset of keys to stack from the source. This regular expression is matched against every path in the data hash from the source. The order in the stacked output array follows the order of iteration over the paths of the data hash. Note that all arrays under included keys should have the same data type and shape.
Replacement: In sources mode: name of new source in output which will contain the stacked array. The stacked array will be in a key with the same name as the key pattern.; In keys mode: name of new key in output which will contain the stacked array. This key will be added to the forwarded output from the source given by the source pattern.
Group type: Specifies whether to stack in sources (supported) or keys (not yet field tested) mode.
Merge method: Specifies whether to stack or interleave data. Stacking works just like np.stack; arrays are appended along the desired axis and this adds another axis to the output. Stacking is for example used to combine multiple detector modules into one array with a module dimension for tools like OnDA which expect this format.; Interleaving does not add an additional axis as it instead interleaves the arrays along the desired axis. The output data has shape similar to the output of np.concatenate (though the underlying data layout is equivalent to stacking). When interleaving N sources, source i will be placed along the specified axis in positions i::N. Interleaving is for example used to combine the two receiver data streams for GOTTHARD-II 25 μm modules.
Axis: Specifies the axis along which to stack (see numpy.stack) or interleave. In the use case of stacking detector data, this parameter will be the module axis in the output.

Example: stacking full AGIPD1M for OnDA at SPB

These settings are at the time of writing found on SPB_DET_AGIPD1M-1/CALNG/FULL_MATCHER in the AGIPD_ONLINE_PREVIEW_NEXTGEN project. The sources for the matcher forward each module individually via the group matcher outputs:

SPB_DET_AGIPD1M-1/DET/0CH0:xtdf@SPB_DET_AGIPD1M-1/CALNG/MATCH_G1:output
SPB_DET_AGIPD1M-1/DET/1CH0:xtdf@SPB_DET_AGIPD1M-1/CALNG/MATCH_G1:output
...
SPB_DET_AGIPD1M-1/DET/15CH0:xtdf@SPB_DET_AGIPD1M-1/CALNG/MATCH_G4:output

Note that it is important that the sources are listed correctly in this order in the sources table; recall that the stacking order is informed by this table. The stacking configuration is set to stack the image data from all the module sources along axis 1 as OnDA expects the axis order to be: frame, module, slow scan, fast scan.

Source pattern: SPB_DET_AGIPD1M-1/DET/\d+CH0:xtdf
Key pattern: image.data
Replacement: SPB_DET_AGIP1M-1/DET/STACKED:xtdf
Group type: sources
Axis: 1

Example: interleaving two GOTTHARD-II modules in 25 μm configuration

At the time of writing, there are no GOTTHARD-II 25 μm detectors deployed, so we will use placeholder Karabo IDs instead. Suppose we have the two receiver channels in the sources table:

INSTRUMENT/DET/MODULE-1:daqOutput
INSTRUMENT/DET/MODULE-2:daqOutput

Then the following row would go in the "Array stacking" table:

Source pattern: INSTRUMENT/DET/MODULE-\d:daqOutput
Key pattern: data.adc
Replacement: INSTRUMENT/DET/25UM-MODULE (arbitrary name, should fit online analysis expectations)
Group type: sources
Merge method: interleave
Axis: 1 (interleaving pixels)

With this setup, the new source INSTRUMENT/DET/25UM-MODULE will contain data.adc with shape (2700, 2560) (with each underlying detector module having produced (2700, 1280)).

One group, all frames

This is the output one can get from group matchers. It is by default per source, though image data can be stacked.

If only a subset of modules are needed for online analysis, this is the recommended data stream to use. Refer to group configuration for notes on how to rearrange modules between groups.

All modules, all frames

If data from the full detector is needed for analysis, one can set up a full matcher. This pattern includes forwarding data from each of the group matchers - due to network limits, careful throttling will most likely be necessary.

Image data can optionally be stacked (relevant when running OnDA, for instance).

All modules, one frame, assembled

This is the output from preview assemblers. They provide fully assembled preview single-frame images for the full detector. Although they only send up two 2 Hz of preview to the GUI, they typically received and process at 10 Hz and on a separate output provide the 10 Hz stream. Refer to preview assembler for details on configuration and outputs.

Keep in mind that all-trains, all-frames, all-modules data is generally not possible to gather on one node for fast detectors, assembled or not. Hence the trade-off for the preview data flow is to reduce data to a single frame per train.