Karabo Bridge¶

Clients¶

Python Client¶

The karabo_bridge Python package provides a client interface to receive data from the Karabo bridge.

Karabo bridge client for Karabo pipeline data.

This class can request data to a Karabo bridge server. Create the client with::

from karabo_bridge import Client
krb_client = Client("tcp://153.0.55.21:12345")

then call data = krb_client.next() to request next available data container.

Parameters:

Name	Type	Description	Default
`endpoint`	`str`	server socket you want to connect to (only support TCP socket).	required
`sock`	`str`	socket type - supported: REQ, SUB.	`'REQ'`
`ser`	`str, DEPRECATED`	Serialization protocol to use to decode the incoming message (default is msgpack) - supported: msgpack.	`'msgpack'`
`context`	`zmq.Context`	To run the Client's sockets using a provided ZeroMQ context.	`None`
`timeout`	`int`	Timeout on :meth:`next` (in seconds) Data transfered at the EuXFEL for Mega-pixels detectors can be very large. Setting a too small timeout might end in never getting data. Some example of transfer timing for 1Mpix detector (AGIPD, LPD): 32 pulses per train (125 MB): ~0.1 s 128 pulses per train (500 MB): ~0.4 s 350 pulses per train (1.37 GB): ~1 s	`None`

Raises:

Type	Description
`NotImplementedError`	if socket type or serialization algorithm is not supported.
`ZMQError`	if provided endpoint is not valid.

`next()` ¶

Request next data container.

This function call is blocking.

Returns:

Name	Type	Description
`data`	`dict`	The data for this train, keyed by source name.
`meta`	`dict`	The metadata for this train, keyed by source name. This dictionary is populated for protocol version 1.0 and 2.2. For other protocol versions, metadata information is available in `data` dict.

Raises:

Type	Description
`TimeoutError`	If timeout is reached before receiving data.

C++ Client¶

The karabo bridge C++ client provides a client interface to receive data from karabo bridge.

Example Usage for accessing image data from the stream.

#include "kb_client.hpp"
using namespace std;

int main (int argc, char* argv[]) {
    std::string addr;

    if (argc >= 2) addr = argv[1];

    else throw std::invalid_argument("Server address required!");

    karabo_bridge::Client client;
    client.connect(addr);

    auto livedata = client.next()

    for (auto it = livedata.begin(); it != livedata.end(); ++it)
    {
        if (it -> first == "SPB_DET_AGIPD1M-1/DET/detector")
        {
        karabo_bridge::kb_data data(it->second);
        auto images = data.array["image.data"].as<std::vector<uint32_t>>();
        }
    }

}

More Information

Installation and usage information available at the Karabo Bridge C++ Repository

Data Container¶

The data object returned by :meth:Client.next is a tuple of two nested dictionaries, holding data and metadata respectively.

The first level of keys in both dictionaries are source names, which typically correspond to Karabo device names.

In the data dictionary, each source has a dictionary representing a flattened Karabo hash, with dots delimiting successive key levels. The values are either basic Python types such as strings and floats, or numpy arrays.

In the metadata dictionary, each source has a dictionary with keys: source, timestamp, timestamp.tid, timestamp.sec, timestamp.frac and ignored_keys, which is a list of strings identifying keys which were filtered out of the data by configuration options on the bridge server.

(
    {   # Metadata
        'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf': {
            'source': 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf',
            'timestamp': 1526464869.4109755,
            # Timestamps can have attosecond resolution: 18 fractional digits
            'timestamp.frac': '410975500000000000',
            'timestamp.sec': '1526464869',
            'timestamp.tid': 10000000001,
            'ignored_keys': []
        },
    },
    {   # Data
        'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf': {
            'image.data': array(...),
            'header.pulseCount': 64,
            # ... other keys
        }
    }
)

Karabo Bridge Recorder¶

Karabo bridge recorder is a utility to record and replay data sent through the Karabo bridge.

This is for software engineering purposes, to conveniently test code with the exact data format sent by the bridge. We maintain a simulator for the same purpose, but sometimes the data format changes. Making a new recording is much quicker than updating the simulator.

Do not use this to store scientific data for later access! The file format is undocumented, and we will not support such use. EuXFEL has much better ways to store data; ask your contact people if you're not sure how to record or access data.

Usage:

# Record some data (creates a new folder)
karabo-bridge-record tcp://10.253.0.51/4510

# Play back recorded data
karabo-bridge-replay tcp://0.0.0.0:4545 karabo-bridge-recording-2019-02-19T16.49.25Z/

It is recommended to use this application from pre-installed path on the Online and Offline Clusters.

Karabo Bridge Protocol¶

The Karabo bridge protocol is based on ZeroMQ and msgpack.

The connection can use one of two kinds of ZeroMQ sockets. The client and the server must be configured to use matching socket types.

REQ-REP: The client sends the raw ASCII bytes next (not using msgpack) as a request, and the reply is a message as described below. This is the default option.
PUB-SUB: The client subscribes to messages published by the server. This is simpler, but can create a lot of network traffic.

Note

The data messages are documented here in terms of msgpack. The code can also use Python's pickle serialisation format, but since this is Python specific, it is not recommended for new code.

Versions¶

Message Format 1.0¶

In the original message format (version 1.0), data is sent in a single ZMQ message part containing a nested dictionary (a msgpack map).

The first level of keys are source names, which typically correspond to Karabo device names. Each source has a data dictionary representing a flattened Karabo hash, with dots delimiting successive key levels. Arrays are serialised using msgpack numpy.

Each source data dictionary also has a key metadata, which contains a further nested dictionary with keys: source, timestamp, timestamp.tid, timestamp.sec, timestamp.frac, and ignored_keys, which is a list of strings identifying keys which were filtered out of the data by configuration options on the bridge server.

{
    'SPB_DET_AGIPD1M-1/DET/detector': {
        'image.data': array(...),
        # ... other keys
        'metadata': {
            'source': 'SPB_DET_AGIPD1M-1/DET/detector',
            'timestamp': 1526464869.4109755,
            'timestamp.frac': '410975500000000000',
            'timestamp.sec': '1526464869',
            'timestamp.tid': 10000000001,
            'ignored_keys': []
        },
    }
}

Message Format 2.2¶

We have skipped describing message formats 2.0 and 2.1, as we don't know of any code that used them before version 2.2 was defined.

The data is split up into a series of pieces, allowing arrays to be serialised more efficiently. Each piece has two ZeroMQ message parts: a msgpack-serialised header, followed by a data part. A full message is therefore a multipart ZeroMQ message with an even number of message parts.

Each header part is a dictionary (a msgpack map) containing at least the keys source and content. The former is a source name such as 'SPB_DET_AGIPD1M-1/DET/detector'. The latter is one of:

'msgpack': The following data part is a msgpack map containing the data for this source, excluding any arrays. The header map also includes the metadata information as described for message format 1.0 above.
'array': The following data part is a raw array. The header has additional keys describing the array:
path: The key of this data, e.g. 'image.data'.
dtype: A string naming a (numpy) dtype, such as 'uint16' for 16-bit unsigned integers.
shape: An array of integers giving the dimensions of the array.

A multipart message might contain data from several sources. For each source, there is one header-data pair with 'msgpack' content, followed by zero or more header-data pairs for arrays.

Image data¶

Karabo ImageData objects, holding images from cameras, are represented by a number of keys with a common prefix. This keys following this prefix include:

.data: numpy array
.bitsPerPixels int
.dimensions list of int
.dimensionScales str
.dimensionTypes list of int
.encoding str
.geometry.alignment.offsets list of float
.geometry.alignment.rotations list of float
.geometry.pixelRegion list of int - seems optional
.geometry.subAssemblies list of hashes - seems optional
.geometry.tileId int
.header user defined Hash
.ROIOffsets list of int
.binning list of int

Minor changes to this list may occur without a new protocol version.

Protocol Implementations¶

Clients:

Servers:

PipeToZeroMQ Karabo device: sends data from a live Karabo system.
EXtra-data Python module: can stream data from XFEL HDF5 files.
The Python client includes a server to send simulated random data.