Data Analysis Software¶

Are you looking for how to use software environments?

This section of the documentation describes what packages we provide and some of their use cases. For information on how activate the environments containing this software and how to use the Environment Modules system on Maxwell see the Software Environments section.

Software to Access and Inspect Data¶

EXtra Data¶

extra_data is a Python library for accessing and working with data produced at European XFEL. It can:

Conveniently access data from an experimental run, which is often spread across dozens of 'sequence' files.
Read data into pandas and xarray, two popular Python libraries which support powerful, efficient data analysis.
Assemble image data for multi module detectors like LPD and AGIPD, using geometry files in different formats.
Stream data from files over a ZeroMQ socket. This stream of data can then be accessed using karabo bridge clients, to test live-processing tools with data from real experiments.

More Information

EXtra-data documentation

Karabo Bridge¶

We provide client libraries in Python and C++ to receive data from the karabo bridge, allowing users to integrate their tools with the karabo framework and receive live data during an experiment run.

More Information

HDF5 Command Line Tools¶

We hope most users will be able to access data through existing tools such as EXtra-data, or by converting it to standard formats such as CXI. But if none of these options work for you, you may need to look directly at the HDF5 files.

Basic HDF5 command line tools are available by default::

h52gif         h5copy         h5fc-64        h5perf_serial  h5unjam
h5c++          h5debug        h5import       h5redeploy
h5c++-64       h5diff         h5jam          h5repack
h5cc           h5dump         h5ls           h5repart
h5cc-64        h5fc           h5mkgrp        h5stat

You can inspect HDF5 files in the terminal with the h5glance tool, available in the exfel-python module - see Software Environments - Module System.

The hdfview interactive viewer is available in the xray module (see https://confluence.desy.de/display/MXW/hdfview). To use it, first run module load xray.

HDF5 Virtualise¶

We also provide a tool to create a virtual CXI file, which can be used with any tools that take CXI-style data: HDF5 Virtualise.

Software with scientific purposes¶

EXtra-foam¶

One of our main programs for online analysis is EXtra-foam, which is a GUI program for easily doing common kinds of analysis.

More Information

EXtra-metro¶

EXtra-metro is a processing framework designed to let you implement your own analysis.

More Information

Online Analysis EXtra-metro section

GeoAssembler¶

More Information

geoAssembler Repository

The tool can be seen as an alternative to the calibration mode of CrysFEL's hdfsee. The calibration can either be based on a starting geometry that needs to be refined or a completely new geometry. In the latter case the initial conditions for the geometry are defined so that all modules are 29px apart from each other.

The geometry calibration is supported by two modes of graphical user interfaces. A Qt-based and a jupyter notebook based interface.

It is recommended to use this GUI application through the pre-installed path on the on- and offline cluster (see Software Environments - Module System.).

XasTim¶

A toolchain for real-time data analysis and visualization of XAS (X-ray Absorption Spectroscopy) experiments using the TIM (Transmission Intensity Monitor) device.

More Information

SCS Toolbox - XasTim

Cheetah¶

The CFEL group at DESY provides Cheetah on the Maxwell cluster. See their documentation for how to get started using it.

More Information

Cheetah documentation on European XFEL

EXtra-xwiz¶

A command-line tool for managing serial femtosecond crystallography (SFX) workflows using CrystFEL on EuXFEL-calibrated detector data. It is currently in a beta-testing phase.

More Information

EXtra-xwiz documentation

Access on Online and Maxwell Cluster¶

More Information

For information on how activate the environments containing this software and how to use the Environment Modules system see the Software Environments section.

The tools described above are readily available on both the online cluster and the Maxwell cluster. We recommend using the already setup applications available in the XFEL specific Anaconda3 distribution:

module load exfel exfel-python

This will provide access to all these libraries and tools for data analysis:

extra_data
EXtra-foam
karabo_bridge
extra_data_interactive
geoAssembler
pyFAI
xas-tim-view
karabo-bridge-record
karabo-bridge-replay
h5glance
extra-data-validate

There are many other applications and libraries available on the Maxwell cluster, maintained by DESY. There is a listing at: https://confluence.desy.de/display/IS/Alphabetical+List+of+Packages

Not all of this software is on the online cluster (maintained by EuXFEL).

Adding Extra Software¶

Both the Offline Cluster and the Online Cluster environment feature a set of data analysis tools. If an experiment requires access to additional analysis packages or applications, this user requirement should be discussed and agreed ahead of the experiment; please contact da-support@xfel.eu in such a case.

In addition, users can bring their own tools and install them in their user space. This will make them available for immediate use in both offline and online environment.

Make your Software Available to All Users at EuXFEL¶

In general, users who create their own data analysis packages are encouraged to share the progress on such data analysis development with European XFEL and seek help to make them functional at EuXFEL.

EuXFEL is keen to make such existing tools available to all users, and to provide the corresponding software as a service as part of a collaboration with the authors of the package. Please approach da-support@xfel.eu for further information and discussion if this is of interest to you.