July 2020

EXtra-data 1.2

EXtra-data release 1.2 is out and comes with many new tools, features and improvements! We present a few of them in this letter. The full list can be seen here.

lsxfel

lsxfel has a new --detail option to show more detail on selected sources.

Example, checking if instrument sources have missing data:

% lsxfel --detail "*" /gpfs/exfel/exp/SQS/201901/p002388/raw/r0138 | grep % -B 2
  - SA3_XTD10_XGM/XGM/DOOCS:output
    - data:
      data for 6474 trains (100.00%), up to 1 entries per train
--
  - SQS_AQS_LAS/SPEC/800:output
    - data:
      data for 5645 trains (87.19%), up to 1 entries per train
--
  - SQS_DAQ_SCAN/MDL/KARABACON:output
    - data:
      data for 21 trains (0.32%), up to 1 entries per train
--
  - SQS_DIAG3_PAM/CAL/SPEC_REF:daqOutput
    - data:
      data for 0 trains (0.00%), up to 0 entries per train

extra-data-validate

extra-data-validate is significantly faster, and validating a run with several terabytes of data now takes only a few seconds.

extra-data-validate.gif

Validate run data integrity

extra-data-make-virtual-cxi

extra-data-make-virtual-cxi has a new --fill-value option to set the default value for missing data.

extra-data-locality

extra-data-locality is a new command line tool to check how the files are stored.

Reading files may hang for a long time if files are unavailable or require staging in dCache from the tape. The program helps finding problematic files. If it finds problems with the data locality, the program will produce a list of files located on tape, lost or at unknown locality.

% extra-data-locality /gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002
Checking 120 files in /gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002
120 on disk, 0 only on tape, 0 unavailable

karabo-bridge-serve-files

karabo-bridge-serve-files has a new --append-detector-modules option to combine data from multiple detector modules. This makes streaming large detector data more similar to the live data streams.

Note

Documentation of all EXtra-data command line tools is available here

write_frames

EXtra-data detector components have a new write_frames method to write selected detector frames to a new EuXFEL HDF5 file.

Oscovida

The Data Analysis team and members of the PaNOSC project have been working on a COVID19 tracking website as part of efforts to improve the public awareness and understanding of the current coronavirus situation.

The Open Science COVID19 Analysis (OSCODIA) package is used to programatically create a website that provides COVID19 tracking information, which can be used by scientists and the wider public to understand spread of the global pandemic, and in some regions, where the first wave has passed, to detect, monitor and study localised or wider renewed outbreaks. The availability of the reproduction number R and its behaviour as a function of time helps to put the discussions in politics and media into context, the change in daily deaths and cases can help motivate measures such as social distancing or easing of restrictions. You can view the tracking page (updated twice a day) here: https://oscovida.github.io

For example, this is the Hamburg tracking page, each tracking page has the same set of plots and the meaning of the plots is explained in our ‘standard plots’ educational page. As the analysis pages themselves are generated from Jupyter Notebooks, we provide a link on each page to easily open the notebook in an interactive MyBinder instance which allows users to explore the data themselves however they want.

Currently tracking pages are provided for: all countries, German Landkreise, American states, and Hungarian counties. If you would like more detailed regions for your country to be included on the website please create an issue on the Oscovida GitHub repository.

Did you know… the data analysis documentation?

Did you know that the data analysis group maintains an extensive documentation surrounding many aspects of analyzing data at European XFEL? The latest version can always be found here: Data analysis at European XFEL

It covers a wide array of topics from the internal data structure of files to the tools and services readily available in our infrastructure down to specific deployments provided at each instrument. It is continuously updated and extended, and if you notice anything missing or have further questions, please dont’t hesitate to email us at da-support@xfel.eu. You can always find the link on the front page of this newsletter, too.