Data rates and throttling
Network limits and data rates
calng correction devices can typically process data faster than we can reliably send it over the network. It is not immediately obvious when a fast data stream runs into network limits - typically, some trains seem to silently go missing. For different output needs, different throttling options are available:
- From detector module to DAQ (outside scope of calng)
- Number of frames per train is important, keep in mind for later steps
- From DAQ to correction device
- Can tweak DAQ train stride via calibration manager: positive integer
nwhere DAQ only sends everynth train on monitoring output (higher value means slower data)
- Can tweak DAQ train stride via calibration manager: positive integer
- From correction device to group matcher
- Can tweak number of frames of frames processed and forwarded using a frame filter
For full detector matching in particular, it is important to choose DAQ train stride appropriately given the desired number of frames per train (either from detector or via frame filter on correction side).
Tested stable configurations
The primary goal of this page is document some configurations for which data flows have been found to be stable in practice on ONC machines. These measurements should be grouped by the different levels of bottlenecks that can occur. These are, broadly, in order of increasing restrictiveness:
- Data input: if exceeded, data may not reliably get from DAQ to correction pipeline in the first place (symptom: preview matching breaks down)
- Group matching
- Full matching: most restrictive, is only stable if all group matching is stable and entire data flow fits through a single network node
Note that these measurements are typically taken while the DAQs are MONITORING.
It has been documented that DAQs ACQUIRING causes data rates reaching the pipelines to fall.
The tables are woefully incomplete; whenever testing has happened, please make notes of tested configurations which work well.
FXE
Note: these measurements are from before a major hardware upgrade at SASE1 ONC at end of 2022 and some network configuration updates at the beginning of 2023. Limited testing suggests we may be able to get sigificantly better numbers going forward. New benchmarks are pending.
LPD1M
Data input
| frames | DAQ train stride | effective FPS |
|---|---|---|
| 150 | 1 | 1500 |
| 160 | 1 | 1600 |
| 300 | 2 | 1500 |
Full matching
Initial tests with OnDA:
| frames | DAQ train stride | effective FPS |
|---|---|---|
| 64 | 3 | 213.33 |
| 160 | 6 | 266.67 |
Tests on 31.08.2022 after binding all channels to Infiniband.
Matching ratio measured on FXE_DA_USR/MATCHER/MATCHER; note that matching ratio fluctuates a lot and the ranges here are estimates from brief period of observation:
| frames | DAQ train stride | matching ratio | effective FPS |
|---|---|---|---|
| 20 | 1 | ~95% | 190 |
| 40 | 1 | ~90% | 360 |
| 50 | 1 | ~86-92% | 430-460 |
| 60 | 1 | ~72-90% | 432-540 |
| 80 | 1 | ~40-55% | 320-440 |
| 100 | 1 | ~25-30% | 250-300 |
| 100 | 2 | ~90-100% | 450-500 |
| 150 | 2 | ~60-70% | 450-525 |
| 200 | 2 | ~20-40% | 200-400 |
MID
AGIPD1M
Brief tests conducted on 08.09.2022 with outputs bound to Infiniband. Interestingly, correction devices received empty hashes from DAQ in several cases rather than the usual silent network-limited drops.
Data input
| frames | DAQ train stride | effective FPS |
|---|---|---|
| 352 | 2 | 1760 |
Full matching
| frames | DAQ train stride | matching ratio | effective FPS |
|---|---|---|---|
| 352 | 10 | ~100% | 352 |
| 352 | 6 | ~100% | 587 |
| 352 | 5 | ~50-80% | 352-563 |
| 352 | 4 | ~20-40% | 176-352 |
The empty hashes observed from the DAQ were an issue at train strides 5 and 3, but much less so for 4 and 2. Thus, one may consider using the frame filter on the pipeline to stay below bandwidth limits:
| frames | DAQ train stride | frame filter | matching ratio | effective FPS |
|---|---|---|---|---|
| 352 | 4 | range(100) |
~100% | 250 |
| 352 | 4 | range(200) |
~100% | 500 |
| 352 | 2 | range(100) |
~95-100% | 475-500 |
SCS/SQS
DSSC1M
Data input
| frames | DAQ train stride | effective FPS |
|---|---|---|
| 400 | 1 | 4000 |
| 800 | not sure |
Group matching
Downcasting corrected data to float16, the first field test of calng at SQS was able to send full-speed data at 400 frames per train for a group over a Karabo bridge to Hummingbird:
| frames | DAQ train stride | effective FPS | note |
|---|---|---|---|
| 400 | 1 | 4000 | float16 |
SPB
Note: these measurements are from before a major hardware upgrade at SASE1 ONC at end of 2022 and some network configuration updates at the beginning of 2023. Limited testing suggests we may be able to get sigificantly better numbers going forward. New benchmarks are pending.
AGIPD1M
Data input
| frames | DAQ train stride | effective FPS |
|---|---|---|
| 352 | 4 | 880 |