Skip to content

Troubleshooting

Preview does not update

  1. Is detector not sending data?
    • Open the device scene for individual correction devices and look at the device status log. If the DAQ is monitoring, but without detector data coming in, log should show warnings about this.
  2. Is pipeline down or misconfigured?
  3. Is preview matching breaking down?
    • Note: for fast detectors, data transfer from DAQ to correction device can hit network limits. This can cause matching on preview assemblers to break down. If rates on correction devices fluctuate or are lower than expected, try increasing the DAQ train stride to slow down incoming data.

Preview is flickering

A common reason for flickering is the max idle parameter on the preview assemblers being set too close to 10/DAQ train stride. If for instance max idle is 1 second and DAQ train stride is 10, then every detector source is expected to provide data to the assembler exactly once per second - and even slight jitter will cause the last sources for a given train to show up too late, thus ignoring them for matching.

Analysis software not receiving data

Assumption: you are running some online analysis suite connected to the Karabo Bridge output of a group matcher or a full matcher. These matchers (along with the preview assemblers) are built around TrainMatchers.

  1. Is the matcher not receiving data?
    • Check that detector is sending data and calibration pipeline generally works.
    • Check that the sources are configured correctly, that the matcher is started, and that it has established connections with the source channels.
  2. Is matching ratio very low?
    • A group or full matcher can easily run into network limits for big detectors. This can cause individual trains from individual modules to randomly not reach the matcher, leading to very low matching ratio. If received rates for individual sources fluctuate or are lower than expected, try increasing the DAQ train stride or lowering the number of frames per train (optionally via frame filter on correction devices) to slow down incoming data.
    • In case the matcher itself has a hard time keeping up (e.g. stacking full matcher for big detector), check performance tweaks like "use thread pool" and disabling Karabo channel outputs.
  3. Is a missing module breaking matching?
    • If a module is known to be missing, consider deselecting it in the sources configuration.
    • To gracefully handle situations where individual modules disappear / stop working for a while, consider using the Max Idle TrainMatcher feature on the matcher. In case of a full detector matcher, do check what this is set to for the individual group matchers, too.

Cannot load new constants

Assumption: you have just generated new constants (for example, dark constants via typical procedure) and the pipeline doesn't seem to load them. While investigating, it is recommended to open the "correction constant overview" found on the calibration manager overview to see the timestamps for currently found constants. If constants are found - individual device logs do not complain about "condition not found" / "calibration not found", but get stuck actually loading - see loading new constants is very slow.

  1. Are the operating conditions used for constant querying correct?
    • These are found in the "Constant retrieval parameters" on the manager overview scene.
    • They need to match the conditions that the constants were injected into the database with (we are working to make it easier to keep the manager configuration up to date - for now, this is entered manually).
  2. Were the constants injected properly into the database?
  3. If constants are there, do the device logs indicate some issue when loading them?
    • Inspect the device status log on the scene for one of the correction devices; it may give some information about the issue at hand.
    • (For experts) Check the values on a specific correction device under "Constants retrieved" - "(correction name)". During querying, the fields tagged with [CalCat] should fill out; if you have sufficient CalCat rights, you can cross-reference the found condition / constant (version) IDs.
  4. Try restarting the pipeline to clear out all caches.

Loading new constants is very slow

Due to file system and caching circumstances, constants which have just been generated typically take longer to load for the first time than older constants. This was for a time exacerbated by an issue with the filesystem configuration. Said issue has been resolved and newer calng devices additionally include mitigation steps in case constant loading hangs.

  • The loading process may hang for a while and can fail
    • Mitigation: file loading done by correction device in separate process to protect device process from stalling
    • If a device is taking a long time loading new constants, the status log should give some information about current progress
  • Loading many files at once may exacerbate issues
    • Instead of using the "load most recent constants" slot on the manager, consider using macro to stagger loading for different devices