General Detector DAQ Troubleshooting

The following general points should be verified in case a detector is reported to have DAQ problems:

No data written to disk

  • Verify that indeed no data is written to disk. You need to be on the control-network to see, Logging onto exflgateway in case you are not in the hutch. Go to one of the online cluster servers:

    SASE suggested server
    1 exflonc10
    2 exflonc17
    3 exflonc14

    Then check file sizes for the last run:

    ssh <username>@<server>
ll -alh /gpfs/exfel/exp/<instrument>/<cycle>/<proposal>/raw/<run>

    For the MHz detector you expect file sizes in the gigabyte range.

  • Is the DAQ in the RECORD state? If not put the DAQ into the RECORD state.

  • Is the detector sending data? Check that the detector is powered, configured and sending data. You can further verify this via the RunDeck tool, which is linked in the browser bookmarks of each instrument.

  • Are the data aggregators in an error state? The DAQ projects are subprojects of the MHz detector top-level project; check the 16 XTDF data aggregators therein in the project panel. If they are in error state, call ITDM OCD.

No data on Online Views

  • Is the DAQ in the MONITOR or RECORD state? If not put the DAQ into into the MONITOR.
  • Is the detector sending data? Check that the detector is powered, configured and sending data. You can further verify this via the RunDeck tool, which is linked in the browser bookmarks of each instrument.
  • Is the online calbration pipeline started, and are constants loaded? You can check this on the MANAGER scene, all devices should be active, and indicator lights should be green for any device connected to a data output. Date and times should be shown for all constants.
  • Is the pipeline output data to the GUI? This can be checked in the MANAGER scene. The APPENDER devices show time of flight and update rates. Are these updating? If not, check if the minimum number of modules configured matches the number of modules in the detector actually producing data. If updates are happening, the GUI is not displaying them: call CAS OCD.