.. _devices/fastCCDDAQ:
************************
FastCCD Data Acquisition
************************
.. ifconfig:: includeDevInfo in ('true')
The fastCCDDAQ device receives the data from the camera and assembles the image frames.
Configuration
=============
The network configuration for this device requires the following:
* DAQ IP: IP address where the data are sent from
* DAQ Port: port from which data are sent
* DAQ Local Port: port on which the data are received on the machine running
the device
The local IP is automatically used as the receiving IP for the data stream.
The image size can be tricky to configure. A priory knowledge is required
depending on how the camera is configured through the timing file. Image
size x and y are used to allocate memory for the incoming frame. If it is
too small, the device will crash, if it is too large an extra copy step will
take place. The correct size is also needed to create the proper image format
for the output channels.
States and Operation
====================
FastCCD State Diagram
---------------------
The possible states for this karabo device are the following:
* **UNKNOWN**: in case of any timeout errors
* **ERROR**: in case of any other errors
* **OFF**: after initialization
* **ACTIVE**: after auto-start, applying basic configuration to the camera, has finished
* **ON**: after main power is enabled and all other power sources are also on
* **ACQUIRING**: during data acquisition
During normal operation, the work flow is depicted in :numref:`figflow`
.. _figflow:
.. figure:: pics/fastCCD_StateFLow.png
:width: 70 %
:alt: pic of FastCCD state flow
State diagram for FastCCD detector.
When the device is instantiated a handshake is sent to the camera to
configure the data output to be sent to the correct address. If the camera
has been power cycled, i.e., the power to the CIN
was interrupted, the configuration is lost.
In such a case the handshake has to be re-sent using the "Reconnect"
button of the this device.
After instantiation, the device will automatically start to listen for data
and be in the ACTIVE state. Listening to data can be started and stopped with
the respective buttons. To recover from an error, the reset button can be used,
which will put the device into the PASSIVE state and the start button has to
be used to switch to the ACTIVE mode and to take data again.
Data Output
===========
The complete image is sent out on two output channels, one for the DAQ chain
and one for the preview.
For testing and debugging there is the option to write data to file, which
will write one frame per file. The default operation should however include
the proper standard DAQ chain from ITDM.
Online Preview
==============
Raw Preview
-----------
The DAQ panel in the Control scene contains a small preview of the raw data acquired by the detector. The gain bits are not processed in this preview, so Medium and Low gain pixels have a very large, arbitrary offset with respect to High gain pixels.
In general you should set the limits of the z axis (color scale) of this preview to 3400 - 4300 ADU (ADC Units).
.. _CalibratedPreview:
Corrected Preview
-----------------
This preview (:numref:`CalPreview`) shows the offset corrected images based on the most recent calibration constants (from dark run) for particular gain and temperature settings of the detector. It is therefore a processed form of the raw data preview.
The calibrated preview is part of the Control Scene (see :numref:`ControlScene`). To update this preview to use the most recent dark runs:
1. Process the dark runs (see :numref:`ProcessingDarkRuns`) with high, medium and low gain settings, which adds the most recent calibration constants to the Calibration Database.
2. On the Calibrated Preview part of the Control scene, you will see a table with 8 inputs. Change the settings in the table to match the current detector configuration and hit Return twice to commit the changes. Make sure the table does not have a blue edge as that indicates the change is not saved in Karabo. If you see the blue edge around the table, press Return again.
#. Click on Reset button and wait for the status to become PASSIVE.
#. Click on Init button to initialize the change and wait for the status to become ACTIVE.
#. At this point, the output of "Offset is from" field should change to the time the last dark runs were acquired. If it does not make sure that the condition table matches the condition in which the dark runs were acquired.
#. Even though the "Offset is from" field is updated, the preview is still based on the previous calibration constants. This is a known bug. You need to press Reset and Init one more time in the correct order to actually update the calibrated preview with the most recent calibration constants.
When a recent set of calibration constants has been loaded, the z-axis (color scale) of the image preview (and that of the histograms) should be centered around 0 for the high gain, a reasonable default is min:-100 and max:100 ADU.
If the beam intensity is too high, the preamplifiers of the CCD camera may be saturated. If that is the case, you will see dark red lines (or a dark red region) on the calibrated preview. In this case, ask the SCS beam-line scientists to reduce the intensity.
.. _CalPreview:
.. figure:: pics/cal_preview_Screenshot.png
:width: 70 %
The calibrated preview.
Preview Histograms
------------------
There are two online histograms in the histogram scene in the SCS_CDIDET_FFCD2M_CAL subproject (see :numref:`histogram`): both based on calibrated preview described above. In the first one only the offset correction is applied while the second one has the relative gain correction. Due to a known bug if you are using the Low or Medium gain, you will need to adjust the range according to the calibrated preview and ranges.
.. ifconfig:: includeDevInfo in ('true')
When the FastCCD is obtaining data using the x-ray beam, these two histograms should be monitored. From these online histogras, you can:
1. guide the SCS scientists on shift as to whether or not they are operating in single photon zone, where each detector pixel is only hit with one photon. In this case, you will see two peaks on the online histograms: one has a centroid of zero (only if the gain is set to 8 (high gain)) and the other peak should represent the scattered beam detected by the camera. Therefore, if there are more peaks, this means the detector pixels are being hit with more than one photon at the time, which is not single photon zone.
2. determine whether or not the image is being saturated. If the beam intensity is too high or if the beam is hitting the camera (which is a serious source of damage to the detector), the peaks on online histograms will show saturation representing a clipped signal on the oscilloscope.
3. determine whether or not the offset correction is accurately obtained for high gain setting (the medium and low gain distributions are not centered around zero at the moment).
.. _histogram:
.. figure:: pics/histo_Screenshot.png
:width: 70 %
The online histograms.
Offline Data
============
.. _ProcessingDarkRuns:
Processing Dark Runs
--------------------
After taking a dark run for each gain setting, log into Metadata Catalog (`https://in.xfel.eu/metadata <https://in.xfel.eu/metadata/>`_). Click on Proposals link on the top right hand side, then on the current proposal number (if you cannot find the current proposal make sure the SCS beam-line scientists have given you the permission to access it). Open the Runs page; there you should see the run numbers corresponding to the dark runs you just took. It might take a few minutes for the runs to appear on the list (you will need to reload the page manually).
If the dark runs were OK, change the Data Assessment to Good (if you cannot do this step, you do not have the right permissions for this proposal and need to ask a member of the SCS group) and migrate the data run to Maxwell cluster.
To inject the new dark runs, please follow the following steps:
#. Wait a few minutes for the data has been migrated to Maxwell.
#. Connect to Maxwell: :code:`ssh username@max-exfl`
#. load anaconda: :code:`module load anaconda/3`
#. For each gain/run, execute this command replacing cycle, proposal and run numbers accordingly (you need to have access to the proposal for this step to work): :code:`python /gpfs/exfel/data/scratch/haufs/request_darks.py --instrument SCS --cycle ### --proposal ### --run ###`
#. You can launch multiple instances of the script simultaneously, or cancel it (with :code:`Ctrl+C`) after it reports initial feedback (:code:`DONE`, sometimes truncated to :code:`ONE`). In case you leave it running, it will report feedback on the process.
Once the dark runs are analyzed and the calibration constants are injected into the Calibration Database, you can setup the calibrated preview (see :numref:`CalibratedPreview`) or reprocess acquired data.
Reprocessing Acquired Data
--------------------------
The instructions for the web service based calibration via Metadata Catalog are found at `https://in.xfel.eu/readthedocs/docs/detector-documentation/en/doc-lpd_non_lin/calibration/offline_calibration.html <https://in.xfel.eu/readthedocs/docs/detector-documentation/en/doc-lpd_non_lin/calibration/offline_calibration.html>`_.
After the process is over you will find in the proc/ folder of your proposal two files for each raw file:
1. those starting with CORR contain offset and relative gain corrected data in the "pixels" array (see Figure :numref:`DataView`). These files process quite quickly.
#. those starting with CORR-D contain additionally common-mode ("pixels") and charge split corrected data in the "pixels_classified" array, as well as a "pattern array", which contains event multiplicity.
.. ifconfig:: includeDevInfo in ('true')
The event multiplicity is evaluated during split charge correction, and encodes the following:
#. 100, 101: single events
#. 200-203: charge split into two pixels in four different orientations.
#. 300-303: charge split into three pixels in four different orientations.
#. 400-403: charge split into four pixels in four different orientations.
#. 1000: charge in more than four neighboring pixels. Cannot be produced by a single photon alone.
Accessing the Raw Data
----------------------
.. _DataView:
.. figure:: pics/Data_Screenshot.png
:width: 100 %
Configuring :code:`hdfviewer` to show the raw image data.
If you need to access the raw data, follow these steps:
1. Make sure the SCS beam-line instrument staff have given you the permission to the current proposal. If not, ask them to add your username to the proposal under study.
#. Open a terminal and connect to Maxwell cluster (:code:`ssh username@max-exfl`)
#. Go to the directory where the current raw data are being stored. An example is: :code:`/gpfs/exfel/exp/SCS/201930/p900074/raw/r0100`, where SCS is the instrument, 201930 the proposal cycle, p00074 the proposal number, and r0100 is the run number 100.
#. For a specific run number, there should be a few files in that directory. All of them look like :code:`RAW-R0100-DA##-S000##.h5`, where DA## refers to the data aggregators and DA05 is for FastCCD and S##### refers to the file sequence. If the DAQ is setup correctly and the connection between DAQ device and data aggregator is established correctly, you should see one or more :code:`RAW-R0100-DA05-S#####.h5` files in the directory.
#. To take a look at what is inside these raw data files, on the terminal type: module load x-ray.
#. Open the raw data file structure with this command: :code:`hdfview RAW-R0100-DA05-S00001.h5`
#. Browse to the following array to see the raw image data: :code:`/INSTRUMENT/SCS_CDIDET_FCCD2M/DAQ/FCCD:daqOutput/data/image/pixel` (see :numref:`DataView`)
To ensure data is being properly written on disk make sure that the "pixel" is non empty and non 0.
To visualize the data right click on pixel, select Open As, select the Image as display type then, in the Dimension box, select Height: dim 1, Width: dim 2, Depth: dim 0.