.. _configuration:

Configuration
=============

The European XFEL Offline Calibration is configured through `settings.py`
and `notebooks.py` files. Both can be found in `xfel_calibrate` source directory. The `notebook.py` file exposes and configures the
the notebooks by the `xfel-calibrate` command line interface.

Settings
--------

The `settings.py` is a python configuration file, which configures the tool's environment.::

    # path into which temporary files from each run are placed
    temp_path = "{}/temp/".format(os.getcwd())

    # Path to use for calling Python. If the environment is correctly set, simply the command
    python_path = "python"

    # Path to store reports in
    report_path = "{}/calibration_reports/".format(os.getcwd())

    # Also try to output the report to an out_folder defined by the notebook
    try_report_to_output = True

    # the command to run this concurrently. It is prepended to the actual call
    launcher_command = "sbatch -p exfel -t 24:00:00 --mem 500G --mail-type END --requeue --output {temp_path}/slurm-%j.out"

A comment is given for the meaning of each configuration parameter.


Notebooks
---------

The `xfel-calibrate` tool exposes configured notebooks to the
command line by automatically parsing the parameters given in the notebooks first cell.
The configuration is given in the form of a python dictionary::

    notebooks = {
        "AGIPD": {
            "DARK": {
                "notebook": "AGIPD/Characterize_AGIPD_Gain_Darks_NBC.ipynb",
                "concurrency": {"parameter": "modules",
                                "default concurrency": 16,
                                "cluster cores": 16},
             },
             "PC":   {
                 "notebook": "AGIPD/Chracterize_AGIPD_Gain_PC_NBC.ipynb",
                 "concurrency": "parameter": "modules",
                                "default concurrency": 16,
                                "cluster cores": 16},
             },
             "CORRECT":   {
                 "notebook": "notebooks/AGIPD/AGIPD_Correct_and_Verify.ipynb",
                 "concurrency": {"parameter": "sequences",
                 "use function": "balance_sequences",
                 "default concurrency": [-1],
                 "cluster cores": 32},
             ...
         }
     }

The first key is the detector, e.g. AGIPD. The second key is the calibration type name, e.g. DARK or PC.
A dictionary is expected for each calibration type with a notebook path and concurrency configuration.
For the concurrency three values are expected. Key `parameter` with a value name of type list, which is defined in the first notebook cell.
The key `default concurrency` to define the range of values for `parameter` in each concurrent notebook, if it is not defined by the user.
e.g. `"default concurrency": 16` leads to running 16 concurrent jobs, each processing one module with values of [0,1,2,...,15].
Finally, a hint for the number of `cluster cores` that is used if the notebook is using ipcluster parallelization, only.
This value should be derived e.g. by profiling memory usage per core, run times, etc.

.. note::

    It is good practice to name command line enabled notebooks with an `_NBC` suffix as shown in
    the above example.

The AGIPD `CORRECT` notebook (last notebook in the example) makes use of a concurrency generating function
by setting the `use function` parameter. This function must be defined in the first cell in the notebook
its given arguments should be named as the first cell notebook parameters. It is expected to return a list of parameters
to concurrently run notebooks. Above the used function is :func:`xfel_calibrate.calibrate.balance_sequences`.

.. note::

    The function only needs to be defined, but not executed within the notebook context itself.