.. _configuration: Configuration ============= The European XFEL Offline Calibration is configured through `settings.py` and `notebooks.py` files. Both can be found in `xfel_calibrate` source directory. The `notebook.py` file exposes and configures the the notebooks by the `xfel-calibrate` command line interface. Settings -------- The `settings.py` is a python configuration file, which configures the tool's environment.:: # path into which temporary files from each run are placed temp_path = "{}/temp/".format(os.getcwd()) # Path to use for calling Python. If the environment is correctly set, simply the command python_path = "python" # Path to store reports in report_path = "{}/calibration_reports/".format(os.getcwd()) # Also try to output the report to an out_folder defined by the notebook try_report_to_output = True # the command to run this concurrently. It is prepended to the actual call launcher_command = "sbatch -p exfel -t 24:00:00 --mem 500G --mail-type END --requeue --output {temp_path}/slurm-%j.out" A comment is given for the meaning of each configuration parameter. Notebooks --------- The `xfel-calibrate` tool exposes configured notebooks to the command line by automatically parsing the parameters given in the notebooks first cell. The configuration is given in the form of a python dictionary:: notebooks = { "AGIPD": { "DARK": { "notebook": "AGIPD/Characterize_AGIPD_Gain_Darks_NBC.ipynb", "concurrency": {"parameter": "modules", "default concurrency": 16, "cluster cores": 16}, }, "PC": { "notebook": "AGIPD/Chracterize_AGIPD_Gain_PC_NBC.ipynb", "concurrency": "parameter": "modules", "default concurrency": 16, "cluster cores": 16}, }, "CORRECT": { "notebook": "notebooks/AGIPD/AGIPD_Correct_and_Verify.ipynb", "concurrency": {"parameter": "sequences", "use function": "balance_sequences", "default concurrency": [-1], "cluster cores": 32}, ... } } The first key is the detector, e.g. AGIPD. The second key is the calibration type name, e.g. DARK or PC. A dictionary is expected for each calibration type with a notebook path and concurrency configuration. For the concurrency three values are expected. Key `parameter` with a value name of type list, which is defined in the first notebook cell. The key `default concurrency` to define the range of values for `parameter` in each concurrent notebook, if it is not defined by the user. e.g. `"default concurrency": 16` leads to running 16 concurrent jobs, each processing one module with values of [0,1,2,...,15]. Finally, a hint for the number of `cluster cores` that is used if the notebook is using ipcluster parallelization, only. This value should be derived e.g. by profiling memory usage per core, run times, etc. .. note:: It is good practice to name command line enabled notebooks with an `_NBC` suffix as shown in the above example. The AGIPD `CORRECT` notebook (last notebook in the example) makes use of a concurrency generating function by setting the `use function` parameter. This function must be defined in the first cell in the notebook its given arguments should be named as the first cell notebook parameters. It is expected to return a list of parameters to concurrently run notebooks. Above the used function is :func:`xfel_calibrate.calibrate.balance_sequences`. .. note:: The function only needs to be defined, but not executed within the notebook context itself.