ska_sdp_batchlet.utils.dask_cluster.handler module

class ska_sdp_batchlet.utils.dask_cluster.handler.DaskHandler(stack, command, generate_reports_on_failure=True, dask_cli_option=None, dask_report_dir=None, **dask_cluster_params)

Bases: object

Handles all dask related operations.

It is responsible for instantiating and starting both DaskCluster (based on the infrastructure) and dask's performance monitoring.

Parameters:

• stack (ExitStack) -- The ExitStack to register the dask cluster and monitor contexts with. NOTE: The "stack" is pass by reference, and DaskHandler will modify it.

• command (list[str]) -- The subprocess command as passed by the user

• generate_reports_on_failure (bool, default=True) -- Whether to generate reports even if some exception occurs during exit

• dask_cli_option (str, optional) -- The cli option corresponding to the dask scheduler as passed to the underlying subprocess

• dask_report_dir (str, optional) -- Directory where the dask performance report will be stored

• **dask_cluster_params -- All other parameters passed in the "dask_params" key of the user provided configuration

cluster: SpecCluster

append_scheduler_address(command)

Appends the dask cli option of the subprocess, and the dask scheduler address, to the command to be executed. If cluster has not been started, then it returns the original command without modifying.

Parameters:

command (list of str) -- The subprocess command to be executed by batchlet

Returns:

The new command with dask scheduler address appended

Return type:

list of str