Batchlet Configuration Details ============================== Top-level configuration schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------+-------------------------------------------------------------------------------------------------------------+-------------+-------------+----------+----------+ | Key | Description | Type | Default | Required | Nullable | +====================================+=============================================================================================================+=============+=============+==========+==========+ | ``command`` | List containing the command and its arguments. | list | | Yes | No | +------------------------------------+-------------------------------------------------------------------------------------------------------------+-------------+-------------+----------+----------+ | ``dask_params`` | Configuration options for the Dask cluster. If ``null``, batchlet does not start the dask cluster, and runs | object | ``null`` | No | Yes | | | the ``command`` directly as a subprocess. | | | | | +------------------------------------+-------------------------------------------------------------------------------------------------------------+-------------+-------------+----------+----------+ | ``monitor`` | Configuration options for monitoring. If ``null``, monitoring is disabled. | object | ``null`` | No | Yes | +------------------------------------+-------------------------------------------------------------------------------------------------------------+-------------+-------------+----------+----------+ | ``generate_reports_on_failure`` | Whether to generate performance/monitoring reports even if the ``command`` fails for any reason. If set to | boolean | ``true`` | No | No | | | ``false`` and if the ``command`` subprocess fails, batchlet will clear reports generated by it. | | | | | +------------------------------------+-------------------------------------------------------------------------------------------------------------+-------------+-------------+----------+----------+ ``dask_params`` schema ~~~~~~~~~~~~~~~~~~~~~~ +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | Key | Description | Type | Default | Required | Nullable | +==============================+=======================================================================================================+===========+========================+===========+===========+ | ``dask_cli_option`` | The cli option of the command which takes dask scheduler ip as input. If not given, batchlet does not | string | ``null`` | No | Yes | | | spin up a dask cluster, and runs the ``command`` directly as a subprocess. | | | | | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``nodes`` | Number of nodes to use to spin up the dask cluster. Only relevant for ``DaskSlurmCluster``. If | integer | ``null`` | No | Yes | | | ``null``, this will be automatically picked by the cluster using slurm exported environment | | | | | | | variables. | | | | | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``workers_per_node`` | Number of Dask workers per node. | integer | ``null`` | No | Yes | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``threads_per_worker`` | Number of threads per Dask worker. | integer | ``null`` | No | Yes | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``memory_per_worker`` | Memory allocated per dask worker. | string | ``"auto"`` | No | Yes | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``resources_per_worker`` | Dask `“Worker Resources” `__ assigned per | string or | ``null`` | No | Yes | | | worker. Accepts either a dictionary or a string. If string, multiple resources can be passed as comma | dict | | | | | | seperated key-value pairs. Example values: ``"mem=2,gpu=2"`` ; ``{"CPU": 10}`` | | | | | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``worker_scratch_directory`` | Scratch directory assigned to workers. | string | ``null`` | No | Yes | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``use_entry_node`` | Whether to use the entry node for scheduling workers. Only relevant for ``DaskSlurmCluster``. | boolean | ``true`` | No | No | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``silence_logs`` | Set logging level for scheduler and worker. Should be an integer value representing standard logging | integer | 20 | No | No | | | level in python ``logging`` module. For example, the default value ``20`` corresponds to | | | | | | | ``logging.INFO`` level. | | | | | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``name`` | Name of the cluster. This also becomes the prefix for the names of the dask workers started by this | string | ``"DaskSlurmCluster"`` | No | No | | | cluster. | | or | | | | | cluster. | | ``"DaskLocalCluster"`` | | | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ | ``dask_report_dir`` | Enable Dask performance tracking and store the report at the specified path, if it is valid. | string | ``null`` | No | Yes | +------------------------------+-------------------------------------------------------------------------------------------------------+-----------+------------------------+-----------+-----------+ ``monitor`` schema ~~~~~~~~~~~~~~~~~~ ============= ======================================================================= ======= ======= ======== ========= Key Description Type Default Required Nullable ============= ======================================================================= ======= ======= ======== ========= ``resources`` Configuration for starting hardware resource monitoring using benchmon. dict {} No Yes ``logs`` Configuration for starting log monitoring using batchlet plugins. dict {} No Yes ============= ======================================================================= ======= ======= ======== ========= ``resources`` schema ^^^^^^^^^^^^^^^^^^^^ ===================== =========================================================================================== ======= ==================== ========== ========== Key Description Type Default Required Nullable ===================== =========================================================================================== ======= ==================== ========== ========== ``working_directory`` The working directory for benchmon processess. str ./benchmon-work-dir No No ``start`` Benchmon-start cli-options. This should be a mapping between option string to value string. dict {} No Yes ``stop`` Benchmon-stop cli-options. This should be a mapping between option string to value string. dict {} No Yes ===================== =========================================================================================== ======= ==================== ========== ========== ``logs`` schema ^^^^^^^^^^^^^^^ ==================== ================================================= ================== ======== ======== ======== Key Description Type Default Required Nullable ==================== ================================================= ================== ======== ======== ======== ``filter_plugins`` List of filter plugins with their configuration list of dictionary ``null`` Yes No ``consumer_plugins`` List of consumer plugins with their configuration list of dictionary ``null`` Yes No ==================== ================================================= ================== ======== ======== ======== Each filter or consumer plugin may contain following keys: ========== =================================================================================================== ========== ======= ======== ======== Key Description Type Default Required Nullable ========== =================================================================================================== ========== ======= ======== ======== ``name`` The plugin class. An instance of this class will be created by log monitor. string Yes No ``kwargs`` Dictionary of arguments to pass to the plugin (specific to the plugin). dictionary No No ``path`` Optional path to the plugin module file from which the plugin class will be imported by log monitor string No No ========== =================================================================================================== ========== ======= ======== ========