Batchlet Configuration Details

Top-level configuration schema

Key	Description	Type	Default	Required	Nullable
command	List containing the command and its arguments.	list		Yes	No
dask_params	Configuration options for the Dask cluster. If null, batchlet does not start the dask cluster, and runs the command directly as a subprocess.	object	null	No	Yes
monitor	Configuration options for monitoring. If null, monitoring is disabled.	object	null	No	Yes
generate_reports_on_failure	Whether to generate performance/monitoring reports even if the command fails for any reason. If set to false and if the command subprocess fails, batchlet will clear reports generated by it.	boolean	true	No	No

dask_params schema

Key	Description	Type	Default	Required	Nullable
dask_cli_option	The cli option of the command which takes dask scheduler ip as input. If not given, batchlet does not spin up a dask cluster, and runs the command directly as a subprocess.	string	null	No	Yes
nodes	Number of nodes to use to spin up the dask cluster. Only relevant for DaskSlurmCluster. If null, this will be automatically picked by the cluster using slurm exported environment variables.	integer	null	No	Yes
workers_per_node	Number of Dask workers per node.	integer	null	No	Yes
threads_per_worker	Number of threads per Dask worker.	integer	null	No	Yes
memory_per_worker	Memory allocated per dask worker.	string	"auto"	No	Yes
resources_per_worker	Dask “Worker Resources” assigned per worker. Accepts either a dictionary or a string. If string, multiple resources can be passed as comma seperated key-value pairs. Example values: "mem=2,gpu=2" ; {"CPU": 10}	string or dict	null	No	Yes
worker_scratch_directory	Scratch directory assigned to workers.	string	null	No	Yes
use_entry_node	Whether to use the entry node for scheduling workers. Only relevant for DaskSlurmCluster.	boolean	true	No	No
silence_logs	Set logging level for scheduler and worker. Should be an integer value representing standard logging level in python logging module. For example, the default value 20 corresponds to logging.INFO level.	integer	20	No	No
name	Name of the cluster. This also becomes the prefix for the names of the dask workers started by this cluster. cluster.	string	"DaskSlurmCluster" or "DaskLocalCluster"	No	No
dask_report_dir	Enable Dask performance tracking and store the report at the specified path, if it is valid.	string	null	No	Yes

monitor schema

Key	Description	Type	Default	Required	Nullable
resources	Configuration for starting hardware resource monitoring using benchmon.	dict	{}	No	Yes
logs	Configuration for starting log monitoring using batchlet plugins.	dict	{}	No	Yes

resources schema

Key	Description	Type	Default	Required	Nullable
working_directory	The working directory for benchmon processess.	str	./benchmon-work-dir	No	No
start	Benchmon-start cli-options. This should be a mapping between option string to value string.	dict	{}	No	Yes
stop	Benchmon-stop cli-options. This should be a mapping between option string to value string.	dict	{}	No	Yes

logs schema

Key	Description	Type	Default	Required	Nullable
filter_plugins	List of filter plugins with their configuration	list of dictionary	null	Yes	No
consumer_plugins	List of consumer plugins with their configuration	list of dictionary	null	Yes	No

Each filter or consumer plugin may contain following keys:

Key	Description	Type	Default	Required	Nullable
name	The plugin class. An instance of this class will be created by log monitor.	string		Yes	No
kwargs	Dictionary of arguments to pass to the plugin (specific to the plugin).	dictionary		No	No
path	Optional path to the plugin module file from which the plugin class will be imported by log monitor	string		No	No