pipeline

Class for simulating a pipeline step in a scheduling block instance.

class ska_sdp_resource_model.simulate.pipeline.Pipeline(name, config)

Bases: object

Class for simulating a pipeline step in a scheduling block instance.

__init__(name, config)

Initialises the Pipeline object with the given name and configuration.

run(env, sdp, batch_name)

Simulates the run of a single pipeline step in a scheduling block instance.

allocate_compute_nodes(env, sdp)

Allocates compute nodes for a given pipeline.

get_runtime()

Calculates the total runtime based on node hours, percentage of parallelism, and nodes available.

get_event_log()

Return the event log for the pipeline.

retain_data_products(sdp)

Simulate retention of data products. This method will simulate holding data in capacity storage until the time given by the data retention parameter in the scheduling block types config and then releasing them from storage after this time.

allocate_capacity_storage_data_products(env, sdp)

Allocates capacity storage for data products.

allocate_capacity_storage_data_products(env, sdp)

Allocate storage for data products.

This method requests a number of capacity storage based on the size of the data products. It logs the request and allocation process, and records the wait time for capacity storage allocation.

Parameters:

• env (simpy.Environment) – A simpy simulation environment.

• sdp (ScienceDataProcessor) – A ScienceDataProcessor object.

Returns:

simpy.events.Event – Events for requesting and receiving capacity storage.

allocate_compute_nodes(env, sdp)

Allocate compute nodes for a given pipeline.

This method requests a number of compute nodes based on the percentage of parallelism required by the pipeline. It logs the request and allocation process, and records the wait time for compute node allocation.

Parameters:

• env (simpy.Environment) – A simpy simulation environment.

• sdp (ScienceDataProcessor) – A ScienceDataProcessor object.

Returns:

simpy.events.Event – Events for requesting and receiving compute nodes.

get_event_log()

Return the event log for the pipeline.

get_runtime()

Calculate the total runtime based on node hours, percentage of parallelism, and nodes available.

retain_data_products(env, sdp)

Simulate retention of data product.

This method will simulate holding data in capacity storage until the time given by the data retention parameter in the scheduling block types config and then deleting them from storage after this time.

Parameters:

sdp (ScienceDataProcessor) – A ScienceDataProcessor instance.

Yields:

simpy.events.Process – The process of requesting and storing retained data.

run(env, sdp, batch_name)

Simulate resource usage of a single pipeline step in a scheduling block instance.

Parameters:

• env (simpy.Environment) – A simpy simulation environment.

• sdp (ScienceDataProcessor) – A ScienceDataProcessor object.

• batch_name (str) – The name of the scheduling block instance.

Yields:

simpy.events.Event – Events for allocating compute nodes and running the pipeline.