science_data_processor

Module containing class for simulating the SKA Science Data Processor (SDP).

class ska_sdp_resource_model.simulate.science_data_processor.ScienceDataProcessor(env, hardware_config)

Bases: object

A class to simulate resource usage of a Science Data Processor (SDP).

storage_capacity

A container representing the storage capacity in petabytes (PB).

Type:

simpy.Container

storage_performance

A container representing the storage performance in terabytes (TB).

Type:

simpy.Container

compute_nodes

A container representing the number of compute nodes.

Type:

simpy.Container

nodes_log

A list to store logs of compute node usage.

Type:

list

storage_capacity_log

A list to store logs of storage capacity usage.

Type:

list

storage_performance_log

A list to store logs of storage performance usage.

Type:

list

logger

A logger instance for logging resource usage.

Type:

Logger

log_nodes_metrics()

Logs the current node usage of the SDP.

log_storage_capacity_metrics()

Logs the current storage capacity usage of the SDP.

log_storage_performance_metrics()

Logs the current storage performance usage of the SDP.

log_resource_usage()

Logs the current resource usage of the SDP.

resource_usage_df(observation_start_time)

Converts the resource logs to a DataFrame.

initialise_monitor()

Initialises the logging for the resources.

initialise_monitor()

Initialise logging for resources.

log_nodes_metrics()

Get resource usage metrics.

log_resource_usage()

Log resource usage of SDP.

Number of compute nodes, GB performance storage and GB capacity storage available.

log_storage_capacity_metrics()

Log resource usage for capacity storage.

log_storage_data_product_metrics()

Log resource usage for data product storage.

log_storage_metrics(kind)

Append resource usage metrics to the appropriate log.

log_storage_performance_metrics()

Log resource usage for performace storage.

resource_usage_df()

Convert resource logs to DataFrame.

Merges the logs for compute nodes, capacity storage and performance storage into a single DataFrame using the simulation time (time_s). Missing values are filled forward and backward. Duplicates are dropped and rows sorted by simulation time. Time columns are also added to store the simulation time in hours and days.

Returns:

resource_log_df (pd.DataFrame) – DataFrame containing resource logs.

ska_sdp_resource_model.simulate.science_data_processor.patch_resource(resource, pre=None, post=None)

Patch for resource requests.

Patch resource so that it calls the callable pre before each put/get/request/release operation and the callable post after each operation. The only argument to these functions is the resource instance.

Parameters:

• pre (callable) – A callable to be called before each request.

• post (callable) – A callable to be called after each request.