Welcome to the SDP Resource Model’s documentation!

The SDP Resource Model is a Python application for simulating and visualising the data storage and compute requirements of the SDP for during batch processing, given an observation schedule.

Simulation overview

A flow chart of the simulation process is shown below (click to zoom).

SDP Resource Model

Assumptions

The SDP Resource Model makes the following assumptions:

  • There is only one telescope available

  • Observations and data processing are executed in the order they are scheduled (unless the shuffle option is enabled - see here for more information)

  • Receive & real-time processing can be ignored

  • Scheduling blocks can comprise multiple scheduling block instances (see here for more information on how these are defined) - raw visibilities will accumulate in storage and batch processing will be executed after the final observation of the last instance. There is no data retention period for raw visibilities.

  • Data products generated by pipelines are kept in data product storage from when the pipeline finishes until a data retention period, which is specified in the scheduling block type configuration and is the same for all pipelines in the block.

  • Observations are delayed if there is insufficient capacity storage for raw visibilities.

  • Storage for pre-processed visibilities is allocated and released for each scheduling block when processing of these starts and ends.

  • The number of nodes allocated to an individual pipeline is fixed for the duration of the pipeline (set by its num_nodes parameter).

  • Pipelines with the same priority level will be executed in parallel only if resources are available.

  • There is no overhead in spinning up a node.

Note

The SDP Resource Model is a work in progress and is subject to change.