The required functionality of the workflow library is as follows.
Starting, monitoring and ending a workflow¶
At the start
Claim the processing block.
Get the parameters defined in the processing block. They should be checked against the parameter schema defined for the workflow.
Make requests for input and output buffer space. The workflow will calculate the resources it needs based on the parameters, then request them from the processing controller. This is currently a placeholder.
Declare workflow phases
Workflows will be divided into phases such as preparation, processing, and clean-up. In the current implementation, only one phase can be declared, which we refer to as the ‘work’ phase.
Execute the work phase
On entry to the work phase, it waits until the resources are available. Meanwhile it monitors the processing block to see it has been cancelled. For real-time workflows, it also checks if the scheduling block instance has been cancelled.
Deploy execution engines to execute a script/function.
Monitor the execution engines and processing block state. Waits until the execution is finished, or the processing block is cancelled.
At the end
Remove the execution engines to release the resources.
Update processing block state with information about the success or failure of the workflow.
Get IP and MAC addresses for the receive processes.
Monitor receive processes. If any get restarted, then the addresses may need to be updated.
Write the addresses in the appropriate format into the processing block state.
The library can be installed using
pip but you need to make sure to use the
SKA artefact repository as the index:
pip install \ --index-url https://artefact.skao.int/repository/pypi-all/simple \ ska-sdp-workflow
To install it using a
requirements.txt file, the
pip options can be
added to the top of the file like this:
--index-url https://artefact.skao.int/repository/pypi-all/simple ska-sdp-workflow
Once the SDP workflow library have been installed, use: