Example Usage with EAGLE

In this page we briefly describe how to use JACAL in a DALiuGE graph. This assumes you already built JACAL.

Outline

In this example we will replicate one of the unit tests run in the GitLab CI pipeline, namely test_basic_imaging. This test performs basic imaging on an input MeasurementSet using the CalcNE and SolveNE JACAL components, via the EAGLE web interface.

In DALiuGE a program is expressed as a graph, with nodes listing applications, and the data flowing through them. Graphs come in two flavours: logical, expressing the logical constructs used in the program (including loops, gather and scatter components), and physical, which is the fully-fledged version of a logical graph after expanding all the logical constructs.

This test is expressed as a logical graph. After translation into a physical graph it is submitted for execution to the DALiuGE managers, which were started by running the docker containers during installation. During execution one can monitor the progress of the program. This is all handled via a browser using EAGLE.

Preparing the Graph

For this test you will need to download the JSON logical graph. and the input dataset. When running Yandasoft applications, one typically provides a text file known as a parset containing configuration options. However this graph uses a DALiuGE drop to provide the configuration options. First, download the graph to a local directory:

$> wget https://gitlab.com/ska-telescope/ska-sdp-jacal/-/raw/master/jacal/test/daliuge/test_basic_imaging_parset.json

Running image icrar/daliuge-engine:jacal generates a temporary work directory that is accessible both inside and outside the docker container:

$> ls -l /tmp/.dlg
code/
logs/
testdata/
workspace/

The graph assumes that the dataset is located in the testdata directory, so place it there:

$> cd /tmp/.dlg/testdata
$> wget https://gitlab.com/ska-telescope/ska-sdp-jacal/-/raw/master/data/chan_1.ms.tar.gz
$> tar xf chan_1.ms.tar.gz

The final assumption in the graph is the specificatin of the JACAL library within the deocker container, which is set to /home/build/jacal/libjacal.so.

Inspecting and Editing the Graph

Deploying a Graph Locally

After installation on the local host, the DALiuGE translator will be connected to port 8084 and EAGLE to port 8888. Open a browser and go to http://localhost:8888/. This will bring up the EAGLE visual workspace. From the Graph drop-down menu, select Local Storage and then Load Graph. Navigate to test_basic_imaging_parset.json and load it into the browser.

_images/test_basic_imaging_parset.png

This graph performs imaging with a single major cycle of deconvolution. The graph is just for demonstration and the cleaning is not very deep. See the imaging overview for more details. Clicking on each component brings up an Inspector in the right-hand panel that can be used to read and edit run-time parameters. Of particular interest are Component Parameters and I/O Ports. The Config component generates imaging parset, and imaging parameters can found by inspecting its Component Parameters.

Note

To edit Component Parameters some of the Advanced Editing setting may need to be altered via the cog in the top right corner.

Before translating and deploying the graph, a few options need to be configured. The most important is the location of the EAGLE translator. The cog in the top right corner brings up EAGLE configuration settings, and under External Services is the Translator URL box. Set this to http://localhost:8084/gen_pgt, the local translator running in the container with port 8084 forwarded from the localhost. Setting GitHub and GitLab Access Tokens can also be done under External Services, allowing EAGLE to access various repositories.

The logical graph is now ready to be partitioned into a physical graph. From the Translation tab in the right-hand panel, select Generate PGT to generate a Physical Graph Template that can be mapped to compute nodes. This should open the DALiuGE translator interface in a new browser tab. Click on the Translator settings cog and check the DALiuGE Manager URL. This should be set to port 8001 of the actual IP address, e.g. http://130.155.199.71:8001/. Now select Deploy to generate and deploy the physical graph. This brings up a new browser tab which displays the progress of the graph.

_images/graph_progress.png

When the graph has completed, the resulting images can be found in /tmp/.dlg/workspace. These are shown in the following figure. From the top left is the initial dirty image, the clean component image, the residual image and the restored image.

_images/test_basic_imaging_parset_output.png

Runtime logs can be displayed via docker:

$> docker logs daliuge-engine

Deploying a Graph Remotely

Same as above but point to hostname or IP address rather than localhost.

Deploying a Graph on a Cluster

Todo

Translating

Deploying