SKA Mid.CBF Engineering Console
Documentation on the Developer’s portal: ReadTheDocs
Code repository: ska-mid-cbf-engineering-console
The Engineering Console is being built in a Docker container, which insulates it from variations in the server environment. In addition to enabling MCS-Talon integration and testing, this container can be used to provide a controlled environment for automated end-to-end Talon HW testing.
The Engineering Console Docker is built in the pipeline and deployed to the Central Artefact Repository CAR.
Installation
git clone https://gitlab.com/ska-telescope/ska-mid-cbf-engineering-console
cd ska-mid-cbf-engineering-console
git submodule init
git submodule update
poetry install # to create the virtual environment with all dependencies
poetry shell # to run shell in the virtual environment
make oci-build # or "poetry run make oci-build" if not in the poetry shell
make run # runs "hello world" test
Usage
Run the Docker interactively
To run the docker interactively:
make run-interactive
which opens a bash shell in the docker. To see available test script options:
./talondx.py --help
To install vim/nano editors while in interactive mode:
apt-get update
apt-get -y install vim nano
Generate Talondx Config FIle
To auto-generate the talondx config file based on the board configuration. Run the following command:
make generate-talondx-config BOARDS="<BOARDS>"
where
Download Artefacts from CAR
To download FPGA bitstreams and Talon Tango device binaries from CAR to the local folder specified in the Makefile (TALONDX_DEST_DIR
):
make download-artifacts
or specify a different destination folder:
make download-artifacts TALONDX_DEST_DIR="destination-folder"
A different config JSON can be specified if it exists as well (default value in the Makefile);
make download-artifacts
To upload new FPGA bitstreams to CAR for use, see the ska-mid-cbf-talondx project
Optional: Override DS Artefacts with local build
In order for this script to work, ensure to clone and build your device servers in the same root directory: Example: If clone ds-vcc and ds-lstv-gen device servers ensure both are cloned under the same directory which would like like:
/home/user/dev/ds/ds-lstv-gen
/home/user/dev/ds/ds-vcc
To override the device servers (ds-lstv-gen,ds-vcc in this example) run the following command:
make ds_list=ds-lstv-gen,ds-vcc ds_basedir=<path to ds base directory> mcs_dir=<path to mcs checkout> ds-override-local
where ds_basedir is the path to the device server root directory of clone, /home/user/dev/ds from the previous example
Update the Tango DB inside MCS
make config-db
This command adds the Talon device servers as specified in the talondx-config.json
file.
Note: the artefacts need to be downloaded before updating the database (the artefacts contain detri JSON files needed for the DB update).
Pull and run the Docker from CAR
docker pull artefact.skao.int/ska-mid-cbf-engineering-console:0.0.2
docker run artefact.skao.int/ska-mid-cbf-engineering-console:0.0.2
Read the Docs
The Engineering Console project auto-generates Read the Docs documentation, which includes this README.
To re-generate the documentation locally prior to checking in updates to Git:
make documentation
To see the generated documentation, open /ska-mid-cbf-engineering-console/docs/build/html/index.html
in a browser – e.g.,
firefox docs/build/html/index.html &
Configure the Talon-DX Boards from MCS
The Talon DX boards can be configured with binaries from CAR using a combination of Engineering Console and MCS both running the Dell Server – see MCS-Talon+Integration for details.
How to Run the On Command Sequence
1. Install MCS and Engineering Console
Install MCS, then Install Engineering Console.
2. Download Artefacts
Follow the instructions in Download Artefacts from CAR
3. Start MCS
Follow the instructions in MCS up to the make install-chart
step to get MCS running.
4. Configure Tango DB
Follow the steps in Update the Tango DB inside MCS.
5. Ensure that Talon DX Boards are Off
Run the commands on either the Dell1 or Dell2 servers:
To check the current power status:
/shared/talon-dx-utilities/bin/talon_power_lru.sh <lru>
If the boards are powered on, power off the boards:
/shared/talon-dx-utilities/bin/talon_power_lru.sh <lru> off
Where
6. Send On Command from MCS
Run the required MCS On command sequence using:
make mcs-on
7. Read Talon HPS Device Version and Status Information
To display Talon DS version information (version, build date, Git commit hash):
make talon-version
To repeatedly display the current Talon DS state and status:
make talon-status
Configure and Execute a VCC Scan
This section assumes that you have followed all the steps in the previous section to power on and configure the Talon boards. At this point the three MCS devices that were configured in the previous step should be in the ON state. To perform a VCC scan operation, use the following command:
make mcs-vcc-scan
Configure and Execute an FSP Scan
This section assumes the previous VCC Scan section has been completed successfully. To perform an FSP scan operation, follow the sequence below:
Update the delay models from inside the engineering console docker container.
make run-interactive
python3
>>> import fsp_integration
>>> fsp = fsp_integration.FspIntegrationTools(<ID-number-of-talon-under-test>)
>>> fsp.update_delay_models()
Outside of the docker, execute the SetFunctionMode, ConfigureScan and Scan sequence
make mcs-fsp-scan
Notes on Dish Packet Capture
1. On Command Sequence
3. Set up BITE devices
From the root directory of the engineering console run the following:
make talon-bite-config BOARDS="<BOARDS>"
where
NOTE: Only the talon board defined in TALON_UNDER_TEST will be configured with the destination MAC address therefore only data from this board will reach the destination interfaces.
4. Dish Packet Capture
Open a new terminal. From the root directory of the engineering console run the following:
make dish-packet-capture
This command will not exit until the following command is run.
5. Start LSTV Replay
From the root directory of the engineering console run the following:
make talon-bite-lstv-replay BOARDS="<BOARDS>"
where
Notes on Signal Chain Verification
Wideband State Count
Collect WB state count histogram and power spectrum vectors
make wb-state-count-capture
Generate a report from the WB state count vectors collected by the previous command.
make wb-state-count-report
Set WB_STATE_COUNT_LOCAL_DIR to specify the directory to store the outputs. By default this is ./mnt/wb-state-count
Notes on MCS Interfaces
Commands
MCS commands can additionally be sent from Taranta (previously known as Webjive) or the itango3 shell.
Send the On command to CBF Controller from Taranta
Taranta needs to be enabled in MCS – see Taranta instructions for details.
Send commands to CBF Controller from itango3 shell
$ kubectl exec -it cbfcontroller-controller-0 -n ska-mid-cbf -- itango3
Defaulted container "device-server" out of: device-server, wait-for-configuration (init), check-dependencies-0 (init), check-dependencies-1 (init), check-dependencies-2 (init), check-dependencies-3 (init), check-dependencies-4 (init), check-dependencies-5 (init), check-dependencies-6 (init), check-dependencies-7 (init)
ITango 9.3.3 -- An interactive Tango client.
Running on top of Python 3.7.3, IPython 7.21 and PyTango 9.3.3
help -> ITango's help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
IPython profile: tango
hint: Try typing: mydev = Device("<tab>
In [1]: cbf_controller = DeviceProxy("mid_csp_cbf/sub_elt/controller")
In [2]: cbf_controller.State()
Out[2]: tango._tango.DevState.ON
In [3]: cbf_controller.Status()
Out[3]: 'The device is in OFF state.'
In [4]: cbf_controller.On()
Out[4]: [array([0], dtype=int32), ['On command completed OK']]
In [5]: cbf_controller.Status()
Out[5]: 'The device is in OFF state.'
In [6]: cbf_controller.State()
Out[6]: tango._tango.DevState.ON
Send ConfigureScan command from itango3 shell
In [1]: controller = DeviceProxy("mid_csp_cbf/sub_elt/controller")
In [2]: subarray = DeviceProxy("mid_csp_cbf/sub_elt/subarray_01")
In [3]: controller.On()
Out[3]: [array([0], dtype=int32), ['On command completed OK']]
In [4]: subarray.AddReceptors(["MKT000", "MKT001", "MKT002", "MKT003"])
Out[4]: [array([0], dtype=int32), ['CBFSubarray AddReceptors command completed OK']]
In [5]: f = open("tests/data/ConfigureScan_basic.json")
In [6]: subarray.ConfigureScan(f.read().replace("\n", ""))
Out[6]: [array([0], dtype=int32), ['CBFSubarray Configure command completed OK']]
or paste the following into the itango3 shell:
controller = DeviceProxy("mid_csp_cbf/sub_elt/controller")
subarray = DeviceProxy("mid_csp_cbf/sub_elt/subarray_01")
controller.On()
subarray.AddReceptors(["MKT000", "MKT001", "MKT002", "MKT003"])
f = open("tests/data/ConfigureScan_basic.json")
subarray.ConfigureScan(f.read().replace("\n", ""))
Note: the test file tests/data/ConfigureScan_basic.json
is part of the MCS codebase and is available when connected using itango3.
Send Scan command to VCC from itango3 shell
vcc = DeviceProxy("mid_csp_cbf/vcc/002")
vcc.simulationMode = 0
vcc.adminMode = 0
vcc.On()
vcc.ConfigureBand("1") # Only bands 1 and 2 are supported by the HPS software
vcc.ConfigureScan("{\
\"config_id\": \"test_config\",\
\"frequency_band\": \"1\",\
\"frequency_band_offset_stream_1\": 5,\
\"frequency_band_offset_stream_2\": 0,\
\"rfi_flagging_mask\": "",\
\"fsp\": [\
{\
\"fsp_id\": 1,\
\"frequency_slice_id\": 3,\
\"function_mode\": \"CORR\"\
}\
]\
}") # This is an example of the expected argument format for the VCC
vcc.Scan("6") # Use any arbitrary integer ID
vcc.EndScan()
Logs
View logs from a single MCS pod in the terminal
To see the CBF controller logs:
kubectl logs -f cbfcontroller-controller-0 -n ska-mid-cbf
where cbfcontroller-controller-0 is the pod name shown when running make watch
in MCS.
View logs using K9S
k9s -n ska-mid-cbf
then select the pod (e.g., cbfcontroller-controller-0) and press l
to view the logs.
Notes
Raw Repository
FPGA bitstreams are uploaded manually to the raw repository in CAR (Common Artefact Repository, https://artefact.skatelescope.org/) here:
raw-internal/ska-mid-cbf-talondx/fpga-test/talon_dx-{_bitstream-name_}-v{_version_}.tar.gz
Example - manually package the BITE bitstream files
mkdir bin
cp bite5.json bin/
cp mvp5_wip02.core.rbf bin/
cp mvp5_wip02.dtb bin/
tar -cvf talon_dx-bite-v0.5.0.tar bin
gzip -k talon_dx-bite-v0.5.0.tar
Example - manually unpackage the BITE bitstream files
gzip -d talon_dx-bite-v0.5.0.tar.gz
tar -xvf talon_dx-bite-v0.5.0.tar
where {version} is in the X.Y.Z format.