Installation/Usage

Installation

git clone https://gitlab.com/ska-telescope/ska-mid-cbf/infra/ska-mid-cbf-emulators
cd ska-mid-cbf-emulators
git submodule init
git submodule update

# add certificate files, if needed
cp <my_cert_file(s)> ./certs

Using Minikube

CPU Usage

IMPORTANT: Minikube must be started with a significant amount of CPU power, otherwise RabbitMQ may randomly crash when it hits a CPU spike. We recommend 8 cores minimum (ideally 10), but slightly fewer may also work. The default of 4 cores is wholly insufficient.

Helm Configuration

Configurations for the spin-up of the emulator exist in values.yaml. A custom values file called generated_values.yaml is generated at deploy time which is based on this original file.

The properties field specifies default values for all emulator instances. Any field allowed in instances (see below) may have a default specified here.
The instances field contains the list of emulators you wish to deploy.
When running a full FHS deployment, this will also serve as the list of VCC or FSP device stacks that will be deployed. Each entry inherits properties from the properties field, including emulatorId and emulatorVerbosity, which may be overridden at the instance level. Each entry looks like:
```
- name: fhs-vcc-1
  deviceId: "1"
  bitstreamPath: "/app/mnt/bitstream"
  bitstreamId: "agilex-vcc"
  bitstreamVersion: "0.0.12"
  emulatorId: "vcc-emulator-1"
  emulatorVerbosity: "3"
  initialSignal:
    packet_rate: 1500000
    sample_rate: 3960000000
    dish_id: "MKT001"
    band_id: "1"
    regular_sky_power: [0.2, 0.3]
    noise_diode_on_power: [0.3, 0.4]
    duty_cycle: 0.50
```
- Each property may be omitted if there is a suitable default in the properties field (except for initialSignal, which may always be omitted).
- One pod will be spun up for each entry in this list, with the pod name matching the emulatorId field.
- The bitstreamId and bitstreamVersion fields are used to look up the bitstream package for this emulator, by pulling the file at https://artefact.skao.int/repository/raw-internal/ska-mid-cbf-<bitstreamId>-<bitstreamVersion>.tar.gz.
- The emulatorVerbosity field will set the default console verbosity level of all emulators. Defaults to 3 if not provided. Available console verbosity levels are:
  - 0 [NONE] – print no logs
  - 1 [ERROR] – print only ERROR and/or CRITICAL messages, i.e. those which should garner immediate attention
  - 2 [WARNING] – additionally print WARNING messages, i.e. those which should be noted, but may not necessarily denote critical issues
  - 3 [INFO] (default) – additionally print INFO messages, i.e. those which provide general neutral information that may be useful to an average user
  - 4 [DEBUG] – additionally print DEBUG messages, i.e. those which provide information that may be useful to an advanced user, sysop, or developer
  - 5 [TRACE] – additionally print TRACE messages, i.e. those which refer to a specific code location and are intended only for developers
- If the initialSignals property is specified, its contents will override the default initial_signals_config.json file for that instance (see Initial signals configuration file for details). If this property is not provided, the default file will be used instead. This property is never required, but if it is provided, its elements should adhere to the signal specification.
- Each emulator’s API is exposed at <emulatorId>.<namespace>.svc.cluster.local:5001.

The router.routes field defines a route mapping between bitstream emulators for emulated signals (see: Signal Router). The YAML format is shown below. Refer to Route Configuration for details on the values.

routes:
  <source_emulator_id>:
    <source_ip_block_id>:
      <tag>:
        - dest_bitstream_emulator_id: "<id>"
          dest_ip_block_id: "<id>"
        - dest_bitstream_emulator_id: "<another_id>"
          dest_ip_block_id: "<another_id>"
      <tag2>:
        ...
      default:
        ...
    <source_ip_block_id2>:
      ...
  <source_emulator_id2>:
    ...

To test locally with a custom bitstream, there are 2 options:
- For a bitstream in a Gitlab branch, add e.g. the following to values.yaml:
```
gitlab_bitstream_url_override: "https://gitlab.com/ska-telescope/ska-mid-cbf/infra/ska-mid-cbf-bitstreams/-/archive/cip-2957/ska-mid-cbf-bitstreams-cip-2957.tar.gz?path=raw/ska-mid-cbf-agilex-vcc"
```
  You can find this URL by navigating to the bitstream repo on GitLab (https://gitlab.com/ska-telescope/ska-mid-cbf/infra/ska-mid-cbf-bitstreams), navigating into your branch, and then navigating into the raw/ska-mid-cbf-<bitstream_id> folder. Then click the “Code” dropdown, and copy the “tar.gz” link under “Download this directory”. Alternatively, you can just copy the above and substitute in your branch name and bitstream ID.
- For a completely local bitstream, add it to the root folder of the local repo, e.g. ska-mid-cbf-emulators/bitstreams/0.0.12/agilex-vcc/, and then change the value of bitstreamMountPath in values.yaml to /app/bitstreams. Ensure the bitstream download job is not being run (add ENABLE_BITSTREAM_DOWNLOAD=false when running k8s-deploy).

Running the Emulator (Minikube)

Ensure Minikube is running, and that you have run:

eval $(minikube docker-env)

Then run

make k8s-deploy [DEV=false] [SKIP_BUILD=false] [ENABLE_BITSTREAM_DOWNLOAD=true]

to spin up the pods.

Available options are:

DEV (default=false): If true, run in “development mode”: the emulator images will be built and loaded locally from the current repo state. Otherwise, run in “production mode”: pull images from Harbor using the tags specified in values.yaml.

SKIP_BUILD (default=false): If true and DEV=true, will skip re-building the images before loading them into minikube. It is assumed in this case that the images have already been built locally at least once. Ignored if not in development mode.

ENABLE_BITSTREAM_DOWNLOAD (default=true): If true, will download the bitstream from CAR before instantiating the emulators. Only set this to false if sourcing a bitstream from elsewhere (e.g. in a higher level umbrella chart) or manually overriding the bitstream (see above).

Local images and SSL bypass

Depending on your circumstances, you may want to build the emulator images locally, and/or may need to deal with minikube SSL issues.

To build the emulator images locally, run:

make oci-build-all

If facing SSL issues, make sure all required images have been either built or pulled via Docker.

Once all necessary images have been built or pulled, you can then run

minikube image load <image>:<tag>

to manually load the image into the minikube environment (this takes a little while). From there, ensure that the imagePullPolicy of all containers is set to Never (to force the local image to be used). You can view all the currently loaded images via:

minikube image ls

Once the chart is installed, you can use a GUI like k9s, or run Kubernetes commands through minikube CLI:

minikube kubectl -- <command> -n <namespace>

For example, to view the list of pods:

minikube kubectl -- get pods -n ska-mid-cbf-emulators

Or to open an interactive shell inside a pod:

minikube kubectl -- exec -n ska-mid-cbf-emulators <pod_name> -i -t -- bash

To stop running the emulator and terminate all pods, run:

make k8s-uninstall-chart

Windows GUI

If using minikube through WSL2 on Windows, you can copy the kubeconfig file at ~/.kube/config to a mounted drive to then use in a K8s GUI such as Lens.

Windows VPN

If using a VPN on Windows, you may face network issues when pulling Docker images in WSL. Run the command

Get-DnsClientServerAddress

in a PowerShell window, then find your primary connection and copy each IP address listed there to a new line in /etc/resolv.conf, in the same format as whatever is already in there. For example, for the result

Ethernet 2                          18 IPv4    {1.2.3.4, 10.11.12.13}

you should add the lines

nameserver 1.2.3.4
nameserver 10.11.12.13

to /etc/resolv.conf.

If you continue to get TLS handshake errors, simply keep trying the Docker command, as it tends to throttle pulls a lot.

If you get certificate errors in Docker (not within minikube), you may simply have to close and reopen your WSL instance.

Using Docker

There are three methods of running a bitstream emulator via Docker:

Default/production: Runs the emulator and injector automatically and immediately.
Dev: Sets up the necessary utilities and environment for working on the emulators; does not start the emulator automatically.
Firmware Dev: Similar to Dev but with additional modifications to facilitate development from a firmware perspective. Mounts a FW workspace directly and provides extra command wrappers for simple interaction with the emulator.

Environment variables are used to determine the bitstream emulator ID and bitstream path. Keep in mind that the Docker Compose setup only supports running a single bitstream emulator at a time.

LOGGING_MODE is the logging mode to use for the bitstream emulator. May be dev (output to console), prod (output only to logfile), or all (output to console & logfile).

EMULATOR_ID is the ID to use for the bitstream emulator. In the dev environment, this is set by default to default-emulator.

BITSTREAM_ROOT_PATH is the first of two possible ways used to find the IP block emulators for this bitstream emulator. This path should point to the root of a bitstream tarball (i.e., the folder containing emulators) such that the following path is valid: $BITSTREAM_ROOT_PATH/emulators/<ip_block>/emulator/api.py. In the dev environment, this is downloaded and set by default to /emulator_dev/ska-mid-cbf-bitstreams/raw/ska-mid-cbf-agilex-vcc/. This variable must exist if not using EMULATORS_PATH.

EMULATORS_PATH is the second of two possible ways used to find the IP block emulators for this bitstream emulator. This path should point to a folder such that the following path is valid: $EMULATORS_PATH/<ip_block>/emulator/api.py. Examples are the emulators folder in a bitstream package, or a firmware workspace folder. In the firmware dev environment, this is set by default to the mounted workspace folder /workspace. This variable is optional and will take priority over BITSTREAM_ROOT_PATH if provided.

From within the container, the bitstream emulator’s API is exposed at http://emulator:5001 (preferred by Docker) or http://localhost:5001.

Running the Production Images

make run  # build and run the emulator/injector

make enter  # open a bash shell in the emulator image (remember that the emulator process is already running in the background)

make view-logs  # view the emulator image logs

make remove  # tear down the images

Running the Standard Dev Environment

Build and Start the Dev Environment

This will automatically clone the bitstream into /emulator_dev/ska-mid-cbf-bitstreams/raw/ska-mid-cbf-agilex-vcc/, and set the environment variable BITSTREAM_ROOT_PATH to this path. You may edit the bitstream files on the host machine, and they will be automatically synced to the emulator container’s emulator_dev folder through a mount.

make run-dev # build and run the dev environment
make run-dev-no-injector # alternatively, build and run the dev environment without a separate injector image

make enter-dev  # open a bash shell in the dev environment

make view-logs-dev  # view the dev environment image logs

make remove-dev  # tear down the images
make remove-dev-no-injector

Running the Emulator and/or Injector

Before anything else, ensure you have the emulator repository mounted or cloned somewhere accessible within the dev environment.

Run the emulator in the dev environment

cd /emulator_dev/<path_to_emulators_repo>/images/ska-mid-cbf-emulators-emulator
poetry install  # only need to do this the first time, or if dependencies are updated
poetry run python app.py [-v <console_verbosity>] [-p <pulse_interval>] [-c <config_file>] [-s <signal_config_file>]

Options:

-p INTERVAL, --pulse-interval=INTERVAL: Interval, in seconds, between pulses. Must be at minimum 0.1 (lower values will be capped). Note that setting this value too low may result in unexpected behavior, especially on slower systems. Default is 1.0.
-c FILE, --config-file=FILE: A custom IP block configuration file to override the config.json file included with the bitstream, if desired.
-s FILE, --signal-config-file=FILE: A custom initial signal configuration file to override the initial_signal.json file in the emulator app folder, if desired.
-v VERBOSITY, --verbosity=VERBOSITY: Logging verbosity of the emulator. See the Helm Configuration section for available verbosity options. Defaults to the EMULATOR_VERBOSITY variable if it exists, or 3 (INFO) if it doesn’t.

Run the injector manually in the injector-less dev environment

cd /emulator_dev/<path_to_emulators_repo>/images/ska-mid-cbf-emulators-injector
poetry install  # only need to do this the first time, or if dependencies are updated
poetry run python app.py [-v <console_verbosity>]

Options:

-v VERBOSITY, --verbosity=VERBOSITY: Logging verbosity of the injector. See the Helm Configuration section for available verbosity options. Defaults to the EMULATOR_VERBOSITY variable if it exists, or 3 (INFO) if it doesn’t.

Interact with the emulator and injector via the API

# GET the config.json (as a JSON string) that was used at instantiation of the controller
curl -v http://emulator:5001/config | jq

# GET a PNG of the graph of the emulator configuration
curl -v http://emulator:5001/graph -o <output_filename>.png | jq

# GET the state of the emulator controller
curl -v http://emulator:5001/state | jq

# POST start command to the ethernet_200g block
curl -v -X POST http://emulator:5001/ethernet_200g/start | jq

# GET the status of the ethernet_200g block
curl -v http://emulator:5001/ethernet_200g/status | jq

# POST configure command with data to the wideband_input_buffer block
curl -v -d '{"expected_sample_rate": 1000, "expected_dish_band": 1}' http://emulator:5001/wideband_input_buffer/configure | jq

# POST injector events from commandline (this will, in all likelihood, rarely if ever happen)
curl -v -d '{"injector_event_groups": [{"bitstream_emulator_id": "default-emulator", "ip_block_emulator_id": "ethernet_200g", "events": [{"severity": "WARNING", "value": {"id": "inj0001", "injection_type": "update_link_badness", "message": "Degrade the link.", "badness": 0.75}}]}]}' -H 'Content-Type: application/json' http://localhost:5002/inject | jq

# POST injector events from a JSON file
curl -v -d '@path/to/events.json' -H 'Content-Type: application/json' http://injector:5002/inject | jq

Running the Firmware Dev Environment

Note: see Emulator Development Using the FW-Dev Container for a detailed guide on setting up this environment.

Build and Start the Dev Environment

This will mount the folder you specify in the WORKSPACE=<FW_workspace_path> option to /workspace in the dev environment, and will automatically clone the emulators into /ska-mid-cbf-emulators/.

make run-fw-dev WORKSPACE=<FW_workspace_path>  # build and run the FW dev environment

make enter-fw-dev  # open a bash shell in the FW dev environment

make view-logs-fw-dev  # view the dev environment image logs

make remove-fw-dev  # tear down the images

Running the Emulator

Run the emulator in the FW dev environment

run-emulator [options]

OR

cd /ska-mid-cbf-emulators/images/ska-mid-cbf-emulators-emulator
poetry run python app.py [-v <console_verbosity>] [-p <pulse_interval>] [-c <config_file>] [-s <signal_config_file>]

See Run the emulator in the dev environment for the detailed options.

Interact with the Emulator and Injector APIs

Common tasks can be invoked using the invoke command. The general syntax for commands is:

invoke <command_name> <command_options>

For IP block API calls:

invoke ip.<command_name> <ip_block_id> [--params=<params_json>] <json_body>

for example

invoke ip.configure ethernet_200g '{"rx_loopback_enable": false}'
invoke ip.status ethernet_200g --params='{"clear": false}'

Available commands are:

invoke --list
  get                             Perform a GET request to the emulator controller
  post                            Perform a POST request to the emulator controller
  start-emulator                  Start the emulator server engine located in images/ska-mid-cbf-
                                  emulators-emulator.
  start-injector                  Start the injector located in images/ska-mid-cbf-
                                  emulators-injector.
  ip.configure
  ip.deconfigure
  ip.get                          Perform a GET request to an IP block
  ip.post                         Perform a POST request to an IP block
  ip.recover
  ip.start
  ip.status
  ip.stop
  controller.config               Get the emulator configuration from this emulator bitstream.
  controller.initial-signals      Get the initial signals used to instantiate this emulator.
  controller.graph                Plot the IP block graph as a PNG image.
  controller.server-healthcheck   Verify the API is accepting requests. Always returns 200 OK
                                  and an empty body.
  controller.start                Start the emulator if it is stopped. Pulses and event listeners will
                                  reactivate.
  controller.state                Get the current state of the emulator.
  controller.stop                 Stop the emulator if it is started. The API server will remain active
                                  but pulses and event listeners will be halted.
  controller.terminate            Terminate the emulator.
  injector.inject                 Injects a JSON InjectorEvents request.

Details of each command can be retrieved via the --help option, for example:

invoke --help get
Usage: inv[oke] [--core-opts] get [--options] [other tasks here ...]

Docstring:
  Perform a GET request to the emulator controller.

  Usage: invoke get <endpoint> [--params=<params_json>]

Options:
  -e STRING, --endpoint=STRING   command endpoint
  -h STRING, --hostname=STRING   hostname url of emulator
  -p STRING, --params=STRING     query parameters