Installation/Usage ****************** Installation ============ .. code-block:: bash git clone https://gitlab.com/ska-telescope/ska-mid-cbf/infra/ska-mid-cbf-emulators cd ska-mid-cbf-emulators git submodule init git submodule update # add certificate files, if needed cp ./certs Using Minikube ============== .. admonition:: CPU Usage IMPORTANT: Minikube must be started with a significant amount of CPU power, otherwise RabbitMQ may randomly crash when it hits a CPU spike. We recommend 8 cores minimum (ideally 10), but slightly fewer may also work. The default of 4 cores is wholly insufficient. Helm Configuration ------------------ Configurations for the spin-up of the emulator exist in `values.yaml `_. A custom values file called ``generated_values.yaml`` is generated at deploy time which is based on this original file. * The ``properties`` field specifies default values for all emulator instances. Any field allowed in ``instances`` (see below) may have a default specified here. * The ``instances`` field contains the list of emulators you wish to deploy. * When running a full FHS deployment, this will also serve as the list of VCC or FSP device stacks that will be deployed. Each entry inherits properties from the ``properties`` field, including ``emulatorId`` and ``emulatorVerbosity``, which may be overridden at the instance level. Each entry looks like: .. code-block:: yaml - name: fhs-vcc-1 deviceId: "1" bitstreamPath: "/app/mnt/bitstream" bitstreamId: "agilex-vcc" bitstreamVersion: "0.0.12" emulatorId: "vcc-emulator-1" emulatorVerbosity: "3" initialSignal: packet_rate: 1500000 sample_rate: 3960000000 dish_id: "MKT001" band_id: "1" regular_sky_power: [0.2, 0.3] noise_diode_on_power: [0.3, 0.4] duty_cycle: 0.50 * Each property may be omitted if there is a suitable default in the ``properties`` field (except for ``initialSignal``, which may always be omitted). * One pod will be spun up for each entry in this list, with the pod name matching the ``emulatorId`` field. * The ``bitstreamId`` and ``bitstreamVersion`` fields are used to look up the bitstream package for this emulator, by pulling the file at ``https://artefact.skao.int/repository/raw-internal/ska-mid-cbf--.tar.gz``. * The ``emulatorVerbosity`` field will set the default console verbosity level of all emulators. Defaults to ``3`` if not provided. Available console verbosity levels are: * ``0 [NONE]`` -- print no logs * ``1 [ERROR]`` -- print only ``ERROR`` and/or ``CRITICAL`` messages, i.e. those which should garner immediate attention * ``2 [WARNING]`` -- additionally print ``WARNING`` messages, i.e. those which should be noted, but may not necessarily denote critical issues * ``3 [INFO]`` (default) -- additionally print ``INFO`` messages, i.e. those which provide general neutral information that may be useful to an average user * ``4 [DEBUG]`` -- additionally print ``DEBUG`` messages, i.e. those which provide information that may be useful to an advanced user, sysop, or developer * ``5 [TRACE]`` -- additionally print ``TRACE`` messages, i.e. those which refer to a specific code location and are intended only for developers * If the initialSignals property is specified, its contents will override the default ``initial_signals_config.json`` file for that instance (see :ref:`Initial signals configuration file` for details). If this property is not provided, the default file will be used instead. This property is never required, but if it is provided, its elements should adhere to the signal specification. * Each emulator's API is exposed at ``..svc.cluster.local:5001``. * The ``router.routes`` field defines a route mapping between bitstream emulators for emulated signals (see: :ref:`Signal Router`). The YAML format is shown below. Refer to :ref:`Route Configuration` for details on the values. .. code-block:: yaml routes: : : : - dest_bitstream_emulator_id: "" dest_ip_block_id: "" - dest_bitstream_emulator_id: "" dest_ip_block_id: "" : ... default: ... : ... : ... * To test locally with a custom bitstream, there are 2 options: * For a bitstream in a Gitlab branch, add e.g. the following to values.yaml: .. code-block:: yaml gitlab_bitstream_url_override: "https://gitlab.com/ska-telescope/ska-mid-cbf/infra/ska-mid-cbf-bitstreams/-/archive/cip-2957/ska-mid-cbf-bitstreams-cip-2957.tar.gz?path=raw/ska-mid-cbf-agilex-vcc" You can find this URL by navigating to the bitstream repo on GitLab (https://gitlab.com/ska-telescope/ska-mid-cbf/infra/ska-mid-cbf-bitstreams), navigating into your branch, and then navigating into the ``raw/ska-mid-cbf-`` folder. Then click the "Code" dropdown, and copy the "tar.gz" link under "Download this directory". Alternatively, you can just copy the above and substitute in your branch name and bitstream ID. * For a completely local bitstream, add it to the root folder of the local repo, e.g. ``ska-mid-cbf-emulators/bitstreams/0.0.12/agilex-vcc/``, and then change the value of ``bitstreamMountPath`` in values.yaml to ``/app/bitstreams``. Ensure the bitstream download job is not being run (add ``ENABLE_BITSTREAM_DOWNLOAD=false`` when running ``k8s-deploy``). Running the Emulator (Minikube) ------------------------------- Ensure Minikube is running, and that you have run: .. code-block:: bash eval $(minikube docker-env) Then run .. code-block:: bash make k8s-deploy [DEV=false] [SKIP_BUILD=false] [ENABLE_BITSTREAM_DOWNLOAD=true] to spin up the pods. Available options are: * ``DEV (default=false)``: If true, run in "development mode": the emulator images will be built and loaded locally from the current repo state. Otherwise, run in "production mode": pull images from Harbor using the tags specified in ``values.yaml``. * ``SKIP_BUILD (default=false)``: If true and ``DEV=true``, will skip re-building the images before loading them into minikube. It is assumed in this case that the images have already been built locally at least once. Ignored if not in development mode. * ``ENABLE_BITSTREAM_DOWNLOAD (default=true)``: If true, will download the bitstream from CAR before instantiating the emulators. Only set this to false if sourcing a bitstream from elsewhere (e.g. in a higher level umbrella chart) or manually overriding the bitstream (see above). .. admonition:: Local images and SSL bypass Depending on your circumstances, you may want to build the emulator images locally, and/or may need to deal with minikube SSL issues. To build the emulator images locally, run: .. code-block:: bash make oci-build-all If facing SSL issues, make sure all required images have been either built or pulled via Docker. Once all necessary images have been built or pulled, you can then run .. code-block:: bash minikube image load : to manually load the image into the minikube environment (this takes a little while). From there, ensure that the ``imagePullPolicy`` of all containers is set to ``Never`` (to force the local image to be used). You can view all the currently loaded images via: .. code-block:: bash minikube image ls Once the chart is installed, you can use a GUI like k9s, or run Kubernetes commands through minikube CLI: .. code-block:: bash minikube kubectl -- -n For example, to view the list of pods: .. code-block:: bash minikube kubectl -- get pods -n ska-mid-cbf-emulators Or to open an interactive shell inside a pod: .. code-block:: bash minikube kubectl -- exec -n ska-mid-cbf-emulators -i -t -- bash To stop running the emulator and terminate all pods, run: .. code-block:: bash make k8s-uninstall-chart .. admonition:: Windows GUI If using minikube through WSL2 on Windows, you can copy the kubeconfig file at ``~/.kube/config`` to a mounted drive to then use in a K8s GUI such as Lens. .. admonition:: Windows VPN If using a VPN on Windows, you may face network issues when pulling Docker images in WSL. Run the command .. code-block:: powershell Get-DnsClientServerAddress in a PowerShell window, then find your primary connection and copy each IP address listed there to a new line in ``/etc/resolv.conf``, in the same format as whatever is already in there. For example, for the result .. code-block:: text Ethernet 2 18 IPv4 {1.2.3.4, 10.11.12.13} you should add the lines .. code-block:: text nameserver 1.2.3.4 nameserver 10.11.12.13 to ``/etc/resolv.conf``. If you continue to get TLS handshake errors, simply keep trying the Docker command, as it tends to throttle pulls a lot. If you get certificate errors in Docker (not within minikube), you may simply have to close and reopen your WSL instance. Using Docker ============================= There are three methods of running a bitstream emulator via Docker: * Default/production: Runs the emulator and injector automatically and immediately. * Dev: Sets up the necessary utilities and environment for working on the emulators; does not start the emulator automatically. * Firmware Dev: Similar to Dev but with additional modifications to facilitate development from a firmware perspective. Mounts a FW workspace directly and provides extra command wrappers for simple interaction with the emulator. Environment variables are used to determine the bitstream emulator ID and bitstream path. Keep in mind that the Docker Compose setup only supports running a single bitstream emulator at a time. * ``LOGGING_MODE`` is the logging mode to use for the bitstream emulator. May be ``dev`` (output to console), ``prod`` (output only to logfile), or ``all`` (output to console & logfile). * ``EMULATOR_ID`` is the ID to use for the bitstream emulator. In the dev environment, this is set by default to ``default-emulator``. * ``BITSTREAM_ROOT_PATH`` is the first of two possible ways used to find the IP block emulators for this bitstream emulator. This path should point to the root of a bitstream tarball (i.e., the folder containing ``emulators``) such that the following path is valid: ``$BITSTREAM_ROOT_PATH/emulators//emulator/api.py``. In the dev environment, this is downloaded and set by default to ``/emulator_dev/ska-mid-cbf-bitstreams/raw/ska-mid-cbf-agilex-vcc/``. This variable must exist if not using ``EMULATORS_PATH``. * ``EMULATORS_PATH`` is the second of two possible ways used to find the IP block emulators for this bitstream emulator. This path should point to a folder such that the following path is valid: ``$EMULATORS_PATH//emulator/api.py``. Examples are the ``emulators`` folder in a bitstream package, or a firmware workspace folder. In the firmware dev environment, this is set by default to the mounted workspace folder ``/workspace``. This variable is optional and will take priority over ``BITSTREAM_ROOT_PATH`` if provided. * From within the container, the bitstream emulator's API is exposed at ``http://emulator:5001`` (preferred by Docker) or ``http://localhost:5001``. Running the Production Images ----------------------------- .. code-block:: bash make run # build and run the emulator/injector make enter # open a bash shell in the emulator image (remember that the emulator process is already running in the background) make view-logs # view the emulator image logs make remove # tear down the images Running the Standard Dev Environment ------------------------------------ Build and Start the Dev Environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This will automatically clone the bitstream into ``/emulator_dev/ska-mid-cbf-bitstreams/raw/ska-mid-cbf-agilex-vcc/``, and set the environment variable ``BITSTREAM_ROOT_PATH`` to this path. You may edit the bitstream files on the host machine, and they will be automatically synced to the emulator container's ``emulator_dev`` folder through a mount. .. code-block:: bash make run-dev # build and run the dev environment make run-dev-no-injector # alternatively, build and run the dev environment without a separate injector image make enter-dev # open a bash shell in the dev environment make view-logs-dev # view the dev environment image logs make remove-dev # tear down the images make remove-dev-no-injector Running the Emulator and/or Injector ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Before anything else, ensure you have the emulator repository mounted or cloned somewhere accessible within the dev environment. Run the emulator in the dev environment """"""""""""""""""""""""""""""""""""""" .. code-block:: bash cd /emulator_dev//images/ska-mid-cbf-emulators-emulator poetry install # only need to do this the first time, or if dependencies are updated poetry run python app.py [-v ] [-p ] [-c ] [-s ] Options: * ``-p INTERVAL, --pulse-interval=INTERVAL``: Interval, in seconds, between pulses. Must be at minimum 0.1 (lower values will be capped). Note that setting this value too low may result in unexpected behavior, especially on slower systems. Default is 1.0. * ``-c FILE, --config-file=FILE``: A custom IP block configuration file to override the ``config.json`` file included with the bitstream, if desired. * ``-s FILE, --signal-config-file=FILE``: A custom initial signal configuration file to override the ``initial_signal.json`` file in the emulator app folder, if desired. * ``-v VERBOSITY, --verbosity=VERBOSITY``: Logging verbosity of the emulator. See the `Helm Configuration`_ section for available verbosity options. Defaults to the ``EMULATOR_VERBOSITY`` variable if it exists, or ``3 (INFO)`` if it doesn't. Run the injector manually in the injector-less dev environment """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" .. code-block:: bash cd /emulator_dev//images/ska-mid-cbf-emulators-injector poetry install # only need to do this the first time, or if dependencies are updated poetry run python app.py [-v ] Options: * ``-v VERBOSITY, --verbosity=VERBOSITY``: Logging verbosity of the injector. See the `Helm Configuration`_ section for available verbosity options. Defaults to the ``EMULATOR_VERBOSITY`` variable if it exists, or ``3 (INFO)`` if it doesn't. Interact with the emulator and injector via the API """"""""""""""""""""""""""""""""""""""""""""""""""" .. code-block:: bash # GET the config.json (as a JSON string) that was used at instantiation of the controller curl -v http://emulator:5001/config | jq # GET a PNG of the graph of the emulator configuration curl -v http://emulator:5001/graph -o .png | jq # GET the state of the emulator controller curl -v http://emulator:5001/state | jq # POST start command to the ethernet_200g block curl -v -X POST http://emulator:5001/ethernet_200g/start | jq # GET the status of the ethernet_200g block curl -v http://emulator:5001/ethernet_200g/status | jq # POST configure command with data to the wideband_input_buffer block curl -v -d '{"expected_sample_rate": 1000, "expected_dish_band": 1}' http://emulator:5001/wideband_input_buffer/configure | jq # POST injector events from commandline (this will, in all likelihood, rarely if ever happen) curl -v -d '{"injector_event_groups": [{"bitstream_emulator_id": "default-emulator", "ip_block_emulator_id": "ethernet_200g", "events": [{"severity": "WARNING", "value": {"id": "inj0001", "injection_type": "update_link_badness", "message": "Degrade the link.", "badness": 0.75}}]}]}' -H 'Content-Type: application/json' http://localhost:5002/inject | jq # POST injector events from a JSON file curl -v -d '@path/to/events.json' -H 'Content-Type: application/json' http://injector:5002/inject | jq Running the Firmware Dev Environment ------------------------------------ Note: see :ref:`Emulator Development Using the FW-Dev Container` for a detailed guide on setting up this environment. Build and Start the Dev Environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This will mount the folder you specify in the ``WORKSPACE=`` option to ``/workspace`` in the dev environment, and will automatically clone the emulators into ``/ska-mid-cbf-emulators/``. .. code-block:: bash make run-fw-dev WORKSPACE= # build and run the FW dev environment make enter-fw-dev # open a bash shell in the FW dev environment make view-logs-fw-dev # view the dev environment image logs make remove-fw-dev # tear down the images Running the Emulator ^^^^^^^^^^^^^^^^^^^^ Run the emulator in the FW dev environment """""""""""""""""""""""""""""""""""""""""" .. code-block:: bash run-emulator [options] OR .. code-block:: bash cd /ska-mid-cbf-emulators/images/ska-mid-cbf-emulators-emulator poetry run python app.py [-v ] [-p ] [-c ] [-s ] See :ref:`Run the emulator in the dev environment` for the detailed options. Interact with the Emulator and Injector APIs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Common tasks can be invoked using the ``invoke`` command. The general syntax for commands is: .. code-block:: bash invoke For IP block API calls: .. code-block:: bash invoke ip. [--params=] for example .. code-block:: bash invoke ip.configure ethernet_200g '{"rx_loopback_enable": false}' invoke ip.status ethernet_200g --params='{"clear": false}' Available commands are: .. code-block:: bash invoke --list get Perform a GET request to the emulator controller post Perform a POST request to the emulator controller start-emulator Start the emulator server engine located in images/ska-mid-cbf- emulators-emulator. start-injector Start the injector located in images/ska-mid-cbf- emulators-injector. ip.configure ip.deconfigure ip.get Perform a GET request to an IP block ip.post Perform a POST request to an IP block ip.recover ip.start ip.status ip.stop controller.config Get the emulator configuration from this emulator bitstream. controller.initial-signals Get the initial signals used to instantiate this emulator. controller.graph Plot the IP block graph as a PNG image. controller.server-healthcheck Verify the API is accepting requests. Always returns 200 OK and an empty body. controller.start Start the emulator if it is stopped. Pulses and event listeners will reactivate. controller.state Get the current state of the emulator. controller.stop Stop the emulator if it is started. The API server will remain active but pulses and event listeners will be halted. controller.terminate Terminate the emulator. injector.inject Injects a JSON InjectorEvents request. Details of each command can be retrieved via the ``--help`` option, for example: .. code-block:: bash invoke --help get Usage: inv[oke] [--core-opts] get [--options] [other tasks here ...] Docstring: Perform a GET request to the emulator controller. Usage: invoke get [--params=] Options: -e STRING, --endpoint=STRING command endpoint -h STRING, --hostname=STRING hostname url of emulator -p STRING, --params=STRING query parameters