Developing the QA Display API

We use Poetry to manage the dependencies for this system. The poetry.lock file ensures a consistent set of dependencies for any build.

For local development we provide a docker-compose.yaml file for easy setup, and extra make commands for ease of use.

There are two main ways of developing: inside docker, or directly on the host.

Warning

The usage listed here does not connect to a live config DB. Instead this setup spins up a basic static Confg DB, with entries that represent a “Scanning” observation state, with all required entries pre-configured.

Tooling Pre-requisites

Below are some tools that will be required to work with the QA Data API:

Docker Development

To develop directly inside docker, run make build and then make run to start up all the services required.

For the API side of the development, changes will be picked up on file save, and the API will restart to pick up those changes, however the background tasks will not do this, and the docker container will need to be rebuilt and restarted for changes to take place.

The way to rebuild the cron image would be to run docker compose build cron and then docker compose up -d cron.

For this reason it is usually a lot simpler to develop directly on the host machine.

Running locally

If however you prefer to run the API locally, then at the minimum you will need the dependencies running in Docker. Refer to development section of the documentation for more details on how to set up the display on Docker.

Then to run the services, you can do the following:

make run-services dev-services-setup

And then use the following to bring up the API (this needs to be run within a Poetry shell):

make run-display-api

If you want everything running in Docker and using a single command, use:

make run

Production vs Development Mode

When running the docker version of the QA Data API, there are two modes that can be run. This is configurable in the docker-compose.yml file. The variable that needs to be changed is services.api.build.target. The two values are:

  • Production(prod):

    This is used when you are using the API when testing the other components of the QA Metrics workflow. Prod has no code reload. And has a configurable amount of workers, defaulted 2.

  • Development(dev):

    This is used when testing changes to the QA Data API. It can hot reload code but only has one worker.

Host Development

For developing directly on the host machine you will still need to have the services running. To accomplish this there is a helper command which can be used to start these services up which can be run with make run-services. (You should also run make dev-services-setup to pre-configure the services.)

Note

The Makefile has the variables REDIS_HOST, BROKER_INSTANCE, and NAMESPACE to make running the code locally much easier to accomplish.

There are also helper functions to run any command which is currently supported, for example:

  • make run-services -> brings up Kafka, Redis and etcd

  • make run-display-api -> Using above variables links to the services in Docker, but runs locally.

  • make run-task-* -> Runs the specified task using above variables.

Once the above services are running the API can be run with make run-display-api (it is assumed that you are running in a poetry shell, and have already run poetry install).

For running any of the cron commands you need to be in a poetry shell which you can do by running poetry shell. And then to install the new commands run poetry install and then you can run them directly.

For a list of tasks check pyproject.toml and look at the tool.poetry.scripts section.

A word of warning, you will need to set the Kafka and Redis env variables. To set them add BROKER_INSTANCE=localhost:29092 for Kafka and REDIS_HOST=localhost for Redis before any command that you need to run.

As an example to run the Visibility Receive task run the following:

BROKER_INSTANCE=localhost:29092 REDIS_HOST=localhost signal-task-process-visibility-receive --verbose

The standard arguments that can be added are:

  • --verbose or -v - to change the logging to output INFO

  • --debug or -d - to change the logging to output DEBUG