Set up your development environment

This page is part of the Getting started documentation for the SKA Low MCCS subsystem.

Background

Like many SKA Software projects, the SKA Low MCCS subsystem codebase is built on some heavy dependencies, particularly Tango Controls. So in order to run and test SKA-Low-MCCS, you will need a development environment in which those dependencies are installed. This can be a complicated process.

The obvious solution - to install the dependencies directly into your local system - is not supported because:

  • SKA Software developers use a range of local system types, and we don’t want to have to support them all. It is preferable to provide all developers with a consistent environment.

  • Many SKA Software developers also work on other projects, and it can be problematic maintaining a single environment to support multiple projects, each with their own set of dependencies. It is better to provide developers with an isolated environment just for SKA Software development.

For these reasons, the supported approaches are based on virtualisation. The approach described on the SKA software developer portal is a legacy solution that is not recommended for MCCS development. Instead, we recommend a container development approach.

The advantages of this approach are:

  • SKA Software already has container images that contain all of the dependencies that you need to develop. These images are kept up to date, so you won’t need to worry about maintaining your system.

  • The SKA container images are the ‘canonical’ SKA Software development environment: if your code runs on the SKA container image, your code runs.

  • The only local system requirements for developing SKA-Low-MCCS in a container are: a POSIX shell environment, GNU make, Git and Docker. Other than that, you are free to work on any operating system, to use your favourite IDE, and generally to set up your local system as you please.

Basic development setup

The basic setup described here will allow you to edit code and documentation locally, and to launch basic testing, linting and documentation builds. For occasional dabblers in the MCCS code, this is the only setup required. We recommend for that new developers develop inside a docker container as it makes things a lot easier when working with multiple repos and codebases, however it should be noted that this is a choice for devs to make, should they want they’re free to develop on the bare metal of their machine and set up the environment variables and dependencies themselves.

Should you wish to go via the recommended route and install docker, please follow the instructions in the “Docker” section below. To proceed with the basic installation you may use install_script.sh or install_ansible.yml files as follows.

NOTE: These scripts should only be used with debian/ubuntu flavoured linux machines as they have not been tested with others and may not work

If you already have an sql server installed then provide the password for it, otherwise choose a password that you would like for the server that will be installed.

To use the shell script call

./install_script.sh <your sql password>

Because some tasks in the ansible playbook require sudo privileges you must call the ansible playbook with the –ask-become-pass, like this

ansible-playbook install_ansible.yml -e "SQL_PASSWORD=<your sql password>" --ask-become-pass

To test the script has successfully installed tango, first Set the TANGO_HOST variable .. code-block:: shell-session

export TANGO_HOST=localhost:10000

Then start the device test server .. code-block:: shell-session

/usr/local/tango/bin/TangoTest test &

Test jive .. code-block:: shell-session

/usr/local/tango/bin/TangoTest test &

For more serious developers, further steps are described in subsequent sections.

The basic steps are

  1. Install Docker;

  2. Install and setup Git, and clone the ska-low-mccs repository. (For Windows users, installing Git will also provide you with a POSIX shell.)

  3. Install make.

Details on these steps are provided below.

Docker

Installing Docker should be fairly simple on any operating system.

Windows-specific instructions

On Windows, it is highly recommended to set up WSL2 first, as Docker runs faster when integrated with WSL2. To set up WSL2, follow the instructions at https://docs.microsoft.com/en-us/windows/wsl/install-win10.

The Docker package to be installed on Windows is “Docker Desktop for Windows”. Before installing, ensure that any older Docker versions (such as Docker Toolbox) are fully uninstalled, and that all Docker environment variables have been deleted. Installing is very straight-forward: simply download and run the installer.

Warning

By default, Docker Desktop enables the Hyper-V feature of Windows. Some people find that this interferes in the operation of other hypervisors such as VirtualBox.

If using WLS2, open the Docker Desktop for Windows dashboard, go into Settings | Resources | WSL Integration, and ensure Docker is integrated with your chosen WSL2 distro.

Ubuntu-specific instructions

These instructions assume Ubuntu 20.04 LTE, but may be relevant to other versions / Linux variants.

  1. Install Docker CE. Unfortunately you can’t just sudo apt install docker because that would install a Canonical build of Docker named Docker.io, and this is not recommended. We’ll need to work a little harder to install Docker CE. We can use apt but first we need to add the Docker apt repository, and in order to do that we will need to install the Docker repository public key, and these steps will themselves require installation of packages:

    me@local:~$ sudo apt install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
me@local:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
me@local:~$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
me@local:~$ sudo apt-get update
me@local:~$ sudo apt install docker-ce docker-ce-cli

  2. Test your install:

    me@local:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:6a65f928fb91fcfbc963f7aa6d57c8eeb426ad9a20c7ee045538ef34847f44f1
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

  3. At this point you can only run this command as sudo, because you are not a member of the docker group. The docker group is created but it is empty. Add yourself to the docker group:

    me@local:~$ sudo usermod -aG docker $USER

  4. Log out and log back in. Then verify that you can run docker without sudo:

    me@local:~$ docker run hello-world

Great! You are ready to run a SKA Docker container.

Git

  1. Install git. This should be simple on any operating system.

  2. Set up git:

    me@local:~$ git config --global user.name "Your Name"
me@local:~$ git config --global user.email "youremail@domain.com"

  3. At some point you will need to set up git commit signing too. Now is as good a time as any. Follow the instructions at the SKA Working with Git page.

  4. Clone the SKA-Low-MCCS repository:

    me@local:~$ git clone --recurse-submodules https://gitlab.com/ska-telescope/ska-low-mccs.git

POSIX shell

The ska-low-mccs makefiles assume a POSIX shell environment. Thus, in order to run them, you will need a POSIX shell and the make executable.

If you are running on a Linux variant (including MacOS), then your terminal already provides a POSIX shell; for example, bash.

On Microsoft Windows, neither the Command Prompt nor Powershell are POSIX shells. There are various options for installing POSIX shells on Windows. These include WSL2, Cygwin and MinGW. Here we take the easiest option: your installation of Git comes with git-bash, which provides a POSIX shell (actually a copy of MinGW). Let’s use that.

Note

If following these instructions on Windows, remember that whenever instructed to run a command in a local terminal, you must run it in your POSIX shell e.g. git-bash.

Make

Linux instructions

On Linux, you can install Make via your package management system. For example, on Ubuntu:

me@local:~$ sudo apt install build-essential

will install a number of tools common to building tool-chains, including Make.

Windows instructions

On Windows (assuming git-bash), you’ll need to download a make executable and put it where git-bash will find it:

  1. Go to https://sourceforge.net/projects/ezwinports/files/

  2. Download the zipfile for make (without the dependency on guile); for example, make-4.3-without-guile-w32-bin.zip.

  3. Extract the zipfile.

  4. Copy the contents to your Git\mingw64\ folder. Merge the folders, but do not overwrite/replace any existing files.

Basic development tools

You now have a basic development setup. The following Make targets are available to you:

  • poetry run make python-test - run the tests in a SKA docker container

  • poetry run make python-lint - run linting in a SKA docker container

  • poetry run make python-format - run formatting in a SKA docker container

Try it out:

me@local:~$ cd ska-low-mccs
me@local:~/ska-low-mccs$ docker run --rm -it -v `pwd`:/app artefact.skao.int/ska-tango-images-pytango-builder:9.4.3 bash
root@703cb23c4406:/app# poetry config virtualenvs.create false
root@703cb23c4406:/app# poetry install --no-root (It will take some time to complete the process)
root@703cb23c4406:/app# poetry run make python-test
Skipping virtualenv creation, as specified in config file.
     Makefile:46: warning: overriding recipe for target 'python-do-build'
     .make/python.mk:115: warning: ignoring old recipe for target 'python-do-build'
     Makefile:49: warning: overriding recipe for target 'python-do-publish'
     .make/python.mk:165: warning: ignoring old recipe for target 'python-do-publish'
     Skipping virtualenv creation, as specified in config file.
     pytest 6.2.5
     PYTHONPATH=./src:/app/src poetry run pytest  \
      --cov=src --cov-report=term-missing --cov-report xml:build/reports/code-coverage.xml --junitxml=build/reports/unit-tests.xml tests/
     Skipping virtualenv creation, as specified in config file.
     PyTango 9.3.3 (9, 3, 3)
     PyTango compiled with:
       Python : 3.7.3
       Numpy  : 1.19.2
       Tango  : 9.3.4
       Boost  : 1.67.0

       PyTango runtime is:
       Python : 3.7.3
       Numpy  : 1.19.2
       Tango  : 9.3.4

     PyTango running on:
     uname_result(system='Linux', node='278b54bd07b5', release='5.4.0-107-generic', version='#121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022', machine='x86_64', processor='')
                                                                     ============ test session starts ========
                                                                     ------------ JSON report ----------------
                                                                     report saved to: build/reports/report.json
———– coverage: platform linux, python 3.7.3-final-0 ———–

38 files skipped due to complete coverage. Coverage HTML written to dir build/htmlcov Coverage XML written to file build/reports/code-coverage.xml

================================================================== 1403 passed, 125 skipped, 1 xfailed, 8 warnings in 1347.75s (0:22:27) ==================================================================

(The first time you run these commands, they may take a very long time. This is because the Docker image has to be downloaded. Once downloaded, the image is cached, so the command will run much faster in future.)

Advanced development setup

The approach described above provides a few basic tools, but serious developers will want more than this. For example, make python-test runs all the tests, but serious developers will want fine-grained control of what tests to run.

To run tests in a specific file or directory change the PYTHON_TEST_FILE variable in the Makefile. This can also be done from the command line, for example: make PYTHON_TEST_FILE=tests/unit/tile python-test will run all tests found in the tile directory.

Since the repository is read-write mounted in the container, it is possible to edit the code from inside the container. However this is not recommended: Docker containers are deliberately lightweight and streamlined, containing nothing that isn’t needed for them to do their job. This Docker container was built to run SKA Software python code, not for you to edit code in. It doesn’t even contain vi. You could install what you need, but it makes more sense to edit the code on your local system, where you have your favourite IDE, and everything else you need, set up just the way you like it. Then, after saving your changes, switch over to the container terminal session to run the tests.

IDE integration

The workflow described above - editing locally but deploying to a remote container for testing - is well supported by IDEs. The SKA-Low-MCCS repository is already set up for remote container development in Visual Studio Code (“vscode”). It is highly recommended that you use vscode to develop. To set up vscode, follow the instructions at Set up Visual Studio Code.