SKA SDP TMLITE Server

Documentation Status

There is likely to be a wider implementation of a more capable telescope model - however for the purposes of quick SDP development this is fastAPI based server - which will deliver JSON formatted products - backed by a JSON formatted telescope model.

I have decided to isolate the server, the model and the model maintainer into three objects. The server will be this repository. The physical form of the model will be a JSON structure, The creation and maintenance of the JSON structure is provided by a third product.

Requirements

The system used for development needs to have Python 3 and pip installed.

Install

Always use a virtual environment. Pipenv is now Python’s officially recommended method, but we are not using it for installing requirements when building on the CI Pipeline. You are encouraged to use your preferred environment isolation (i.e. pip, conda or pipenv while developing locally.

For working with Pipenv, follow these steps at the project root:

First, ensure that ~/.local/bin is in your PATH with:

> echo $PATH

In case ~/.local/bin is not part of your PATH variable, under Linux add it with:

> export PATH=~/.local/bin:$PATH

or the equivalent in your particular OS.

Then proceed to install pipenv and the required environment packages:

> pip install pipenv # if you don't have pipenv already installed on your system
> pipenv install
> pipenv shell

You will now be inside a pipenv shell with your virtual environment ready.

Use exit to exit the pipenv environment.

Testing

Put tests into the tests folder
Use PyTest as the testing framework
- Reference: PyTest introduction
Run tests with python setup.py test
- Configure PyTest in setup.py and setup.cfg
Running the test creates the htmlcov folder
- Inside this folder a rundown of the issues found will be accessible using the index.html file
All the tests should pass before merging the code

Code analysis

Use Pylint as the code analysis framework
By default it uses the PEP8 style guide
Use the provided code-analysis.sh script in order to run the code analysis in the module and tests
Code analysis should be run by calling pylint ska_python_skeleton. All pertaining options reside under the .pylintrc file.
Code analysis should only raise document related warnings (i.e. #FIXME comments) before merging the code

Writing documentation

The documentation generator for this project is derived from SKA’s SKA Developer Portal repository
The documentation can be edited under ./docs/src
If you want to include only your README.md file, create a symbolic link inside the ./docs/src directory if the existing one does not work:

$ cd docs/src
$ ln -s ../../README.md README.md

In order to build the documentation for this specific project, execute the following under ./docs:

$ make html

The documentation can then be consulted by opening the file ./docs/build/html/index.html

Development

PyCharm

As this project uses a src folder structure, under Preferences > Project Structure, the src folder needs to be marked as “Sources”. That will allow the interpreter to be aware of the package from folders like tests that are outside of src. When adding Run/Debug configurations, make sure “Add content roots to PYTHONPATH” and “Add source roots to PYTHONPATH” are checked.

Todo

Insert todo’s here

TMLite Server

This server uses fastAPI to serve the contents of the prototype_model.json file via a web interface.

Starting the Server

I would suggest that new users check out the documentation for fastAPI as this will clearly demonstrate how this is set up.

The simplest way to launch the current server is via docker-compose. Running the following: docker-compose up --build

This will execute the following docker-compose which will start a container running the server and exposing port 80 of the container:

version: '2'
services:
    tmlite:
        build:
        context: .
        dockerfile: Dockerfile
        container_name: ska-sdp-tmlite-server
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
        hostname: localhost
        expose:
            - "80"
        ports:
            - "80:80/tcp"
        command: ["uvicorn", "ska.tmlite.main:app", "--host", "0.0.0.0", "--port", "80"]

Accessing The Server

One of the advantages of fastAPI is that it self documents. So once you have the server running simply connect to it:

> docker-compose up --build
.....
> ska-sdp-tmlite-server | INFO:     Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)

Assuming you kept the same port exposure open your browser at http://localhost:80/docs

For example the request to obtain the full model is:

>curl -X 'GET' 'http://localhost/model/current' -H 'accept: application/json'

And this will return the full model

The Model Structure

On server construction there are two models created a “default” model and a “current” model. The default model should not be altered but the current one can be changed in part. Also new models can be added and altered at will. WHen the server shuts down all alterations are lost. THis scheme is not for the long term storage of models - but the short term access of them.

Storage backends

The initial Telescope Model data is loaded from one of the supported storage backends. The storage to be used is selected by setting the STORAGE_BACKEND environment variable to the name of one of the supported storage backends.

Currently only a GitLab storage backend is supported,

`gitlab` backend

The gitlab storage backend loads a Telescope Model file from the SKA SDP TMLite data repository repository. Please read its documentation, as it explains how data is organised and presented. Note that like this project, the TMLite data repository is also currently designed to work in a read-only fashion from the standpoint of this TMLite server.

A set of environment variables control this process, all of which must be prefixed with STORAGE_BACKEND__:

CLONE_DIRECTORY is the local directory holding the clone of the repository, defaults to <OS-temp-dir>/tmplite_gitlab_repository.
PRIVATE_TOKEN, JOB_TOKEN and OAUTH_TOKEN, if defined, are used for authentication against GitLab.
MODEL_PATH is the file to be loaded from the repository clone as the Telescope Model, it defaults to prototype_model.json.
REPOSITORY_REF indicates the git reference (SHA, branch, tag) to retrieve when cloning the repository. If not given, the default repository branch is used.

If the CLONE_DIRECTORY doesn’t exist then a clone of the repository is created in that location. If it exists no further action occurs. Note that this implies that this backend can be used to point to an existing directory/file in the local filesystem containing a valid Telescope Model, even if that directory is not a git repository.

At the moment, and as mentioned earlier, all editions are ephemeral: once the server shuts down they are all lost.

Example API tasks

Probably the simplest way to access this is via the python requests module - but of course for the GET methods you can even use a browser if you want.

Getting the full default model

A simple python api:

> import request
> url = "http://localhost/model/default/"
> response = requests.get(url)

Getting the current model

Is as simple as changing the URL:

> url = "http://localhost/model/current/"

But there are a number of methods coded up for you to get subsets of the model and even change things for example:

> url = "http://localhost/current/ska1_low/update_antennas"
> mccs = {"station_ids": ['0','1','2','3']}
> response = requests.post(url,json=mccs)

This will update the current model to only include antennas from the list.

Extending the Model and Server

This is a minimal hello world implementation - You should have enough information to extend the server to respond more smartly to requests and return slices through the model for example.

Public API

The FastAPI is documented internally at http://localhost:80/docs once the server is running. The functions and classes currently implemented in the model are:

Models

Server functions

Client

TMLite Documentation

These are all the packages, functions and scripts that form part of the project.

TMLite Server