SKA Archiver

This project provides archiving solution for SKA. The archiver solution is based on HDB++ Archiver provided by Tango community. The repository contains set of Dockerfiles, helm charts and scripts that are useful for building, deploying and configuring the HDB++ archiver.

Build

To build docker image of HDB++ navigate to docker directory from terminal and enter command:

    make oci-image-build

Testing

    make k8s-test

Deployment

To deploy the archiver enter following command:

    make k8s-install-chart ARCHIVER_DBNAME=<dbname> ARCHIVER_TIMESCALE_HOST_NAME=<hostname> ARCHIVER_TIMESCALE_PORT=<port> ARCHIVER_TIMESCALE_DB_USER=<dbuser> ARCHIVER_TIMESCALE_DB_PWD=<dbpassword>

The archiver gets deployed in namespace ska-tango-archiver

On gitlab ci-cd Masked Environment variables are created ARCHIVER_TIMESCALE_DB_PWD ARCHIVER_TIMESCALE_DB_USER ARCHIVER_TIMESCALE_HOST_NAME ARCHIVER_TIMESCALE_PORT which contains archiver database connection details.

To deploy the archiver using local archiver DB details, enter following command:

    make k8s-install-chart ARCHIVER_DBNAME=<dbname> ARCHIVER_TIMESCALE_HOST_NAME=<hostname> ARCHIVER_TIMESCALE_PORT=<port> ARCHIVER_TIMESCALE_DB_USER=<dbuser> ARCHIVER_TIMESCALE_DB_PWD=<dbpassword>

To delete the deployment enter following command:

    make k8s-uninstall-chart

  • Local Deployment and testing:

    1.Connect to STFC VPN.

    2.Update database details present in data/configuration.json file. currently host ip for mid :10.200.10.143 or timescaledb.ska-eda-mid-db.svc.cluster.local currently host ip for low : 10.200.12.14 or timescaledb.ska-eda-low-db.svc.cluster.local NOTE:if not able to connect please check k8s-test in pipeline for ip address, might be due to redeployment of timescaledb.

    3.Deploy using folowing command, pass db details if different then the default one.

    make k8s-install-chart ARCHIVER_DBNAME=<dbname> ARCHIVER_TIMESCALE_HOST_NAME=10.200.10.143/<hostname> ARCHIVER_TIMESCALE_PORT=<port> ARCHIVER_TIMESCALE_DB_USER=<dbuser> ARCHIVER_TIMESCALE_DB_PWD=<dbpassword>
for low:
make k8s-install-chart TELESCOPE=ska-low ARCHIVER_DBNAME=<dbname> ARCHIVER_TIMESCALE_HOST_NAME_LOW=10.200.12.14/timescaledb.ska-eda-low-db.svc.cluster.local ARCHIVER_TIMESCALE_PORT_LOW=<port> ARCHIVER_TIMESCALE_DB_USER_LOW=<dbuser> ARCHIVER_TIMESCALE_DB_PWD_LOW=<dbpassword>

    4.To test the deployment:

    make k8s-test

Useful commands

Command to wait till archiver deployment is complete:

    make k8s-wait

Command to delete the namespace:

    make k8s-delete-namespace

Access the TimescaleDB instance in STFC

  • To access the timescaleDB instance in STFC, please check latest k8s-test job(line 57) to get the latest external IP and port.

  • The external IP for mid : 10.200.10.143 or timescaledb.ska-eda-mid-db external IP for low : 10.200.12.14 or timescaledb.ska-eda-low-db and port : 5432 are for current deployment. If it is redeployed, make sure to update it with latest external IP.

  • Install postgres client on user machine :Follow the steps in the given link for installation.

  • Make sure the user’s machine is connected to SFTC vpn.

  • Enter below command to access the TimescaleDB.

    psql -U admin -h <external-ip> -p <port> -d <db-name>

  • Where default db-name is postgres and branch specific db-name can be ska_archiver_<branch_name> ,for eg : ska_archiver_hm_72.

  • Enter the password to access the DB.

Configuring new attributes using Yaml2archiving

  • Steps to Configure archiver.

      1. Clone ska-tango-archiver repository : https://gitlab.com/ska-telescope/ska-tango-archiver.git

      1. cd ska-tango-archiver

      1. Download KUBECONFIG file.

      1. Run following make target with suitable variables.

            make configure_archiver KUBECONFIG=<kubeconfig file> ATTR_CONF=<yaml file>
    KUBE_NAMESPACE=<namespace where archiver is deployed>

    • NOTE: ATTR_CONF requires yaml file as specified in yaml2archiving documentation https://gitlab.com/tango-controls/hdbpp/yaml2archiving.

    • NOTE: Manager and archiver name in configuration file would be:

              manager: mid-eda/cm/01
        archiver: mid-eda/es/01

      for reference check attribute_config.yaml file.

Data Extraction

  • libhdbpp-python CLI:

Pre-requisite: 1. PyYAML 2. psycopg2

  1. INSTALLATION Steps:

    1. clone https://gitlab.com/tango-controls/hdbpp/libhdbpp-python

    2. change directory to libhdbpp-python and run following command

    pip install .

  • OR

    1. Run make target from ska-tango-archiver repo make install-libhdbpp-python

  1. Usage Manual

    hdbpp-reader [-h] [-v] [-D] [--syslog] [-b BACKEND] [-P] [-c CONFIG] [-C CONNECT] [-d DATABASE] [-H HOST] [-u USER] [-p PASSWORD]
                    [-t TIMEFORMAT]
                    {list,read} ...

for timescaledb use --backend argument as follows:
--backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader

    1. connection details can be provided using connection string ,config file or argument:

    host: <timescaledb_host_name>
database: <database name>
user: <database user>
password :<database password>
port: <port>

    1. List Function: to list attributes: -P –pattern = this option can be used to search pattern

    using config file for connection:
hdbpp-reader --backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader -c config.yaml list -P "*"

using connection details as arguments:

hdbpp-reader --backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader  list -P "*"
host (default localhost):10.200.10.143 or timescaledb.ska-eda-mid-db.svc.cluster.local <for mid>
                                  or
host (default localhost): 10.200.12.14 or timescaledb.ska-eda-low-db.svc.cluster.local <for low>
port (default 3306):5432
database (default hdb):ska_archiver_hm_71
user:admin
password:admin
output:
2022-11-07 17:17:21 hdbpp_reader[685660]: Loaded config file: config.yaml
tango://tango-host-databaseds-from-makefile-test.ci-ska-tango-archiver-965447b4.svc.cluster.local:10000/sys/tg_test/1/double_scalar


Listing attributes double_scalar:
hdbpp-reader --backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader -c config.yaml list -P "*/double_scalar"

Listing attributes of test device:
hdbpp-reader --backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader -c config.yaml list -P "*/sys/tg_test/1/*"

    1. READ Function: to list attributes: -P –pattern = this option can be used to search pattern

    using config file for connection:
hdbpp-reader --backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader -c config.yaml read <attribute_full_name>

for example:
hdbpp-reader --backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader -c config.yaml read tango://tango-host-databaseds-from-makefile-test.ci-ska-tango-archiver-e6c90079.svc.cluster.local:10000/sys/tg_test/1/double_scalar
output:
2022-11-07 17:36:00 hdbpp_reader[705016]: Loaded config file: config.yaml
(datetime.datetime(2022, 11, 7, 11, 29, 1, 489121, tzinfo=datetime.timezone.utc), 87.55708576347412, 0.0, 0, None)


using connection details as arguments:

hdbpp-reader --backend=pyhdbpp.timescaledb.timescaledb.TimescaleDbReader  read <attribute_full_name>
host (default localhost):10.200.10.143 or timescaledb.ska-eda-mid-db.svc.cluster.local <for mid>
                                  or
host (default localhost): 10.200.12.14 or timescaledb.ska-eda-low-db.svc.cluster.local <for low>
port (default 3306):5432
database (default hdb):ska_archiver_hm_71
user:admin
password:admin

  2. Using Python Console:

    For using Python Console please refer Timescale DB Reader Jupyter Notebook (timescaledb_reader.ipynb)