SKA Automated Network Deployment
SKA Automated Network Deployment is a project that implements an automated PnP solution based on software services to provision Cisco devices in the SKAO network with minimal human intervention. The solution enables Day 0 onboarding of new factory-default devices and the ability to seamlessly replace devices when required, both using the ZTP mechanism. Moreover, it also supports a Day N continuous provisioning capability, enabling real-time updates to active devices across the network.
The solution is mainly composed of 4 services running on a Linux-based host, as part of the same orchestrator system named ska-network-deploy.
DHCP server - isc-dhcpd
TFTP server - tftpd-hpa
Web/HTTP server - nginx + FastAPI Python apps
Cache server - redis
This document provides important instructions for setting up and configuring the host server and the ska-network-deploy services.
For a deeper technical overview please look at Onboarding new network devices - ZTP.
Note: the sections below were executed in a host Ubuntu 24.04.01 and tested against the following Cisco models:
Cisco switch C9500-32Cw/ IOS XE Software
For the Cisco C9200 family, which does not support python-based ZTP, please refer to Automation solution for upcoming Low AA2 rollout as a complement to these guidelines.
Build
Before jump into how to install and operate the ska-network-deploy on a host server, the user needs first to know how to create or access to either an official or a testing release artifact.
Start by cloning the GitLab repo ska-network-deploy-automation locally:
> git clone git@gitlab.com:ska-telescope/ser/ska-network-deploy-automation.git
To create an official release - major, minor or patch release - the user must follow the How to Make a Release guidelines.
This will create a Git tag on the main branch for the new version <major.minor.patch> and automatically triggers the Gitlab CI/CD pipeline, which builds a new artifact ska-network-deploy-<major.minor.patch>.tar.gz and uploads it to the SKAO Binary Artefacts Repository (BAR) under the repository raw-artefacts.
Optionally, the user can create an official release locally without running the CI/CD pipeline by executing the following make command.
> make raw-package-all
This will create the artifact ska-network-deploy-<major.minor.patch>.tar.gz in build/raw/ska-network-deploy folder.
To create a local testing release, bypassing the How to Make a Release, the user can execute the following make command.
> make test
This will create a tarball with the timestamp appended in the format <%Y%m%d%H%M>: ska-network-deploy-<%Y%m%d%H%M>.tar.gz.
The artifact tarballs contain all the necessary files to install and run the ska-network-deploy services.
Note: For more details on how to generate official release artifacts please look at User Manual
Requirements
The recommended network setup and resources - CPU, memory - for the ska-network-deploy host machine are based on the current configuration:
OS: Linux Ubuntu 24.04.01 LTS
CPU: min. 4 CPU cores Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
Memory: min. 4GB
The host MUST have connectivity to the Internet, as well as to the following SKAO platforms:
The network connectivity to the target devices for configuration MUST also be guaranteed.
The following services, libs and frameworks are required by the ska-network-deploy and are automatically installed on the host by the install.sh script:
isc-dhcp-server
nginx
tftpd-hpa
redis-server
python3.10 or python3.12
python3-pip
python3.10-venv or python3.12-venv
Installation
To install either a testing or a versioned release artifact from the the official raw-artefacts repository, the user must execute the following steps:
Copy the artifact tarball to the target host machine and extract it.
> mkdir ska-network-deploy-<major.minor.patch>
> tar xzvf ska-network-deploy-<major.minor.patch>.tar.gz -C ska-network-deploy-<major.minor.patch>/
Before run the install.sh script the user must properly set some environment-specific parameters. These are defined in the ztp-server.ini file, located in the same place as the install.sh.
ztp-server.ini
# Configuration parameters for ska-network-deploy services
# consumed by install.sh and services below
[DEFAULT]
server_ip=<host IP to bind the services (intf IP to devices)>
server_intf=<host interface to bind the services (intf name to devices)>
# log_level: critical, error, warning, info, debug
log_level=critical | error | warning | info | debug
[isc-dhcpd]
network=<dhcpd.conf network IP>
netmask=<dhcpd.conf netmask IP>
routers=<dhcpd.conf nexthop IP>
range_start=<dhcpd.conf start IP>
range_end=<dhcpd.conf end IP>
[nginx]
port=<nginx http listen port>
[fapi-webserver]
port=<fapi http listen port>
netbox_url=<Netbox URL>
netbox_token=<Netbox API token>
vault_url=<HashiCorp Vault URL>
vault_role=<role id for AppRole authentication>
vault_secret=<secret id for AppRole authentication>
vault_mount=<top-level location/URL path where the secrets are enabled>
[redis]
ip=<redis listen IP>
port=<redis listen port>
########################################
# Workaround for C9200 settings and data
# until we get these from Netbox
[C9200]
os_version=<os version to be installed>
Note: Some parameters in ztp-server.ini are already preconfigured for the most common scenarios and do not usually need to be tuned.
FastAPI Python apps require two internal TCP ports available on localhost:
bootstrapDeploy.py: [fapi-webserver]:port (e.g. 8000) - configured explicitlyconfigDeploy.py: [fapi-webserver]:port + 1 (e.g. 8001) - automatically assigned
The Redis cache server is configured by default to run on localhost:6379 but it can also be deployed on a different/remote host in a distributed architecture.
The user can now install all required services, dependencies and set the environment up on the host by executing the install.sh script.
> sudo bash install.sh
As part of the install.sh execution, a dedicated ska-ztp user and the following directory structure under /opt/ will be created on the host machine:
/opt/ska-network-deploy-<major.minor.patch>/
├── configs # served by nginx and tftpd-hpa (static)
| ├── c9200_bootstrap.cfg # minimal configuration for C9200 family ZTP bootup
│ └── bootstrap.cfg # minimal configuration for python based ZTP devices
├── images # served by Nginx (static)
| # place-holder for OS images and their md5 files
├── scripts # served by Nginx (static)
│ └── ztp.py # bootstrap python script
├── helpers # place-holder for misc scripts and tools
│ └── gen_device_conf.sh # script to generate the config files from Netbox for C9200 family
└── webserver
├── bootstrapDeploy.py # FastAPI webserver to handle the inital OS and bootstrap.cfg deployment
├── configDeploy.py # FastAPI webserver to handle the final/production config deployment
├── ZMQDeployEngine.py # Python file that implements a ØMQ pipeline
├── cache.py # Python file which defines a Cache class to the Redis server
├── utils.py # Python file containing methods and definitions common to all fastAPI apps
├── vaultClient.py # Python file which defines a VaultClient class to the HashiCorp Vault server
├── vaultSyncEngine.py # Python file that implements a Vault client and store/sync the secrets locally for the FastAPI apps.
├── .env.vault # local backup file for storing vault secrets in encrypted format
├── .env # settings for the apps - created by install.sh
└── .venv # python virtual env to run the python apps - created by install.sh
Note: the install.sh script must be executed in sudo mode.
The changes above made by the install.sh can be reverted and all the assets completely removed from the host by running the cleanup.sh script.
> sudo bash cleanup.sh
Note: For more details please look at User Manual
Usage
After the installation, the ska-network-deploy is ready to handle ztp requests from devices and orchestrate OS upgrades plus configuration deployments.
In the case the ska-network-deploy services need to be restarted or new changes in ztp-server.ini need to be applied, the user can simply re-run the install.sh script for the changes to take effect.
> sudo bash install.sh
The ska-network-deploy services can also be individually started, stopped and restarted through the native service manager systemd.
> sudo systemctl restart vault-sync-engine.service
> sudo systemctl restart isc-dhcp-server
> sudo systemctl restart tftpd-hpa.service
> sudo systemctl restart nginx.service
> sudo systemctl restart fapi-conf-deploy.service
> sudo systemctl restart zmq-deploy-engine.service
> sudo systemctl restart fapi-boots-deploy.service
> sudo systemctl restart redis-server.service
It is important to highlight that any new device must be pre-registered in the Netbox platform before they start triggering the ska-network-deploy services. This process is described in detail in the technical page.
Also, all OS image files must be placed in advance in /opt/ska-network-deploy-<major.minor.patch>/images.
The KV secrets, Netbox API token and Cisco local account, must also be pre-provisioned in the HashiCorp Vault under /low-aa/production/networks and /mid-aa/production/networks paths before start to use the services.
For better performance, it is highly recommend, but not mandatory, to create an associated <os_file_name>.md5 file for each the OS image, containing the md5 checksum and store it in the same folder /opt/ska-network-deploy-<major.minor.patch>/images.
Below there is an example of how to create the .md5 file for the Cisco OS image cat9k_iosxe.17.12.05.SPA.bin.
> echo $(md5sum cat9k_iosxe.17.12.05.SPA.bin | awk '{print $1}') > cat9k_iosxe.17.12.05.SPA.md5
> ls -la /opt/ska-network-deploy-<major.minor.patch>/images
total 4789144
drwxr-xr-x 2 ska-ztp ska-ztp 4096 Feb 3 23:32 .
drwxr-xr-x 7 ska-ztp zka-ztp 4096 Jan 26 21:39 ..
-rwxr-xr-x 1 ska-ztp zka-ztp 1310776238 Jan 22 23:48 cat9k_iosxe.17.12.03.SPA.bin
-rw-rw-r-- 1 ska-ztp zka-ztp 33 Feb 3 23:32 cat9k_iosxe.17.12.03.SPA.md5
-rwxr-xr-x 1 ska-ztp zka-ztp 1306917133 Jan 22 23:48 cat9k_iosxe.17.12.04.SPA.bin
-rw-rw-r-- 1 ska-ztp zka-ztp 33 Feb 3 23:32 cat9k_iosxe.17.12.04.SPA.md5
-rwxr-xr-x 1 ska-ztp zka-ztp 1312262395 Jan 22 23:48 cat9k_iosxe.17.12.05.SPA.bin
-rw-rw-r-- 1 ska-ztp zka-ztp 33 Feb 3 23:30 cat9k_iosxe.17.12.05.SPA.md5
-rwxr-xr-x 1 ska-ztp zka-ztp 470296483 Jan 22 23:48 cat9k_lite_iosxe.17.12.04.SPA.bin
-rwxr-xr-x 1 ska-ztp zka-ztp 503782075 Jan 22 23:48 cat9k_lite_iosxe.17.15.04.SPA.bin
Support
Let us know if you have issues. See our slack channel: #team-skanet
Documentation
The documentation, including the API spec, is available in SKAO dev portal
Roadmap
List of some features or enhancements for future releases:
Add Oxidized tool for config backup
Integration with the automated testing scripts and run them at the end
Run the
ska-network-deployservices in a docker containerAbility replace devices
Support real-time updates on active devices
Contributing
Source Code: gitlab.com/ska-telescope/ser/ska-network-deploy-automation