Scripts

The following sections document the scripts available in the ska-low repository.

create_coredump_dockerfile.py

The devices we deploy use PyTango which interfaces with CppTango which is written in C++. Certain things in C++ (for example attempting to read from an invalid memory address) will cause the operating system to kill the process and dump the raw memory to file (a core dump). In order to extract useful debugging information from this core dump we need to know what memory corresponds to what functions which is provided by debug symbols. Every version of Python (down to the build date) puts functions in different places so we need the exact same version of Python (and the libraries python depends on) in order to get debugging information from the core dump. The easiest way to do this in our case is by using a Docker container based on the image of the pod that crashed and then install additional debugging tools into the container. The create_coredump_dockerfile.py script makes it easy to do this process.

If you have a core dump zip file from the SKAO Core Dump Catcher, the process for running the script is:

python3 create_coredump_dockerfile.py /path/to/unzipped/coredump/directory

docker build --tag IMAGE-NAME . && docker run -it --entrypoint=bash IMAGE-NAME

The script will create a Dockerfile and a symlink to the core dump in the current directory. It will also print the will the second line listed above with IMAGE-NAME filled in. If you only have a core dump, run:

python3 create_coredump_dockerfile.py coredump.core artefact.skao.int/IMAGE:VERSION

docker build --tag IMAGE-NAME . && docker run -it --entrypoint=bash IMAGE-NAME

Once inside the container, run:

gdb python /core

This will open gdb and begin downloading debugging symbols from debuginfod.ubuntu.com. The python script can also be run like a script, e.g. ./create_coredump_dockerfile.py.

run_notebooks.py

For information on the run_notebooks script, please see the SFT notebooks section.

todo_tracker.py

The todo_tracker.py module provides a script that checks for references to Jira issues in the Python and notebook code of a specified GitLab project and reports on the status of those issues.

Rationale

When a Jira issue is called out in code, it is assumed that the Jira issue explains why the code is the way it is; that is, the code works around a bug or problem or missing functionality that is described by that Jira issue. If the Jira issue is marked as “Done”, that implies that the code needs to be updated to remove the bug workaround or make use of newly available functionality.

Usage

The script requires:

• A GitLab API token to be stored in the environment variable GITLAB_API_TOKEN.

• A Jira API token to be stored in the environment variable JIRA_API_TOKEN.

The devcontainer provided by this repo is configured to load environment variables from .devcontainer/secrets.env. If using this devcontainer, you just need to create this file with contents like:

GITLAB_API_TOKEN=glpat-pVgLqu8aH35zVYu9eoSU
JIRA_API_TOKEN=uiWpAC2nJjAWQggoNDhUDo5vVRP7ar5qT2clC5

The secrets.env file is gitignored so you won’t accidentally check it in.

For local usage outside the devcontainer (e.g., when running via the Makefile target make todo-tracker), environment variables are loaded and exported from a .env file in the root of the repository. You can create this .env file with the same token format shown above.

The script has a registered entrypoint in the package. It produces a YAML report of tracked issues.

To print the YAML report directly to the standard output:

todo_tracker

To write the YAML report to a specific file, provide the --report-path option:

todo_tracker --report-path my_report.yaml

Progress logs and connection status are written to stderr to ensure that the stdout stream or report files contain only pure YAML.

Future work

The todo_tracker.py script currently acts upon code in a GitLab project. Now that it resides in the same repo as the target code, it makes sense to refactor it to act on local code.

TODO LOW-2341: Document the other scripts.