Debug an environment#

Use Coder to investigate an unhealthy Kubernetes workload, or BinderHub to run a notebook against a live deployment. Coder gives you cluster-aware development tools, while BinderHub builds a shareable environment from a Git repository.

What you’ll learn#

By the end of this tutorial, you’ll have:

  • Created a Coder workspace inside the STFC TechOps cluster

  • Found an unhealthy Pod and read its conditions, events, and logs

  • Tested a Service by its Kubernetes DNS name from inside the cluster

  • Queried a TANGO deployment with tango_admin

  • Attached a VS Code debugger to a Python process over debugpy

  • Launched the SDP notebooks on BinderHub and connected them to a running deployment

  • Diagnosed a TANGO client that cannot reach its database, and understood which cluster a BinderHub environment can reach

Time required: 45–90 minutes, depending on how many sections you follow. The two halves are independent.

Prerequisites:

Debug with Coder#

Use a Coder workspace inside the STFC TechOps cluster to find a failing Pod, inspect its events and logs, and test its Service from inside the cluster.

Prepare the Coder workspace#

You need access to SKAO Coder, the namespace that contains the workload, and the name or label of the application you want to investigate. Coder grants limited Kubernetes permissions. You can inspect resources, but the workspace can deny changes such as deleting a Pod.

  1. Sign in to Coder with the authentication option presented by the service.

  2. Open Workspaces, create a workspace, and select the Kubernetes template.

  3. Give the workspace a recognisable name and select the CPU, memory, and disk values for the session.

  4. Create the workspace and wait until its status is Running.

Coder form for creating a Kubernetes workspace with resource options.

Create a Kubernetes workspace.#

Open the browser terminal for a command-line session. You can also connect with the browser-based VS Code or JupyterLab applications, VS Code Desktop, or SSH when the workspace template offers them.

Coder workspace page showing browser and desktop connection options.

Choose a connection to the workspace.#

Install extra diagnostic tools when the workspace image does not already provide them. Installations outside your home directory do not persist when the workspace is rebuilt.

sudo apt update
sudo apt install -y jq netcat-openbsd dnsutils

Find the unhealthy Pod#

Set the namespace once so that the remaining commands target the same deployment. Replace each value in angle brackets.

export KUBE_NAMESPACE=<namespace>
kubectl get pods -n "$KUBE_NAMESPACE"

Look for a Pod that is not ready, is restarting, or has a status such as Pending, ImagePullBackOff, or CrashLoopBackOff. Save its name.

export POD_NAME=<pod-name>
kubectl describe pod "$POD_NAME" -n "$KUBE_NAMESPACE"
kubectl get events -n "$KUBE_NAMESPACE" --sort-by=.lastTimestamp

Read the Pod conditions and the events at the end of kubectl describe. These identify scheduling failures, failed health probes, missing configuration, and image pull failures.

Inspect the application logs#

Read recent logs from every container in the Pod.

kubectl logs "$POD_NAME" -n "$KUBE_NAMESPACE" --all-containers --tail=200

If a container restarted, inspect the terminated instance as well.

kubectl logs "$POD_NAME" -n "$KUBE_NAMESPACE" --all-containers --previous --tail=200

Compare the first error with the events. For example, a connection error in the application log combined with a healthy Pod schedule points towards Service, DNS, port, or dependency configuration rather than a scheduling problem.

Test the Service from the cluster#

List the Services and their ports.

kubectl get services -n "$KUBE_NAMESPACE"

Test the Service by using the Kubernetes DNS name. The short cluster suffix is <namespace>.svc.

nslookup <service>.${KUBE_NAMESPACE}.svc
nc -vz <service>.${KUBE_NAMESPACE}.svc <port>

A successful DNS lookup with a refused or timed-out connection narrows the problem to the Service port, selector, endpoints, or target application. Inspect the Service and its endpoints to continue.

kubectl describe service <service> -n "$KUBE_NAMESPACE"
kubectl get endpoints <service> -n "$KUBE_NAMESPACE"

Inspect a TANGO deployment#

For a TANGO workload, point the supplied tango_admin client at the database Service and query the running servers. Replace the Service and server names with values from the target namespace.

export TANGO_HOST=<database-service>.${KUBE_NAMESPACE}.svc:10000
tango_admin --ping-database
tango_admin --server-list
tango_admin --server-instance-list <server-name>

Install ITango when you need an interactive TANGO session.

python -m pip install itango
itango3
ITango session running from a Coder workspace.

Query TANGO from an ITango session.#

Stop the Coder workspace when you finish. Files in your home directory persist across workspace restarts; packages installed elsewhere do not necessarily persist.

Attach to a Python debug port#

Use browser-based VS Code in Coder to attach to a Python process running in another Coder workspace or an application Pod. The debugger and the target must use the same source files, and the target path must match the VS Code path mapping.

The following example uses a second Coder workspace as the target. For a shared deployment, arrange a debug build and permission to start the debugger with the owning team.

Create script.py in both workspaces. Put the target copy in /home/tango.

import time


def greet(name):
    print(f"Hello, {name}!")


def calculate_square(number):
    return number**2


def main():
    name = "Alice"
    while True:
        greet(name)
        result = calculate_square(5)
        print(f"The square of 5 is {result}")
        time.sleep(5)


if __name__ == "__main__":
    main()

Install debugpy in the target and bind its debug listener to port 5678. --wait-for-client pauses the application until VS Code attaches.

python -m pip install debugpy
cd /home/tango
python -m debugpy --listen 0.0.0.0:5678 --wait-for-client script.py

Warning

Binding to 0.0.0.0 exposes the debug port to networks that can reach the target. Use this only in a development environment, rely on Kubernetes network policy and RBAC, and never publish the port through an Ingress or public Service.

From the Coder workspace running VS Code, find the target Pod IP and verify that the port is reachable.

export TARGET_NAMESPACE=coder
export TARGET_POD=<target-pod-name>
export TARGET_IP=$(kubectl get pod "$TARGET_POD" -n "$TARGET_NAMESPACE" -o jsonpath='{.status.podIP}')
nc -vz "$TARGET_IP" 5678

Create .vscode/launch.json in the VS Code workspace. Replace <target-pod-ip> with the value of TARGET_IP.

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Attach to remote Python",
      "type": "debugpy",
      "request": "attach",
      "connect": {
        "host": "<target-pod-ip>",
        "port": 5678
      },
      "pathMappings": [
        {
          "localRoot": "${workspaceFolder}",
          "remoteRoot": "/home/tango"
        }
      ]
    }
  ]
}

localRoot identifies the source opened in VS Code. remoteRoot identifies the same source inside the target. Breakpoints do not bind when these paths do not correspond.

Start Attach to remote Python from the VS Code Run and Debug view. The waiting process starts and VS Code attaches to it.

VS Code in Coder attached to a remote Python process.

Attach VS Code to the remote Python process.#

Set a breakpoint inside greet. When execution pauses, inspect the stack and variables or change name from the debug console, then continue execution.

VS Code paused at a breakpoint in the remotely running Python process.

Inspect the target process at a breakpoint.#

Target terminal showing output changed during the remote debug session.

Observe the change in the target process.#

Stop the debug process and close port 5678 when you finish. A restarted Pod receives a new IP address, so repeat the address lookup before the next session.

Connect a notebook to SDP with BinderHub#

Use BinderHub to run the SDP notebooks against a live deployment, diagnose why a notebook cannot reach the TANGO database, and share the working environment.

ska-sdp-notebooks is a collection of notebooks for controlling and inspecting the Science Data Processor. Its notebooks talk to a running SDP deployment over TANGO, so they depend on where BinderHub runs as much as on their own code. That makes it a good way to learn what a BinderHub environment can and cannot reach.

Prepare#

You need an SDP deployment in a namespace on the data-processing platform, and the namespace name. You also need the DP VPN, as described in Configure your development environment.

Launch the notebooks#

The DP platform runs its own BinderHub, separate from the TechOps one. Use it, because it is the one that shares a cluster with your SDP deployment.

  1. Open BinderHub on the DP platform and sign in with your SKA credentials.

  2. Change the repository provider from GitHub to GitLab.

  3. Enter https://gitlab.com/ska-telescope/sdp/ska-sdp-notebooks.

  4. Leave the Git ref as HEAD, or enter a branch or tag.

  5. Select Launch.

BinderHub form with fields for a Git repository and Git reference.

Launch a repository from BinderHub.#

BinderHub resolves the reference and checks its image cache. When no image exists for that commit, repo2docker builds one and stores it for reuse. ska-sdp-notebooks ships a Dockerfile, so repo2docker builds from that rather than from a dependency file. JupyterLab then opens with the notebooks in the file browser.

JupyterLab launcher shown after BinderHub starts a notebook server.

Open the notebooks in the launched environment.#

Reproduce the failure#

Open ska-sdp-tango-tutorial.ipynb and run the first code cell as it ships. It sets the namespace to a placeholder:

KUBE_NAMESPACE = "<update-with-ns!!!>"
DATABASEDS_NAME = "databaseds-tango-base"
os.environ["TANGO_HOST"] = f"{DATABASEDS_NAME}.{KUBE_NAMESPACE}.svc.cluster.local:10000"

The cell succeeds, because setting an environment variable cannot fail. Run the next cell, which asks for a device proxy:

d = DeviceProxy('test-sdp/subarray/01')

This raises DevFailed, reporting that it cannot contact the TANGO database. The error names the connection, not the placeholder, so it is worth confirming where the client is actually pointed.

Diagnose it#

Open a terminal from the JupyterLab launcher and read back the value the notebook set:

python -c 'import os; print(os.environ.get("TANGO_HOST"))'

Resolve that hostname to separate a naming problem from a service problem:

import socket
socket.gethostbyname("databaseds-tango-base.<namespace>.svc.cluster.local")

A name that does not resolve means the namespace is wrong, the SDP deployment is not running, or the notebook server is not in the same cluster as the deployment. A name that resolves but refuses the connection points instead at the Service, its port, or the database Pod.

Correct the namespace#

Set KUBE_NAMESPACE to the namespace holding your SDP deployment, then restart the kernel. The TANGO client reads TANGO_HOST when it first connects, so editing the cell without restarting leaves the old value in the running process.

Run the cells again. DeviceProxy now returns a proxy, and d.state() reports the subarray state — OFF for a subarray that is not yet in use.

Understand the cluster boundary#

databaseds-tango-base.<namespace>.svc.cluster.local is an in-cluster DNS name. It resolves only from inside the cluster that runs the deployment, which is why this tutorial uses the DP platform’s BinderHub rather than the TechOps one. Launching the same repository from the wrong BinderHub produces exactly the same DevFailed, for a reason no amount of correcting the namespace will fix.

The server is not otherwise sandboxed. It reaches the internal network and the internet as normal, so pip install and an external API call both work. The restriction is specific to in-cluster DNS names, which belong to one cluster.

Network reach is not the same as cluster access. A BinderHub server starts without Kubernetes credentials, so it holds no identity the API server will accept. Your notebook can talk to a Service that is already running, but it cannot ask the cluster what is running or why a Pod is missing. Adding kubectl to the image would not change that — the commands would run and be refused.

That gap is the reason to move to the Coder workflow once the question turns from “can I reach this device” into “why is this device not there”. A Coder workspace carries a service account, so kubectl works there within the permissions it is granted.

Share the result#

BinderHub generates a URL that encodes the repository and Git reference, so another developer launching it gets the same environment:

https://sdhp.stfc.skao.int/binderhub/v2/gl/ska-telescope%2Fsdp%2Fska-sdp-notebooks/HEAD

HEAD moves as the repository changes. Use a commit SHA or a tag when someone must reproduce your exact environment.

Warning

Your edits stay in the running server. They are not committed to the repository, and a rebuilt image starts from the repository contents again. Copy anything you want to keep, and never save credentials into a notebook.

Stop or delete servers you no longer need from JupyterHub. Stopping a server retains its storage; deleting the server deletes that storage too.

Next steps#

Use Diagnose a Kubernetes workload with Headlamp when you only need a web view of live Kubernetes state. Follow the Jupyter notebook coding guidelines when preparing notebooks for wider use. Read Debugging tools reference for tool capabilities, configured limits, and service availability.

Was this page helpful?