Troubleshooting

When things go wrong, its best to start by checking on the status of the pods and investigating the logs of the various processes that operate within PST. The following kubectl commands assume that either PST is running in the default namespace or the namespace has been configured with

kubectl config set-context --current --namespace=$KUBE_NAMESPACE

Pod status

To display the status of the pods

# Display a quick summary of all pods
kubectl get pods

# Display detail of the pod running the PST core applications
kubectl describe pods test-ska-pst-core

# See if any applications have exited unexpectedly
kubectl describe pods test-ska-pst-core | grep Exit -A 5 -B 5

To run a bash shell on a specific container

# Run bash on the PST DSP container
kubectl exec -ti test-ska-pst-core -c dsp -- bash

# Run bash on the PST JupyterLab container
kubectl exec -ti test-ska-pst-jupyterlab -- bash

Container logs

To display the logs of all containers, application labels can be used to simplify the k8s command.

# Display logs of all containers in all pods that has the label ska-pst-core
kubectl logs -l app=ska-pst-core

# Display logs of all containers in all pods that has the label ska-pst-lmc
kubectl logs -l app=ska-pst-lmc

To display the logs of specific containers within a pod:

# Display logs of PST RECV
kubectl logs test-ska-pst-core -c recv

# Display logs of PST SMRB
kubectl logs test-ska-pst-core -c smrb

# Display logs of PST DSP
kubectl logs test-ska-pst-core -c dsp

# Display logs of PST SEND
kubectl logs test-ska-pst-core -c send

# Display logs of PST STAT
kubectl logs test-ska-pst-core -c stat

Override files

To confirm compatibility of the values.yaml used for overriding the defaults, the following command can be used to confirm if the templates are rendered as expected.

helm template skao/ska-pst --values=values.yaml