Policy agent#

Every object you send to an SKAO Kubernetes cluster passes through a policy agent before it is stored. The policy agent can reject the object, silently modify it, or create other objects alongside it. This page explains what that means for your deployments and how to react when a policy gets in your way.

The System Team uses Kyverno as its policy agent. Kyverno applies three kinds of behaviour:

Validation — your object is accepted or rejected.
Mutation — your object is modified on the way in, so what runs is not exactly what you applied.
Generation — applying one object causes others to be created for you.

Policies apply cluster-wide. You cannot opt a namespace out of them, and you do not need to enable anything to get them.

Rejected requests#

These policies block the request outright. You see the failure immediately, as an error from kubectl, Helm, or your CI job.

If you try to	Clusters	What happens
Create an Ingress with a `host` field	stfc-techops, stfc-dp, aws-cicd	Rejected. Ingress hostnames are assigned centrally — leave `spec.rules[].host` unset.
Run an image from `bitnami/` or `bitnamilegacy/`	stfc-techops, stfc-dp, aws-cicd	Rejected. Bitnami images are not permitted; use an equivalent from `artefact.skao.int` or another upstream.
Create a `VaultAuth` object in a team namespace	stfc-techops, stfc-dp, aws-cicd	Rejected. Raise an STS ticket if you need special Vault access.
Create a shared-volume PVC pointing at a source that does not exist	stfc-techops, stfc-dp, aws-cicd	Rejected. The `skao.int/clone-pvc` label must name a PVC that exists in the namespace given by `skao.int/clone-pvc-namespace`.
Delete the first volume of a shared volume group	stfc-techops, stfc-dp, aws-cicd	Rejected while other replicated volumes still reference it. Delete those first — the error lists them.
Request a storage class that does not support shared storage	stfc-techops	Rejected for shared PVCs. Use `ceph-cephfs`.

Automatic modifications#

These policies modify your objects as they are admitted. Nothing fails, but kubectl get returns something different from what you applied — which is worth knowing before you go looking for a bug.

What you apply	Clusters	What changes
Any workload in a CI, dev, integration, staging, or taranta namespace	stfc-techops, stfc-dp, aws-cicd	CI/CD labels and annotations are copied down from the namespace onto Deployments, StatefulSets, Jobs, Pods, DeviceServers, and DatabaseDS resources, so the artefacts can be traced back to your pipeline.
A new CI-style namespace	stfc-techops, stfc-dp, aws-cicd	It is labelled as manageable by the GitLab runner, and on stfc-techops it is annotated with author details resolved from your GitLab handle.
A PVC requesting `local-storage` or `standard`	stfc-techops	The storage class is rewritten to `nfss1`. Those two classes do not exist on the cluster.
A shared PVC	stfc-techops, stfc-dp	The backing CephFS PersistentVolume is provisioned for you.
Any pod in a CI namespace	stfc-techops	A preferred affinity is added so pods from the same namespace tend to land on the same node.
A pod annotated `cicd.skao.int/inject-pvc`	stfc-techops	The named PVC is mounted read-only at `/mnt/artefact` in every container.
A workload in a namespace belonging to a team with reserved nodes	stfc-techops	A reserved node selector and a matching toleration are added, pinning the workload to that team’s nodes.

Generated resources#

When you	Clusters	What appears
Create a CI, dev, integration, staging, or taranta namespace	stfc-techops, stfc-dp	The standard RoleBindings, so your team can work in it without requesting access.
Create a pod annotated `cicd.skao.int/inject-artefact`	stfc-techops	A `Bar` resource that fetches the artefact into a PVC for the pod to mount.

Released shared volumes are also reclaimed automatically in the background, so you do not need to clean them up by hand.

Audit-only warnings#

Some policies run in audit mode. They record a result in the policy report but let the request through.

If you	Clusters	What is recorded
Create a `SecretProviderClass`, or a pod annotated `vault.hashicorp.com/agent-inject: "true"`	stfc-techops, stfc-dp	A deprecation warning. The Vault CSI driver and the Vault agent injector are on their way out — migrate when you can, following the secret management migration guide.
Run an image from `artefact.skao.int` or `harbor.skao.int/production`	Not currently enabled	Whether the image carries a valid SKAO signature. An unsigned image is recorded but still admitted. See OCI Images Signature Management for how images get signed.

Troubleshooting policy failures#

A rejection names the policy that caused it. The message is written to be actionable, so read it before anything else:

$ kubectl apply -f ingress.yaml
Error from server: error when creating "ingress.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Ingress/default/hello-world-ingress was blocked due to the following policies

ingress-host-not-allowed:
    ingress-host-not-allowed: 'validation error: Failed to create Ingress resource:
        defining `host` is not allowed. rule ingress-host-not-allowed failed at path /spec/rules/0/host/'

The first name is the policy, the second is the rule inside it, and the path points at the offending field.

Audit-mode results never reach kubectl. To see them, read the policy report for your namespace:

kubectl get policyreport -n <your-namespace>

To see what a policy actually did to a resource, compare what you applied against what the cluster stored:

kubectl get <kind> <name> -n <your-namespace> -o yaml

If a policy blocks something you have a legitimate reason to do, raise an STS ticket. Policies are managed by the System Team and changed centrally.