Vault How-to Guides#

Practical tasks for managing secrets with Vault.

Setup auto rotating secrets#

When a secret leaks (e.g., a Slack webhook committed to a public repository), you need to rotate it quickly. VSO can automatically synchronise the new secret and restart affected workloads.

1. Update the secret in Vault#

Generate a new secret value and update it in Vault. See Vault for login instructions.

2. Configure refresh interval#

Set refreshAfter to control how frequently VSO checks for updates:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: slack-webhook
spec:
  type: kv-v2
  mount: <engine>
  path: <path/to/secret>
  refreshAfter: 60s
  destination:
    name: slack-webhook
    create: true
    overwrite: true
    transformation:
      excludeRaw: true
      includes:
        - webhook

VSO checks Vault at least every 60 seconds.

3. Add rollout restart target#

Even after the Kubernetes secret updates, running pods still have the old value. Add rolloutRestartTargets to automatically restart workloads:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: slack-webhook
spec:
  type: kv-v2
  mount: <engine>
  path: <path/to/secret>
  refreshAfter: 60s
  rolloutRestartTargets:
    - kind: Deployment
      name: slack-notifier
  destination:
    name: slack-webhook
    create: true
    overwrite: true
    transformation:
      excludeRaw: true
      includes:
        - webhook

Supported target kinds: Deployment, StatefulSet, DaemonSet.

Warning

Use automatic rollouts carefully — they can affect application stability during secret rotations.

Supply Helm values from Vault in GitLab CI/CD#

Replace Makefile switches and GitLab CI variables with Vault-stored values files for cleaner, more auditable deployments.

Why use Vault for Helm values?#

Traditional approach using Makefile switches:

helm upgrade --install test \
  --set global.minikube=false \
  --set global.exposeDatabaseDS=false \
  --set global.exposeAllDS=false \
  --set global.tango_host=tango-databaseds:10000 \
  --set global.device_server_port=45450 \
  --set global.operator=false \
  --set global.cluster_domain=techops.internal.skao.int \
  ./charts/ska-tango-umbrella/

Problems: Values are hard to trace, logic scatters across Makefile, poor auditability.

Vault approach:

helm upgrade --install test \
  -f <(envsubst < /path/to/ENVIRONMENT_VALUES) \
  -f <(envsubst < /path/to/DEP_STRATEGY_VALUES) \
  -f <(envsubst < /path/to/APP_VALUES) \
  ./charts/ska-tango-umbrella/

Benefits: Clear precedence, single source of truth, full audit trail.

1. Structure your values in Vault#

Split configurations by purpose:

Environment-specific values (stfc-techops/production/default/values.yml@dev):

global:
  cluster_domain: techops.internal.skao.int
  exposeAllDS: false
  exposeDatabaseDS: false
  minikube: false

Application-specific values (skao-team-system/ska-tango-charts/values.yml@dev):

global:
  tango_host: tango-databaseds:10000
  device_server_port: 45450

Deployment strategy values (shared/default/operator/values.yml@dev):

global:
  operator: true

2. Configure GitLab CI/CD job#

k8s-test:
  variables:
    KUBE_NAMESPACE: 'ci-$CI_PROJECT_NAME-$CI_COMMIT_SHORT_SHA'
    K8S_VALUES_FILES: "${ENVIRONMENT_VALUES} ${DEP_STRATEGY_VALUES} ${APP_VALUES}"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com
  secrets:
    ENVIRONMENT_VALUES:
      vault: ${CLUSTER_DATACENTRE}/${CLUSTER_ENVIRONMENT}/default/values.yml@dev
      file: true
    DEP_STRATEGY_VALUES:
      vault: shared/default/operator/values.yml@dev
      file: true
    APP_VALUES:
      vault: skao-team-system/ska-tango-charts/values.yml@dev
      file: true

file: true injects secrets as files rather than environment variables.

3. Update Makefile#

Replace switch-based parameters with values files:

ifneq ($(K8S_VALUES_FILES),)
K8S_CHART_PARAMS ?= $(foreach f,$(K8S_VALUES_FILES),-f <(envsubst < $(f)))
endif

envsubst substitutes environment variables in the values files.

4. Add contextual values (optional)#

Include deployment context for traceability:

shared/default/context/values.yml@dev#
global:
  context:
    gitlab:
      author: ${CI_COMMIT_AUTHOR}
      ref: ${CI_COMMIT_REF_NAME}
      commit: ${CI_COMMIT_SHA}
      pipelineId: ${CI_PIPELINE_ID}
      projectId: ${CI_PROJECT_ID}
      project: ${CI_PROJECT_PATH}
    kubernetes:
      datacentre: ${CLUSTER_DATACENTRE}
      environment: ${CLUSTER_ENVIRONMENT}
      namespace: ${KUBE_NAMESPACE}

5. Reuse deployed values in tests#

After deploying with values files, retrieve the actual values for use in tests:

RELEASE_VALUES_FILE ?= $(RELEASE_NAME).$(KUBE_NAMESPACE).values.yml

ifneq ($(K8S_VALUES_FILES),)
K8S_CHART_PARAMS ?= $(foreach f,$(K8S_VALUES_FILES),-f <(envsubst < $(f)))
ifneq ("$(wildcard $(RELEASE_VALUES_FILE))","")
$(info Inferring environment from release information ...)
SKA_TANGO_OPERATOR := $(shell jq -r '.global.operator' $(RELEASE_VALUES_FILE))
TANGO_HOST := $(shell jq -r '.global.tango_host' $(RELEASE_VALUES_FILE))
endif
endif

k8s-post-install-chart:
    @helm get values $(RELEASE_NAME) -n $(KUBE_NAMESPACE) -o json > $(RELEASE_VALUES_FILE)

Test jobs then use the exact values from the deployment.

Add a secret to Vault#

  1. Log in to Vault with GitLab SSO

  2. Navigate to your team’s KV engine (dev/<team-slug>/)

  3. Create a subdirectory for your application

  4. Add key-value pairs for your secrets

  5. Create a VaultStaticSecret resource to sync to Kubernetes

Note

Secrets must be in subdirectories — you cannot create secrets at the root of your user or team path.

Request team access to Vault#

Your team needs a corresponding GitLab Group to access team-specific Vault paths.

If your team doesn’t have one, request access via STS.