See Marvin validate, sign, and promote artefacts#

This guided walkthrough takes you through the full artefact lifecycle by feeding Marvin real pushes and watching what it does:

  • a compliant artefact that is promoted (Harbor) or accepted (Nexus);

  • a non-compliant one that is held back, labelled or quarantined, and raised as a merge request;

  • the OCI signature flow end to end — an image that is gated until it is signed, an SBOM that Marvin signs on its own, and the verification chain that proves both.

It is learning-oriented: follow the steps in order and you will see each gate act in turn. For the reasoning behind the gates see How the Artefact Validations service works; for the exhaustive settings, toggles, and retry budgets see Endpoints, toggles, configuration, and metrics; for day-to-day operational recipes see Run and operate the Artefact Validations.

It assumes Marvin is already running (see Deploy with the Helm chart) and only covers exposing it, wiring the webhooks, and feeding it artefacts. Nexus validates every supported format (Helm, Raw, Conan and PyPI); this walkthrough uses a PyPI package in the pypi-test repository, and the Harbor side uses an OCI image.

What you need#

  • Marvin deployed and reachable, with its Celery worker logs to hand (you will follow each decision as it happens).

  • Push access to a Harbor staging test project (test-validations) and visibility of the production test project (test-promotion), plus a Nexus pypi-test repository.

  • For the signature flow, on your own machine:

    • notation 1.3.1 (the version the worker image bundles), configured with a signing key and a trust policy so you can sign and verify.

    • regctl to inspect artefact accessories.

    • Push access to the staging project so you can stand in for the pipeline’s image signer.

Note

<...> marks a value specific to your environment — a host, a project, a tag, or a digest you read off Harbor.

Step 1 — expose Marvin to the registries#

Nexus (artefact.skao.int) and Harbor (harbor.skao.int) must be able to reach Marvin’s webhook receiver over the internet.

  • Running locally: put a tunnel in front of the receiver, for example with ngrok:

    ngrok http <marvin-port>
    

    Note the public https://<subdomain>.ngrok.io URL it prints; that is your <base-url> for the next step.

    If Marvin is deployed to a local cluster (for example one provisioned with ska-cicd-deploy-minikube) the receiver is not directly reachable from your host, so forward its service to a local port first and tunnel that (the receiver listens on port 80):

    kubectl port-forward -n <marvin namespace> --address 0.0.0.0 svc/<marvin-service> 8080:80
    ngrok http 8080
    
    ngrok forwarding a public URL to the local Marvin receiver
  • Running in a cluster with a public Ingress: skip the tunnel and use the receiver’s Ingress host as <base-url> (https://<marvin-ingress-host>); the registries reach it directly.

Step 2 — point the test-project webhooks at Marvin#

Configure the Harbor and Nexus webhooks as described in Register the webhooks, using the <base-url> from Step 1:

  • Harbor uses two webhooks, on two projects. Both set the Auth Header to the value of Marvin’s HARBOR_TOKEN (Harbor sends it as the authorization header and Marvin rejects the request unless it matches):

    • On the staging project (test-validations), an Artifact pushed webhook to <base-url>/harbor — this drives validation and promotion.

    • On the production project (test-promotion), a Scanning finished webhook to <base-url>/harbor/sbom_sign — this drives SBOM signing.

  • Nexus — on the pypi-test PyPI repository, add a Webhook: Repository capability for the component event pointing at <base-url>/nexus. Set the Secret Key to the value of Marvin’s NEXUS_HMAC_SIGNATURE_SECRET (Nexus HMAC-signs the payload with it and Marvin verifies the signature using the same secret).

Check whether these webhooks already exist before creating new ones; often you only need to repoint the URL.

Note

Promotion copies to whatever config.harbor.promotion_repository is set to. The chart’s default is test-promotion, which matches the production test project above, so test-validations → test-promotion works out of the box. A real deployment sets it to production (see Configure the service behaviour).

Harbor Artifact-pushed webhook on the staging project
Harbor Scanning-finished webhook on the production project
Nexus webhook capability pointing at the base URL

Before feeding artefacts, make sure the checks you want to exercise are enabled (see Enable and disable feature toggles) and tail the Celery worker logs so you can follow each decision as it happens.

Step 3 — feed a compliant artefact#

Publish a compliant artefact from your test project’s CI pipeline: one whose name follows the SKAO convention (ska-...) and whose version is valid. Push an OCI image with a SemVer tag to the test-validations project, or a PyPI package with a PEP 440 version to pypi-test.

Marvin runs the checks and, because they all pass:

  • Harbor — copies the image into the test-promotion project (the worker logs ... promoted to test-promotion repository). The image now appears there.

  • Nexus — leaves the PyPI package in place; nothing is quarantined.

The compliant image present in the Harbor production project

Step 4 — feed a non-compliant artefact#

Now publish a deliberately non-compliant artefact — for example, a name without the ska- prefix, or an invalid tag such as latest or v1.

Marvin records the failed checks and holds the artefact back:

  • Harbor — applies an Invalid* label (InvalidName, InvalidTag, …) to the artefact in test-validations and does not promote it.

  • Nexus — moves the PyPI package to the pypi-quarantine repository (the quarantine repo is the source project name up to the first - plus -quarantine, so pypi-test → pypi-quarantine).

In both cases Marvin then opens a merge request in the artefact’s GitLab project, with a table of the failed checks and how to fix them, assigned to the author.

Important

The merge request and the metadata check rely on CI-injected metadata (CI_PROJECT_ID, CI_JOB_ID, GITLAB_USER_ID and the other CI_*/GITLAB_* fields). Publish through the pipeline so these are present. A hand-pushed artefact still shows the labels and quarantine, but Marvin cannot open the merge request without them.

Step 5 — test the OCI signature flow end to end#

So far promotion depended only on the checks. This step adds the signature gate: with it on, an image is promoted only once both the image and its SBOM are signed. You will watch Marvin refuse to promote an unsigned image, sign the image yourself (standing in for the pipeline’s image signer), watch Marvin sign the SBOM, and then verify the whole chain.

Two roles produce the two signatures:

  • The image is signed by an external signer — in production, the CI pipeline. Here you sign it with notation.

  • The SBOM is signed by Marvin: when Harbor reports that a scan has finished, the /harbor/sbom_sign webhook has Marvin mint a short-lived Vault certificate and countersign the SBOM with notation (see How the Artefact Validations service works).

Turn on the gate#

Enable two toggles for the test project (see Enable and disable feature toggles):

  • require-signatures — makes the promotion gate require a signed image and a present SBOM (and a signed SBOM too, if verify_sbom_signature is on).

  • sign-sbom — lets Marvin sign the SBOM when the scan finishes.

Whether the gate also requires the SBOM to be signed is governed by the verify_sbom_signature config field, which is false by default: the gate then requires the SBOM only to be present, not signed. This matches the flow you are testing, where Marvin signs the SBOM after promotion (on the production scan), so requiring a signed SBOM before promotion would deadlock. Leave it false unless your project signs SBOMs in staging — see Configure the service behaviour.

Push an unsigned image and watch the gate hold#

Push a compliant, unsigned single-platform image to staging:

docker push harbor.skao.int/test-validations/pandabox:1.0.0

Harbor fires the Artifact pushed webhook and Marvin runs the checks, then the signature gate. Because the image is not signed yet, the worker logs that the artefact is “not fully signed yet”, polls its accessories signature_check_retries × signature_check_retry_delay_s (5 × 30 s by default), then raises ArtefactNotSignedError and retries the whole task every 10 minutes for up to an hour. The image stays in test-validations, unpromoted.

Sign the image#

Read the image’s manifest digest off its page in Harbor (or with regctl manifest head), then sign that digest — signing the tag would sign a moving target:

notation sign --timestamp-url http://timestamp.digicert.com \
  --timestamp-root-cert <path to DigiCertTrustedRootG4.crt> \
  harbor.skao.int/test-validations/pandabox@sha256:<image-digest>

The --timestamp-* flags countersign the signature with the DigiCert RFC 3161 timestamping authority so it stays verifiable after the signing certificate expires — the same authority Marvin uses for the SBOM.

Watch Marvin promote it#

Either wait for the next task retry (up to 10 minutes) or replay the push immediately without re-pushing the image — see Re-trigger a Harbor validation. This time the image signature is present, so once the SBOM requirement is also met the gate passes and Marvin copies the image into test-promotion (... promoted to test-promotion repository). When Harbor then reports the scan finished, the /harbor/sbom_sign webhook has Marvin sign the SBOM.

Note

If the image never promotes, the signature gate is the usual cause. Walk through Investigate an image that never gets promoted to find which accessory is missing.

Verify the whole chain#

Confirm the image signature verifies on the promoted copy, by tag and by digest:

notation verify -v harbor.skao.int/test-promotion/pandabox:1.0.0
notation verify -v harbor.skao.int/test-promotion/pandabox@sha256:<promoted-image-digest>

Then list the artefact’s accessories and confirm both an sbom.harbor SBOM and the signature.notation signatures are present (the image signature, and Marvin’s signature nested on the SBOM):

regctl image copy --referrers harbor.skao.int/test-promotion/pandabox:1.0.0 ocidir:///tmp:pandabox
regctl artifact list harbor.skao.int/test-promotion/pandabox:1.0.0
signature.notation and sbom.harbor accessories on the promoted image

Prove SBOM signing is idempotent#

Marvin signs each SBOM once. Re-deliver the Scanning finished event for the same image — post a SCANNING_COMPLETED body to /harbor/sbom_sign the same way you replay a push (see Re-trigger a Harbor validation). The body must name the SBOM scan, otherwise it is ignored:

{
  "type": "SCANNING_COMPLETED",
  "occur_at": 1744716446,
  "operator": "admin",
  "event_data": {
    "resources": [
      {
        "digest": "sha256:<image-digest>",
        "resource_url": "harbor.skao.int/test-promotion/<repo>@sha256:<image-digest>"
      }
    ],
    "repository": {"name": "<repo>", "namespace": "test-promotion"},
    "scan": {"scan_type": "sbom"}
  }
}

Because the SBOM already carries a signature.notation accessory, the worker logs that it is “already signed” and returns without creating a second signature. Duplicate webhooks, Celery retries, and re-scans are therefore cheap no-ops.

Swap "scan_type": "sbom" for "vulnerability" and the request is acknowledged with {"message": "Ignored"} instead: Harbor reports both scans under the same event type, and only the SBOM one has an SBOM to sign.

Multi-platform images#

For a multi-platform image (an OCI index) the gate requires the index and every child manifest to be signed, and every child to have its own SBOM (except BuildKit attestation manifests, which only need a signature). Copy a multi-platform image into staging and sign each platform digest in turn:

regctl image copy artefact.skao.int/ska-base:1.0.0-rc.1 \
  harbor.skao.int/test-validations/ska-base:1.0.0-rc.1

notation sign --timestamp-url http://timestamp.digicert.com \
  --timestamp-root-cert <path to DigiCertTrustedRootG4.crt> \
  harbor.skao.int/test-validations/ska-base@sha256:<platform-digest>
# repeat for every remaining platform digest, and for the index digest

After promotion, confirm the index and each child carry their signatures:

regctl manifest get harbor.skao.int/test-promotion/ska-base:1.0.0-rc.1
regctl artifact list harbor.skao.int/test-promotion/ska-base@sha256:<index-or-child-digest>

If a single child is left unsigned the gate holds the whole image back, exactly as in the single-platform case.

Where to next#