Artifact Integrity Verification: Checksums, Signatures, and Transparency Logs

Artifact Integrity Verification: Checksums, Signatures, and Transparency Logs

Problem

Build artifacts pass through multiple stages between source code and production deployment. Source is compiled in CI, packaged into a container image, pushed to a registry, pulled by a deployment tool, and launched in a cluster. At each boundary, an attacker can substitute a modified artifact. A compromised CI runner can alter the binary after compilation. A man-in-the-middle on the registry network path can serve a different image. A compromised deployment controller can deploy an image that was never built by CI.

Checksums alone are insufficient. A SHA-256 digest proves an artifact has not changed since the digest was computed, but it does not prove who built the artifact or from what source. Signatures tie an artifact to an identity, but without a transparency log, a compromised signing key can sign malicious artifacts without detection. End-to-end integrity requires all three: checksums for tamper detection, signatures for provenance, and transparency logs for accountability.

Threat Model

  • Adversary: Compromised CI runner that modifies build output, attacker who gains write access to the container registry, or insider who replaces an artifact between pipeline stages.
  • Objective: Deploy a tampered artifact to production. The tampered artifact may contain backdoors, credential-harvesting code, or cryptocurrency miners.
  • Blast radius: Every environment that deploys the tampered artifact. Without verification at deployment time, the compromise persists until someone notices anomalous behavior.

Configuration

Signing Container Images with Cosign

Sign images immediately after building, using keyless signing backed by Sigstore’s Fulcio and Rekor:

# .github/workflows/build-sign-verify.yml
name: Build, Sign, and Attest
on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write   # OIDC for keyless signing
  packages: write   # Push to GHCR

jobs:
  build-and-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11

      - name: Install cosign
        uses: sigstore/cosign-installer@v3

      - name: Login to GHCR
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" | \
            docker login ghcr.io -u ${{ github.actor }} --password-stdin

      - name: Build and push image
        id: build
        run: |
          IMAGE="ghcr.io/${{ github.repository }}:${{ github.sha }}"
          docker buildx build --push --tag "$IMAGE" .
          DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE")
          echo "image=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Sign image (keyless)
        run: |
          # Keyless signing: uses GitHub OIDC token to get a short-lived
          # certificate from Fulcio. The signature is recorded in Rekor
          # (transparency log) automatically.
          cosign sign --yes ${{ steps.build.outputs.image }}

      - name: Generate and attach SLSA provenance
        run: |
          # Create SLSA provenance attestation
          cosign attest --yes \
            --predicate <(cat <<PROVENANCE
          {
            "_type": "https://in-toto.io/Statement/v0.1",
            "predicateType": "https://slsa.dev/provenance/v1",
            "predicate": {
              "buildDefinition": {
                "buildType": "https://github.com/actions/runner",
                "externalParameters": {
                  "repository": "${{ github.repository }}",
                  "ref": "${{ github.ref }}",
                  "commit": "${{ github.sha }}"
                }
              },
              "runDetails": {
                "builder": {
                  "id": "https://github.com/actions/runner"
                },
                "metadata": {
                  "invocationId": "${{ github.run_id }}",
                  "startedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
                }
              }
            }
          }
          PROVENANCE
          ) \
            --type slsaprovenance \
            ${{ steps.build.outputs.image }}

Verifying Signatures at Deployment Time

Use Kyverno to enforce signature verification before any pod is admitted to the cluster:

# kyverno/policies/verify-image-signatures.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/your-org/*"
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/your-org/*"
                    rekor:
                      url: "https://rekor.sigstore.dev"
          attestations:
            - type: slsaprovenance
              conditions:
                - all:
                    - key: "{{ buildDefinition.externalParameters.repository }}"
                      operator: Equals
                      value: "your-org/*"

In-Toto Attestation for Multi-Stage Pipelines

For pipelines with multiple stages (build, scan, test, deploy), use in-toto to create a verifiable chain of custody:

# in-toto layout defining the expected pipeline stages
# layout.json - signed by the project owner
{
  "_type": "layout",
  "expires": "2027-01-01T00:00:00Z",
  "steps": [
    {
      "name": "build",
      "expected_command": ["docker", "buildx", "build"],
      "expected_materials": [
        ["MATCH", "Dockerfile", "WITH", "PRODUCTS", "FROM", "checkout"]
      ],
      "expected_products": [
        ["CREATE", "image.tar"]
      ],
      "pubkeys": ["build-runner-key-id"],
      "threshold": 1
    },
    {
      "name": "scan",
      "expected_command": ["trivy", "image"],
      "expected_materials": [
        ["MATCH", "image.tar", "WITH", "PRODUCTS", "FROM", "build"]
      ],
      "expected_products": [
        ["CREATE", "scan-report.json"]
      ],
      "pubkeys": ["scan-runner-key-id"],
      "threshold": 1
    },
    {
      "name": "sign",
      "expected_command": ["cosign", "sign"],
      "expected_materials": [
        ["MATCH", "image.tar", "WITH", "PRODUCTS", "FROM", "build"],
        ["MATCH", "scan-report.json", "WITH", "PRODUCTS", "FROM", "scan"]
      ],
      "pubkeys": ["signing-key-id"],
      "threshold": 1
    }
  ],
  "inspect": [
    {
      "name": "verify-no-critical-cves",
      "expected_materials": [
        ["MATCH", "scan-report.json", "WITH", "PRODUCTS", "FROM", "scan"]
      ],
      "run": ["python", "verify_scan.py"]
    }
  ]
}

Generate in-toto link metadata at each pipeline stage:

#!/bin/bash
# Stage 1: Build - generate in-toto link
in-toto-run \
  --step-name build \
  --key build-runner-key \
  --materials Dockerfile requirements.txt \
  --products image.tar \
  -- docker buildx build --output type=tar,dest=image.tar .

# Stage 2: Scan - generate in-toto link
in-toto-run \
  --step-name scan \
  --key scan-runner-key \
  --materials image.tar \
  --products scan-report.json \
  -- trivy image --input image.tar --format json --output scan-report.json

# Stage 3: Sign - generate in-toto link
in-toto-run \
  --step-name sign \
  --key signing-key \
  --materials image.tar scan-report.json \
  -- cosign sign --key cosign.key "ghcr.io/your-org/app@sha256:abc123..."

# Verification: verify the entire supply chain
in-toto-verify \
  --layout layout.json \
  --layout-key project-owner-key.pub

Transparency Logs with Rekor

Rekor provides a tamper-evident log of signing events. Even if a signing key is compromised, the transparency log creates an auditable record:

# Search Rekor for all signing events for your image
rekor-cli search --sha "sha256:abc123def456..."

# Verify a specific entry in the transparency log
rekor-cli verify --artifact image.tar --signature image.tar.sig --pki-format x509

# Monitor for unexpected signing events
# This script should run on a schedule to detect unauthorized signatures
#!/bin/bash
EXPECTED_ISSUER="https://token.actions.githubusercontent.com"
EXPECTED_SUBJECT="https://github.com/your-org/"

# Search for recent entries for your image
rekor-cli search --sha "$IMAGE_DIGEST" --format json | jq -r '.[]' | while read -r entry; do
  ISSUER=$(rekor-cli get --uuid "$entry" --format json | jq -r '.Body.HashedRekordObj.signature.publicKey.content' | base64 -d | openssl x509 -noout -ext subjectAltName 2>/dev/null)

  if [[ "$ISSUER" != *"$EXPECTED_SUBJECT"* ]]; then
    echo "ALERT: Unexpected signer for image $IMAGE_DIGEST"
    echo "Entry: $entry"
    echo "Issuer: $ISSUER"
    # Send alert to security team
  fi
done

Checksum Verification Between Pipeline Stages

When artifacts move between pipeline stages (even within the same CI system), verify checksums at each boundary:

# .github/workflows/multi-stage-pipeline.yml
jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.checksum.outputs.digest }}
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
      - name: Build
        run: docker buildx build --output type=tar,dest=image.tar .
      - name: Compute checksum
        id: checksum
        run: |
          DIGEST=$(sha256sum image.tar | awk '{print $1}')
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with:
          name: image-tar
          path: image.tar

  scan:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: image-tar
      - name: Verify checksum from build stage
        run: |
          ACTUAL=$(sha256sum image.tar | awk '{print $1}')
          EXPECTED="${{ needs.build.outputs.digest }}"
          if [ "$ACTUAL" != "$EXPECTED" ]; then
            echo "INTEGRITY FAILURE: Artifact modified between stages"
            echo "Expected: $EXPECTED"
            echo "Actual:   $ACTUAL"
            exit 1
          fi
          echo "Checksum verified: $ACTUAL"
      - name: Scan
        run: trivy image --input image.tar --severity HIGH,CRITICAL --exit-code 1

Expected Behaviour

  • Every container image pushed to the registry is signed with a keyless cosign signature (OIDC-backed)
  • SLSA provenance attestations are attached to every production image
  • Kyverno verifies image signatures and attestations before admitting pods
  • In-toto link metadata is generated at each pipeline stage, creating a verifiable chain of custody
  • Checksums are verified at every stage boundary to detect inter-stage tampering
  • All signing events are recorded in the Rekor transparency log
  • A monitoring job checks Rekor for unexpected signing events weekly

Trade-offs

Control Impact Risk Mitigation
Keyless signing with Fulcio Eliminates key management; signatures tied to OIDC identity Depends on Sigstore infrastructure availability Sigstore provides 99.5% SLA. For air-gapped environments, deploy private Sigstore stack.
Kyverno image verification Adds 1-3 seconds to pod admission Verification failure blocks all deployments Configure Kyverno with audit mode first. Switch to enforce after validating all images are signed.
In-toto attestation Adds 5-10 seconds per pipeline stage; additional complexity Incorrect layout definition blocks valid deployments Test layouts in staging before production. Version layout files alongside pipeline definitions.
Rekor transparency log Creates permanent, public record of signing events Image names and signing identities are visible in the public log Use a private Rekor instance for sensitive project names.
Inter-stage checksum verification Adds verification steps to pipeline; slightly increases CI time Checksum failures from legitimate causes (compression differences) cause false positives Use consistent artifact formats (tar, OCI) without compression to ensure deterministic checksums.

Failure Modes

Failure Symptom Detection Recovery
Sigstore outage Keyless signing fails; images cannot be signed cosign sign returns connection error; Sigstore status page shows incident Fall back to key-based signing with a locally stored cosign key. Sign with keyless once Sigstore recovers.
Kyverno webhook down All pod creation blocked (fail-closed) or all pods admitted without verification (fail-open) Pods fail to schedule with webhook timeout; Kyverno health check alerts Restart Kyverno pods. If persistent, temporarily switch to audit mode. Run Kyverno in HA (3+ replicas).
In-toto layout mismatch Pipeline verification fails at final stage in-toto-verify returns non-zero with layout violation details Review which step deviated from the layout. Update the layout if the pipeline changed legitimately.
Artifact tampering detected Checksum mismatch between pipeline stages Stage fails with integrity failure message Investigate the runner that produced the mismatched artifact. Rebuild from source on a clean runner.
Unauthorized signature in Rekor Image signed by unexpected identity Rekor monitoring script alerts on unknown issuer/subject Investigate the signing event. If malicious, revoke the signing identity and rebuild the image.

When to Consider a Managed Alternative

Running a complete artifact integrity pipeline (signing, attestation, verification, monitoring) requires expertise in cryptographic tooling and ongoing maintenance of verification infrastructure. Snyk (#48) provides integrated supply chain verification with less operational overhead. For teams that need SLSA compliance but lack the engineering capacity to build the infrastructure, GitHub Artifact Attestations provides built-in SLSA Build L3 provenance for GitHub Actions. Aqua (#123) offers admission-time verification with managed policy infrastructure. For air-gapped environments that cannot use public Sigstore, deploying a private Sigstore stack (Fulcio + Rekor + TUF) requires dedicated infrastructure and key ceremony procedures.

Premium content pack: In-toto layout templates for common pipeline architectures (build-scan-deploy, build-test-scan-stage-deploy). Includes cosign signing workflows for GitHub Actions and GitLab CI, Kyverno verification policies, and Rekor monitoring scripts.