Supply-chain hardening: my blueprint against supply-chain attacks

The supply-chain incident of March 2026 prompted me to harden the build and release chains of all my applications, from PaddleDoc to the internal tools. This post is the blueprint I followed. It is kept product-neutral and carries over to any application with an automated pipeline.

1. Why now

In March 2026, more than 2,500 companies and over 400,000 CI/CD pipelines were attacked through a single compromised open-source component. The pattern: a tool that build pipelines fetch unchecked in whatever version is “latest” was replaced with a malicious build. The malware then ran automatically inside the victims’ release processes and stole credentials and signing keys. Not a single line had changed in the victims’ own code.

This was possible because pipelines fetch their third-party components through mutable references (“take version 4”, “take the latest”) instead of immutable ones:

Process diagram: an attacker captures an automation token and poisons the release of a third-party tool project. Your own pipeline fetches the tool unchecked, runs the malicious code during the build and loses its secrets to the attacker.

The decisive moment is step 3, the unverified fetch. Everything before it happens outside my sphere of influence. Everything after it is mere consequence. That transition is exactly where the blueprint intervenes.

2. Protection goal

No part of my build and release process may change without that change showing up on my side as a visible, review-required code change. Whoever compromises one of my suppliers must not automatically reach my deliverables.

3. The five measures in detail

M1: Immutable references for pipeline building blocks

The problem. CI pipelines are assembled from third-party building blocks: checkout, build tooling, publish steps. They are usually referenced through version labels like @v4. Technically, such a label is nothing but a movable pointer. The vendor can repoint it at any time, an attacker holding the vendor’s credentials just as easily, and every pipeline will execute the new code on its next run without asking. Repointing already-released labels was the core of the incident. Real cases involving widely used CI components existed before it.

The measure. Every third-party building block is pinned to the cryptographic fingerprint (commit hash) of an exact, reviewed state. A fingerprint cannot be repointed. It is the content. The human-readable version stays next to it as a comment. Completeness matters: the rule applies to all pipelines, especially the inconspicuous ones. Docs builds and helper workflows run with the same privileges as the prominent ones.

Effect and effort. A compromised vendor no longer reaches my pipeline without a visible change appearing on my side first. The migration costs less than an hour per application. Ongoing upkeep is handled by the automation in M5.

M2: Authenticity checks for downloaded tools

The problem. Pipelines download binary tools at runtime and execute them directly. HTTPS only protects the path, the source stays unverified. If the download server or the release artifact itself is compromised, the encrypted connection dutifully delivers the manipulated tool. Straight into a pipeline that then continues with publishing privileges.

The measure. For every tool fetched at runtime, I record the SHA-256 checksum the vendor publishes separately in the pipeline code. Before execution, the download is verified against it. Any mismatch fails the build hard.

Effect and effort. A manipulated artifact is caught before it runs. The effort is a few minutes per tool, once.

M3: Full hash-locking of product dependencies

The problem. The shipped product consists largely of third-party packages. That includes the ones named directly and a tree of dozens to hundreds of transitively pulled-in packages. Every unpinned spot in that tree means: the next build may contain different code than the last one, without anyone having changed anything. The incident also demonstrated that package ecosystems can execute code at install time. A poisoned package never needs to be imported to do damage.

The measure. A lock file freezes the entire dependency tree, every package with its exact version and the checksums of its artifacts. Installation runs in strict mode: any artifact whose checksum is missing or different fails the build. That holds even at an unchanged version number and closes the second, subtler attack class, the swap of an already-published artifact.

Two details decide how much this is worth. First, I generate and validate the lock file in exactly the environment that builds the real releases: same OS, same architecture, same interpreter. Dependency resolution is platform-dependent, and a lock produced on a laptop can resolve differently in the release build. Second, strict resolution surfaces real inconsistencies. In my case, the newest release of a central package declared requirements that contradicted those of its own sub-dependency. The previous unpinned build had quietly resolved that conflict “somehow” for years. The lock makes such states visible and decidable.

Effect and effort. Builds become reproducible. Neither a malicious new version nor a swapped old artifact reaches the product unnoticed. The one-off cost is a few hours including validation in the build context. From then on, M5 maintains the file.

M4: A quality gate in front of every release

The problem. In many pipelines a single signal is enough to publish: pushing a release tag. The incident showed that precisely this signal can be forged. And even without an attacker: an untested release is an operational risk.

The measure. Before publishing, the release pipeline runs the complete automated verification suite, on exactly the state that is about to be released. Only a fully green run unlocks the publish steps.

Effect and effort. A manipulated or simply broken state can no longer be published silently. That costs a few extra minutes per release. This verification is what the release process exists for.

M5: Updates with mandatory review

The problem. Pinned states age. Pinning without an update process trades supply-chain risk for the risk of outdated components. And taking updates automatically reopens the door just closed. The incident spread precisely through unchecked adoption of “the latest”.

The measure. An automated service checks all pinned references weekly (pipeline building blocks, product dependencies, base images) and files a review-required change request for every available update: a visible diff, links to the release notes, an automatic test run. Nothing is adopted without my review. Auto-merge is switched off. The service also maintains the fingerprints from M1 and the lock files from M3, which keeps the hardening operable for good.

Effect and effort. Security and currency stop being a trade-off. The ongoing cost is the weekly review of the proposals. Minutes, not hours.

Enforcement points at a glance

Every measure has a clear enforcement point, an enforcing party and a moment in time. That is what separates a catalog of controls from a statement of intent:

Layer diagram: the five measures M1 to M5 mapped to the four enforcement points pipeline definition, build, release, and maintenance and operations, each with enforcing party and timing. The build layer with checksum verification and hash lock is highlighted as the core of the hardening.

4. Before / after

Reference point	Before	After
CI building blocks (actions)	version label, movable after the fact	commit fingerprint, immutable (M1)
Runtime tools	downloaded and executed unchecked	checksum verification before execution (M2)
Product dependencies	partly unpinned, “latest wins”	version + checksum, incl. transitive packages (M3)
Release trigger	release signal suffices	release signal and green verification run (M4)
Updates	implicit with the next build	explicit, review-required change requests (M5)
A change at a supplier	flows in invisibly	shows up as a visible code change on my side

5. Quality assurance of the rollout

I verified every pinned fingerprint independently against its original source, every checksum against the vendor’s official publication. I tested the installability of the locked dependencies in the real build context before adopting the changes. The first real release after the migration went through the hardened chain end to end, including the quality gate and hash-verified installation of every locked package.

6. Limits and residual risk

Honesty is part of any security concept worth relying on. What this blueprint does not do:

Residual-risk diagram: what each layer catches and what remains

Compromise before pinning. Pinning a state that was already malicious pins the damage with it. These measures freeze trust. They cannot create it. It has to be earned in the review of new states (M5).
Quality of the human review. M5 replaces automatic adoption with a human decision. If I approve every proposal unread, the benefit evaporates.
Compromise of my own accounts. Whoever gains write access to the repository or registry bypasses the supply-chain controls. Other layers help there: short-lived minimal-privilege tokens, tag and branch protection, two-factor authentication.
Runtime behaviour. The measures secure the way into the artifact. What a legitimately included package does at runtime is outside their scope.

7. Extension stages (optional, recommended)

Protect release tags and the main branch against retroactive overwriting (a platform setting, not code).
Pin base container images by fingerprint.
Remove “latest” defaults from deployment configurations.
Cryptographically sign your own published artifacts.

8. Applying it to the next application: the checklist

Inventory: record every place the pipeline fetches third-party content (CI building blocks, tool downloads, package installs, base images).
M1: switch CI building blocks to commit fingerprints. The version number stays readable as a comment.
M2: record the official checksum for every tool download.
M3: raise unpinned installs to exact versions and move them into a hash-locked dependency file. Validate in the real build context.
M4: make the publish pipeline depend on the full verification run.
M5: enable automated update proposals, without auto-merge. They also maintain the fingerprints from M1 and the locks from M3.
Verification: independently re-check fingerprints and checksums against the original sources before adopting the changes.

Initial effort is in the range of a few hours per application. What remains is the weekly review of update proposals.

9. What this looks like in CI

Five snippets from the real pipelines, trimmed to the essentials.

M1: Pin actions

# before: a movable label
- uses: actions/checkout@v5

# after: a commit fingerprint, the version stays readable as a comment
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0

M2: Verify tool downloads

- name: Install Helm
  run: |
    curl -fsSL https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz -o helm.tgz
    echo "9318379b847e333460d33d291d4c088156299a26cd93d570a7f5d0c36e50b5bb  helm.tgz" | sha256sum -c -
    tar -xzf helm.tgz
    sudo mv linux-amd64/helm /usr/local/bin/helm

If the checksum does not match, the build fails. A tool update is now a two-line diff: new version, new checksum.

M3: Install against the hash lock

COPY requirements-worker.txt /app/requirements-worker.txt
RUN pip install --no-cache-dir --require-hashes -r /app/requirements-worker.txt

The lock file is generated with pip-compile --generate-hashes in the same container image that builds the releases. Any artifact without a matching hash stops the installation.

M4: Release gate

jobs:
  ci:
    uses: ./.github/workflows/pr-ci.yml
  publish:
    needs: ci

The release tag alone no longer publishes anything. Only a green verification run unlocks the publish job.

M5: Dependabot

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

The same blocks exist for npm, pip and docker. Auto-merge stays off.

10. Who decides what

Dependabot is diligent, but it decides nothing. Every proposal takes the same path:

Process diagram: Dependabot checks the pins weekly and opens one pull request per update, the platform tests it automatically, then I read the diff and release notes and merge or close. On merge, pins and locks are updated.

Signal	Comes from	What needs to happen	When
Patch or minor update (npm, pip, actions)	Dependabot PR	read the diff and release notes, merge on green CI	in the weekly batch review
Major update or new base image	Dependabot PR	check breaking changes and test locally, merge only then	once there is time for a real test
Verification run red on main	CI notification	look immediately, fix or roll back	same day
Release tag pushed	myself	nothing. The gate decides: only a green run publishes	automatic
New dependency in code	development	pin and hash belong in the same commit as the dependency	before merge

And when a proposal is rejected? The PR gets closed with a one-sentence reason. Dependabot only comes back once the next version is released.

Source on the incident: CloudSEK, “AI Supply Chain Breach — 2,500 Companies, 434,000 CI/CD Pipelines” (March 2026).

This blueprint is also published in the PaddleDoc wiki on GitHub and maintained there as a document edition.