Model Supply Chain Risk

Downloading weights runs someone else's file on your hardware. What can hide in a model, an adapter, or a tool server.

Installing a package from a registry is a known risk with known practices — pinned versions, lockfiles, audit tooling. Downloading model weights is the same category of risk with far less established hygiene.

Four things you might pull in, each carrying different exposure.

Weights that execute code

Some serialization formats store executable code alongside data. Loading such a file can run that code, with whatever permissions your process holds.

Python’s pickle format is the classic case, and older model checkpoints frequently use it. Loading an untrusted pickle is equivalent to running an untrusted script.

The mitigation is straightforward: prefer formats that cannot execute. Safetensors was designed for exactly this — it stores tensors and nothing else, so loading it cannot run code. Where a pickle-based checkpoint is unavoidable, load it in a sandbox and convert.

Check the file format before loading, not after.

Weights with modified behaviour

A model can be fine-tuned so it behaves normally in general and differently on a specific trigger — a particular phrase, a particular input pattern.

This is hard to detect by evaluation, because normal benchmarks do not contain the trigger. A backdoored model passes every test you would think to run.

There is no reliable general detection method. The practical defenses are provenance-based: download from the original publisher’s account rather than a mirror, verify checksums where published, prefer widely-used models where many parties would have noticed anomalies, and treat an unknown uploader’s fine-tune of a popular model as unvetted code.

For anything security-relevant, evaluate on your own data rather than trusting reported numbers.

Adapters and merges

LoRA adapters are small and easy to share, which makes them easy to distribute widely and hard to audit. An adapter modifies behaviour, so all of the above applies at a fraction of the download size.

Merged models compound the problem: a merge inherits from every parent, and community merges frequently have several parents whose own provenance is unclear. A model card listing four ancestors is asserting four separate trust relationships.

Tool servers

An MCP server or plugin is unambiguously code you are running, with whatever access you grant.

Two exposures beyond ordinary dependency risk. Tool descriptions enter your model’s context, so a malicious server can shape model behaviour through them — a description containing instruction-shaped text is a prompt injection vector that arrives before any user input. And server results are untrusted content, exactly like any other fetched data.

Treat a third-party tool server as you would a dependency with network access and filesystem permissions: read what it does, pin the version, and scope its credentials narrowly.

Practical hygiene

Pin versions. Model repositories mutate. Pin to a specific revision or commit hash, not a branch name — otherwise your deployment changes when someone updates the repo.

Verify checksums when published.

Prefer safetensors or another non-executing format.

Download from the publisher, not a re-upload.

Load untrusted weights in a sandbox first, with no network access and no credentials.

Record what you deployed. Model, revision, quantization, and adapters. When behaviour changes, this is how you find out why — and it is the same record you need for migration.

Scan for known-bad patterns. Some registries flag unsafe serialization automatically; use that signal where available.

Hosted APIs shift the risk

Using a hosted model removes weight-level supply chain risk entirely — you never load a file.

It replaces it with a different exposure: you are trusting the provider’s pipeline, and the served model can change without notice. Providers update models under stable version names, which is why evals need re-running periodically rather than only at integration time.

Neither option removes the need to verify behaviour yourself. They just move where the uncertainty sits.

What to remember

• Loading weights can execute code in pickle-based formats — prefer safetensors, which cannot.
• Backdoored weights pass normal evaluation; defenses are provenance-based, not detection-based.
• Adapters and community merges inherit trust from every ancestor.
• Tool servers are code, their descriptions reach your context, and their results are untrusted.
• Pin revisions rather than branches, verify checksums, download from the publisher, and record exactly what you deployed.
• Hosted APIs remove weight risk and add silent model change — re-run evals either way.

Next: Content Moderation