Self-Hosting Models

Running open-weight models on your own hardware. When it makes sense, what it actually costs, and the arithmetic to do first.

Self-hosting turns a per-token cost into a fixed infrastructure cost, and turns a vendor relationship into an operations problem.

Both halves matter. The decision is usually made on the first and regretted because of the second.

The three real reasons

Data cannot leave. Regulatory, contractual, or residency requirements that no provider terms satisfy. The strongest reason, and the one where cost is beside the point.

Volume economics. At sustained high throughput, fixed hardware beats per-token pricing. The crossover is genuinely high — do the arithmetic below before assuming you are past it.

Control. Pinned weights that never change under you, no silent model updates, no rate limits, and the ability to fine-tune freely.

Reasons that do not hold up: it feels cheaper (usually is not at low volume), the model is free (the GPUs are not), and it seems more professional.

The memory arithmetic

Do this before anything else. Three components:

Weights. Parameters times bytes per parameter. At 16-bit, a 70B model is about 140GB; quantized to 4-bit, roughly 35GB.

KV cache. Grows with sequence length times layers times heads times concurrent requests. At long context and real concurrency this frequently exceeds the weights, and it is the component people forget.

Overhead. Activations, framework, fragmentation. Budget headroom.

The practical consequence: a model that fits in memory may support only a handful of concurrent long-context requests. Capacity is set by cache, not by whether the weights load.

A larger quantized model usually beats a smaller full-precision one at equal memory — a 4-bit 70B generally outperforms a 16-bit 13B. This is the main decision quantization enables.

Cost, honestly

Compare total cost of ownership against per-token pricing, not GPU rental against per-token pricing.

Hardware or instance cost, which you pay whether or not traffic arrives. This is the crux: hosted APIs cost nothing at idle, and self-hosting costs full price at 3am.

Utilization. A GPU at 10% average utilization is 10× more expensive per token than its headline rate suggests. Bursty traffic is where self-hosting economics fall apart.

Engineering time. Setup, tuning, monitoring, upgrades, incident response. Consistently the underestimated line item, and it does not end after launch.

Redundancy. One machine means one point of failure. Real availability means at least two, doubling hardware cost.

The rough shape: self-hosting wins on steady high volume, loses on bursty or low volume. If your traffic has a daily cycle with a 10× peak-to-trough ratio, you provision for peak and pay for it at trough.

What you have to operate

A serving framework, not a training script. Production inference needs continuous batching, paged attention, and prefix caching — features that separate a serving system from a loop calling generate(). Do not write your own.

Load management. Queueing, admission control, and backpressure. Without these, a traffic spike degrades every request rather than shedding some.

Monitoring. GPU utilization and memory, queue depth, time-to-first-token and tokens-per-second separately, and cache hit rates. See Observability for LLM Systems.

Upgrades. New model versions mean re-testing prompts and re-running evals. Self-hosting does not remove migration work; it means you choose when to do it.

Weight provenance. You are loading files — see Model Supply Chain Risk.

The middle options

The choice is not binary, and the middle is where most projects should land.

Managed open-weight endpoints. Providers serve open models on their infrastructure. Model choice and pinning without operating GPUs, at per-token pricing. Frequently the right answer, and the option most often overlooked.

Dedicated capacity from a hosted provider. Reserved throughput, sometimes with stronger data terms.

Hybrid routing. Sensitive traffic to a self-hosted model, everything else to a hosted API. Resolves the data-residency case without taking on all the volume.

Small models locally, large ones hosted. Classification and extraction on modest local hardware, hard reasoning sent out. Often the best cost profile available.

What to remember

• Three valid reasons: data residency, sustained volume economics, and control over versions.
• Do the memory arithmetic first: weights plus KV cache plus overhead — cache usually sets concurrency, not weights.
• A larger quantized model generally beats a smaller full-precision one at equal memory.
• Compare total cost of ownership at your actual utilization; idle GPUs cost full price and bursty traffic breaks the economics.
• You must operate a real serving framework, load management, monitoring, and upgrades — engineering time is the underestimated cost.
• Managed open-weight endpoints and hybrid routing get most of the benefit without the operations.

You have reached the end of the path. The roadmap has the full map if you want to revisit a layer.