LoRA and Parameter-Efficient Fine-Tuning

Train a small number of extra weights instead of all of them. Nearly all the benefit at a fraction of the cost, plus swappable adapters.

Full fine-tuning updates every parameter. For a 70B model that means holding 70 billion weights plus gradients plus optimizer state — several times the model’s own memory, which puts it out of reach of most hardware.

Parameter-efficient methods train a small number of additional parameters and leave the original weights frozen. The dominant one is LoRA, and its core observation is worth understanding.

The low-rank idea

When you fine-tune, the weights change by some amount. Call that change ΔW — a matrix the same shape as the original weight matrix.

The observation: ΔW has low intrinsic rank. The adaptation needed for a specific task is far simpler than the full space of possible weight changes. It can be well approximated by the product of two much smaller matrices.

So instead of learning a 4096×4096 ΔW directly — about 16.8 million numbers — learn two matrices: 4096×8 and 8×4096. That is roughly 65,000 numbers, a 250× reduction.

At inference, the output is the original frozen weight’s result plus the low-rank product’s result. Nothing about the base model changes.

What this buys

Memory. Optimizer state is proportional to trainable parameters, and that is where full fine-tuning’s memory actually goes. LoRA cuts training memory by a large factor, which is what makes fine-tuning a large model on a single GPU possible at all.

Storage. A LoRA adapter is megabytes rather than the tens or hundreds of gigabytes of a full model copy. You can keep dozens.

Swappability. This is the underrated benefit. One base model in memory, many adapters loaded on demand — a serving system can host many task-specific variants on a single model instance. Full fine-tuning would mean one full model per variant.

No forgetting of the base. Original weights are untouched, so removing the adapter restores original behaviour exactly. The catastrophic forgetting risk is reduced, though not eliminated — a strong adapter can still override useful general behaviour.

The knobs

Rank is the main one — the inner dimension of the two small matrices. Low values (4–8) suffice for narrow tasks like formatting. Higher (32–64) for style or broader behaviour change. Higher rank means more capacity and more parameters; the common mistake is starting too high, since low ranks work better than people expect.

Alpha scales the adapter’s contribution. Typically set to the rank or twice it, and treated together with learning rate rather than tuned independently.

Target modules decide which weight matrices get adapters. Attention projections are the usual choice; including feedforward layers adds capacity and cost. Applying to more modules generally helps more than raising rank on fewer.

QLoRA

The combination that made large-model fine-tuning broadly accessible: quantize the frozen base model to 4-bit, then train LoRA adapters on top of it in higher precision.

The base is only being read during training, never updated, so quantization error there matters much less than it would for the trainable parts. Memory drops enough to fine-tune a 70B model on a single high-memory consumer GPU.

Quality loss versus full-precision LoRA is small in practice. This is one of the higher-leverage tricks in the area.

Where it falls short

Large behaviour changes. For adaptation genuinely requiring broad weight movement — a new language, a substantially different domain — low-rank updates constrain what is reachable. Full fine-tuning or continued pretraining is the honest answer.

Serving complexity if merged. Adapters can be merged into the base weights, eliminating inference overhead but also eliminating swappability. Keeping them separate costs a small amount of compute per forward pass. Pick based on whether you need many variants.

Facts, still. Every limitation from RAG or Fine-Tuning? applies unchanged. Parameter efficiency does not make weight updates a good way to store knowledge.

Other approaches

Prefix and prompt tuning learn continuous vectors prepended to the input rather than modifying weights. Even fewer parameters, generally less effective than LoRA.

Adapter layers insert small trainable modules between existing layers. Predates LoRA; adds inference latency since the modules cannot be merged away.

DoRA and similar variants decompose the update into magnitude and direction components, reporting better results at equal parameter count.

LoRA and QLoRA remain the practical default.

What to remember

• LoRA trains two small matrices approximating the weight change, leaving base weights frozen — often a 100×+ reduction in trainable parameters.
• The premise is that task adaptation has low intrinsic rank.
• Buys training memory, tiny adapter files, and swappable variants on one base model.
• Start with low rank and apply to more modules rather than raising rank.
• QLoRA (4-bit frozen base + LoRA on top) makes large-model fine-tuning fit on one GPU.
• Still the wrong tool for storing facts.

Next: How Models Are Aligned