A/B Testing AI Features

Nondeterministic output and subjective quality break the usual experiment design. What to measure instead, and why offline evals come first.

Testing a button colour is straightforward: two variants, one metric, enough traffic. Testing a prompt change is harder for three reasons.

Output is nondeterministic, so the same variant produces different responses to the same input. Quality is often subjective, with no click to count. And the metric you care about — did this actually help the user — is usually several steps removed from the model call.

Run offline evals first

Live experiments are slow and expose users to regressions. Most changes should be decided before they reach traffic.

Your eval set answers most questions in minutes. Only take a change live when the eval shows improvement or cannot distinguish the variants — and the second case is common, which is precisely when a live test earns its cost.

Sequence that works: eval → shadow → small traffic percentage → full rollout.

Shadow mode

Run both variants on real traffic, log both outputs, serve only the incumbent.

You get comparison on genuine production inputs — including the messy ones absent from your eval set — with zero user risk. Then judge the logged pairs offline, by human review on a sample or a model judge doing pairwise comparison.

This is underused and it is the highest-value step in the sequence. It catches the class of failure that only appears on real input distributions.

Cost is double inference on shadowed traffic, so sample rather than shadowing everything.

What to measure live

Prefer metrics that do not require judging quality.

Behavioural. Did the user accept the suggestion, copy the output, edit it heavily, retry, rephrase, or abandon? A retry immediately after a response is a strong negative signal and costs nothing to instrument.

Task completion. Did they finish the thing they came to do? The metric that actually matters, and usually the hardest to attribute.

Follow-up count. Fewer clarifying turns usually means better first answers — though it can also mean the user gave up, so read it alongside abandonment.

Escalation rate. For support systems, requests reaching a human. Clean and unambiguous.

Explicit feedback. Thumbs up and down. Low volume, heavily biased toward dissatisfaction, still useful as a directional signal — and each negative rating with its trace is a free eval case.

Cost and latency. Always. A quality gain that triples cost is a tradeoff needing a decision, not a win.

Design details that matter here

Randomize by user, not by request. Inconsistency within one session confuses people and pollutes the measurement. A user who gets variant A then variant B mid-conversation is experiencing neither.

Hold variants stable for the whole experiment. Editing a prompt mid-test invalidates it. Version both variants and log which produced each response — see Prompt Versioning.

Expect to need more traffic than usual. Response variance is high, so effects are noisier and take longer to reach significance.

Watch the distribution, not just the mean. A change that improves typical responses while making a small fraction much worse can look neutral in aggregate. Segment by input type — the effect is frequently uneven across question categories.

Pin the model version for the duration. A provider updating the model mid-experiment confounds everything, and it happens without notice.

Where the usual approach breaks

Long feedback loops. If value shows up days later — did the code the agent wrote survive review — attribution is genuinely hard. Proxy metrics are the practical answer, chosen carefully.

Novelty effects. Users react to change itself. Run long enough for that to settle.

Interaction with retrieval. Changing a prompt while the RAG corpus is also changing means two variables moved. Freeze what you can.

Small traffic. Many AI features do not have the volume for statistical significance. Then offline evals plus careful human review of shadow output is the honest method, and pretending otherwise is worse than admitting it.

What to remember

• Decide most changes with offline evals; reserve live tests for when evals cannot distinguish variants.
• Shadow mode — run both, serve one, judge offline — is the highest-value step and the most skipped.
• Measure behaviour (retries, edits, acceptance, escalation) rather than trying to score quality live.
• Randomize by user, hold variants and model version stable, and expect to need more traffic than usual.
• Segment by input type; aggregate means hide changes that help most cases and badly hurt a few.

Next: Migrating Between Models