How Do You Know It Works?

Without evals you are tuning prompts by vibes. A dozen saved examples is enough to start, and it changes how you work.

You change a prompt. Output looks better on the example you were staring at. You ship.

You have no idea whether you improved the system or moved the failures somewhere you were not looking. This is the default state of most LLM work, and it is why an eval — a repeatable test over fixed examples — is the highest-leverage engineering investment available.

Why intuition fails here

Three reasons, all specific to this technology.

Nondeterminism. Sampling means the same prompt gives different answers. One good run proves nothing.

Prompt changes have side effects. Adding a constraint to fix case A frequently breaks case B. Without a fixed set, you never see B move.

Fluency masks error. Output reads well whether or not it is correct. Skimming is not evaluation, and confident wrongness is the specific failure that skimming misses.

Add silent model updates and drifting input distributions, and unmeasured systems degrade without anyone noticing.

Start with twelve examples

An eval does not require infrastructure. It requires saved examples.

Keep a file: input, and what a good output looks like. Ten or twenty cases covering your real distribution plus the failures you have hit. Run it after every prompt change.

That is a real eval. Most of the value arrives at this scale, and the file grows naturally — every production failure becomes a test case. That habit alone changes the trajectory of a project.

Scoring

The hard part is deciding what “correct” means, and it varies by task type.

Exact match. Classification, extraction, structured fields. Trivially automatable and unambiguous. Prefer this shape wherever you can arrange for it.

Contains / must-not-contain. Required facts present, forbidden content absent. Crude and often sufficient.

Schema validity. Does it parse, do fields conform. Automatable, and covers a real class of failure. See structured output.

Human judgment. Slow, expensive, and still the ground truth for anything subjective. A handful of carefully reviewed cases beats a hundred sloppily scored ones.

Model-as-judge. Have a model score the output against criteria. Scales to volume, and needs care: judges favor verbose answers, favor outputs resembling their own style, and cluster scores in a narrow band. Validate the judge against human labels on a sample before trusting it, and prefer pairwise comparison — “which is better” — over absolute scoring, which is more consistent.

For agents, score the outcome, not the trajectory. Many valid paths reach the same result, and grading the path punishes legitimate variation.

What to measure

Beyond correctness:

Consistency. Run each case several times. High variance means an under-constrained prompt, and that is actionable in a different way than a wrong answer.

Failure mode distribution. Not just how many failed, but how. Ten format errors and ten factual errors need entirely different fixes.

Retrieval quality separately, for RAG systems. Was the correct chunk in the retrieved set? Diagnosing this independently of generation is essential, because most RAG failures are retrieval failures and no prompt change touches them.

Cost and latency. A prompt that improves accuracy while tripling cost is a tradeoff, not a win — and you cannot see it without the numbers.

Working with an eval

Baseline before changing anything. Otherwise you cannot tell direction.

One change at a time. Two simultaneous changes with a net improvement teaches nothing about which helped.

Hold out cases. Tuning against all your examples overfits to them. Reserve some you do not look at while iterating.

Re-run on model updates. Provider models change under stable version names. The eval is how you find out.

Accept regressions knowingly. Sometimes a change fixes five cases and breaks one. That is fine — as an explicit decision, not an accident.

What not to do

Do not lean on public benchmarks. They measure general capability, not your task. Useful for narrowing model candidates, useless for validating your system.

Do not build infrastructure first. A file of examples and a script beats a framework you spend a week on.

Do not chase a single number. One aggregate score hides which failure modes moved.

What to remember

• Nondeterminism, cross-case side effects, and fluent wrongness make intuition unreliable — you need fixed examples.
• Twelve saved cases is a real eval. Every production failure becomes a test case.
• Prefer automatable scoring; use model-as-judge with validation and pairwise comparison.
• Measure consistency, failure mode distribution, retrieval quality separately, and cost.
• Baseline first, change one thing at a time, hold out cases, and re-run when models update.

Next: From Demo to Production