Prompt Versioning

Prompts are code that changes behaviour. Treat them accordingly, and keep the eval result attached to the version.

A prompt determines what your system does. Changing one changes behaviour, sometimes substantially, with no compile error and no type check.

That makes prompts code. The practices are the same ones you already use, plus two that are specific to this material.

In version control, with the code

Prompts belong in the repository, in files, reviewed like anything else.

The alternative — prompts in a database or an admin UI, editable at runtime — sounds appealing and creates real problems. Behaviour changes without a deploy, without review, and without a record. Debugging a past incident becomes impossible because you cannot reconstruct what the prompt was.

If non-engineers must edit prompts, keep the editing interface but back it with version history, review, and the ability to see exactly what was live at a given time.

Templates separate from data. Keep the prompt structure in a file with placeholders, and inject variables at runtime. Prompts assembled by string concatenation across several functions cannot be reviewed as a unit.

Attach evals to versions

The practice that distinguishes prompt management from ordinary file management: a prompt version without an eval result is unverified.

Every change should record what it scored on your eval set. Without that, you have a history of edits and no idea which ones helped.

This is what makes the rest work. Rollback becomes meaningful because you know the previous version’s score. Comparison becomes possible. And regressions become visible instead of being discovered by users.

Practically: run the eval in CI on prompt changes, and store results alongside the version. Treat a score drop as you would a failing test.

Version identifiers in logs

Log which prompt version produced each response. See Observability for LLM Systems.

Without it, investigating a bad response from last week means guessing at what the prompt was — and prompts change more often than code does in active development.

A content hash works and requires no discipline to maintain.

Model version is part of the configuration

A prompt is tuned against a specific model. The same prompt on a different model behaves differently, sometimes badly.

So the deployable unit is the pair: prompt version plus model version. Log both, and re-run evals when either changes.

The awkward part is that hosted providers update models under stable version names, so your model can change without any action on your side. Periodic eval runs are the only way to detect it — which makes scheduled eval runs worth setting up even when nothing is changing.

Rolling out changes

For anything consequential, do not swap a prompt globally in one step.

Shadow first. Run the new prompt alongside the old on real traffic, log both, serve the old. Free comparison on real inputs, no user risk.

Then a small percentage of live traffic, with metrics compared. See A/B Testing AI Features.

Keep rollback trivial. A config change, not a deploy.

Organizing prompts

One file per prompt, named for its purpose.

Extract shared fragments — output-format instructions, safety boilerplate, common context. Duplicated fragments drift, and drift produces inconsistency nobody intended.

Comment the non-obvious constraints. Prompts accumulate lines added to fix specific failures, and six months later nobody knows which line prevents which problem. A comment saying added because it invented citations without this prevents someone deleting it as redundant.

That comment habit is worth more than it sounds. Undocumented prompt lines get removed during cleanup, and the bug returns.

What to remember

• Prompts are code: version control, review, files with placeholders rather than scattered concatenation.
• Attach an eval result to every version — an unverified prompt version is the core problem this solves.
• Log the prompt version with every response, and treat prompt plus model version as one deployable unit.
• Hosted models change silently; run evals on a schedule, not only on change.
• Roll out by shadowing, then a small traffic percentage, with trivial rollback.
• Comment why a constraint exists, or it will be deleted and the bug will return.

Next: A/B Testing AI Features