Your Prompt Isn't Working. Now What?

Four distinct failure types wear the same costume. Diagnosing which one you have saves the hours usually spent rewording a prompt that was never the problem.

The instinct when output is wrong is to reword the prompt. Frequently that is the wrong move, because the prompt was not the problem.

Four failure types look identical from the outside. Each has a different fix, and rewording only helps one of them.

The four types

Type 1 · The task was underspecified

Most common by far. The model did something reasonable; you wanted something else and never said so.

Diagnostic: could a competent contractor, given only this prompt, have produced what you wanted? If they would have had to guess, the model guessed.

Fix: specify the missing dimension — length, format, audience, focus, what to omit. Or show an example, which resolves ambiguities you cannot enumerate.

Type 2 · The information was not there

The model cannot know your codebase, your pricing, or last week's decision. Asked anyway, it produces something plausible and wrong.

Diagnostic: does answering correctly require information not in the prompt and not in general public knowledge? Also: does the answer look confidently specific about something the model has no way to know?

Fix: supply the information, or retrieve it. No prompt wording extracts facts the model does not have.

Type 3 · The reasoning had nowhere to happen

Multi-step problem, single-token answer. One forward pass for five steps of work.

Diagnostic: would you need scratch paper? If yes, so does the model.

Fix: ask for the steps before the answer.

Type 4 · The capability is not there

Some tasks are past a given model's ability, and no prompt fixes that.

Diagnostic: try a stronger model with the identical prompt. If it succeeds, this was Type 4. This single test is the fastest diagnostic available, and it is underused.

Fix: stronger model, decompose into easier subtasks, or give it a tool — arithmetic and lookup failures are usually better solved with a calculator or a search call than with better wording.

The diagnostic sequence

In order, because each step is cheap and rules out a category:

1 · Read the output as if it were correct. What question would make this the right answer? The gap between that question and yours is your specification gap.

2 · Check for invented specifics. Confident detail about things the model cannot know means Type 2.

3 · Ask it to reason first. If accuracy improves, it was Type 3.

4 · Try a stronger model, same prompt. If that fixes it, Type 4 — stop rewording.

5 · Run it five times. High variance means an under-constrained prompt; five identically wrong answers mean a systematic misunderstanding. These need different fixes: constrain harder versus re-specify.

6 · Cut the prompt in half. Long prompts hide contradictions, and instructions in the middle get followed less reliably. Removing material sometimes fixes things, which is diagnostic in itself.

Failures with mechanical causes

Some symptoms map directly to mechanics rather than to any of the four types.

It ignored an instruction buried mid-prompt. Middle-of-context instructions are followed less reliably. Move it to the start or end. See Why Long Contexts Cost So Much.

It forgot something from earlier in the conversation. Probably context truncation rather than a model failure. Check whether your framework is silently dropping history.

Output stopped mid-sentence. Token limit, not a prompt issue. Raise the limit or ask for less.

Same prompt, different answers each run. Temperature. Set it to 0 for extraction and classification.

It followed the example's content instead of its pattern. Your examples are too close to the real input. Make them clearly distinct.

It obeyed text from inside your input data. Content containing instruction-shaped text got read as instruction. Delimit input clearly and state that content inside delimiters is data.

Two habits worth having

Change one thing at a time. Rewriting a whole prompt teaches you nothing about which change helped.

Keep failing examples. Three or four saved failures become a regression set — and that is the beginning of an eval, which is the only way to know whether a prompt change actually improved anything rather than shifting the failures around.

What to remember

Four types: underspecified, missing information, no reasoning space, insufficient capability. Rewording only fixes the first.
Read the output as if correct — the implied question reveals your specification gap.
Trying a stronger model with the identical prompt is the fastest way to rule out capability.
High variance means under-constrained; consistently wrong means misunderstood.
Truncation, token limits, temperature, and mid-prompt placement cause symptoms that look like prompt problems.

Next: Why RAG Exists

The instinct when output is wrong is to reword the prompt. Frequently that is the wrong move, because the prompt was not the problem.

Four failure types look identical from the outside. Each has a different fix, and rewording only helps one of them.

The four types

Type 1 · The task was underspecified

Most common by far. The model did something reasonable; you wanted something else and never said so.

Diagnostic: could a competent contractor, given only this prompt, have produced what you wanted? If they would have had to guess, the model guessed.

Fix: specify the missing dimension — length, format, audience, focus, what to omit. Or show an example, which resolves ambiguities you cannot enumerate.

Type 2 · The information was not there

The model cannot know your codebase, your pricing, or last week's decision. Asked anyway, it produces something plausible and wrong.

Fix: supply the information, or retrieve it. No prompt wording extracts facts the model does not have.

Type 3 · The reasoning had nowhere to happen

Multi-step problem, single-token answer. One forward pass for five steps of work.

Diagnostic: would you need scratch paper? If yes, so does the model.

Fix: ask for the steps before the answer.

Type 4 · The capability is not there

Some tasks are past a given model's ability, and no prompt fixes that.

Diagnostic: try a stronger model with the identical prompt. If it succeeds, this was Type 4. This single test is the fastest diagnostic available, and it is underused.

Fix: stronger model, decompose into easier subtasks, or give it a tool — arithmetic and lookup failures are usually better solved with a calculator or a search call than with better wording.

The diagnostic sequence

In order, because each step is cheap and rules out a category:

1 · Read the output as if it were correct. What question would make this the right answer? The gap between that question and yours is your specification gap.

2 · Check for invented specifics. Confident detail about things the model cannot know means Type 2.

3 · Ask it to reason first. If accuracy improves, it was Type 3.

4 · Try a stronger model, same prompt. If that fixes it, Type 4 — stop rewording.

6 · Cut the prompt in half. Long prompts hide contradictions, and instructions in the middle get followed less reliably. Removing material sometimes fixes things, which is diagnostic in itself.

Failures with mechanical causes

Some symptoms map directly to mechanics rather than to any of the four types.

It ignored an instruction buried mid-prompt. Middle-of-context instructions are followed less reliably. Move it to the start or end. See Why Long Contexts Cost So Much.

It forgot something from earlier in the conversation. Probably context truncation rather than a model failure. Check whether your framework is silently dropping history.

Output stopped mid-sentence. Token limit, not a prompt issue. Raise the limit or ask for less.

Same prompt, different answers each run. Temperature. Set it to 0 for extraction and classification.

It followed the example's content instead of its pattern. Your examples are too close to the real input. Make them clearly distinct.

It obeyed text from inside your input data. Content containing instruction-shaped text got read as instruction. Delimit input clearly and state that content inside delimiters is data.

Two habits worth having

Change one thing at a time. Rewriting a whole prompt teaches you nothing about which change helped.

What to remember

Four types: underspecified, missing information, no reasoning space, insufficient capability. Rewording only fixes the first.
Read the output as if correct — the implied question reveals your specification gap.
Trying a stronger model with the identical prompt is the fastest way to rule out capability.
High variance means under-constrained; consistently wrong means misunderstood.
Truncation, token limits, temperature, and mid-prompt placement cause symptoms that look like prompt problems.

Next: Why RAG Exists