Vision-Language Models
Put image patches and text tokens in one sequence and attention does the rest. How a language model gains eyes, and what it still cannot do.
On this page
Once an image is a sequence of patch tokens and text is a sequence of text tokens, a single obvious idea presents itself: put them in the same sequence.
That is essentially what a vision-language model is. Attention operates across both, so a text token can attend to an image patch and vice versa.
The standard architecture
Three components, and the middle one is where the interesting work happens.
A vision encoder turns the image into patch embeddings. Usually a pretrained image model, often frozen.
A projection layer maps those embeddings into the language model’s embedding space. The vision encoder was trained separately and produces vectors in its own space, which the language model would not understand. A small learned adapter — sometimes just a linear layer — bridges the two.
A language model receives the projected image vectors as if they were token embeddings, interleaved with the actual text.
So the prompt “What is in this image?” becomes a sequence like [img_1] [img_2] ... [img_196] What is in this image?, and the language model does what it always does: predict the next token.
The elegance is that the language model needs almost no modification. It already accepts a sequence of vectors. Nothing requires those vectors to have come from text.
Why training is staged
Training all three parts from scratch would be enormously expensive and largely redundant — good vision encoders and good language models already exist.
The usual sequence:
- Start with a pretrained vision encoder and a pretrained language model.
- Train only the projection layer on image-caption pairs, with both large models frozen. Cheap, and it teaches the adapter to speak the language model’s dialect.
- Optionally unfreeze parts and fine-tune on instruction data — images paired with questions and good answers.
This is why vision capability could be added to existing language models relatively quickly. The hard parts were already trained; only the bridge was new.
What they do well
Description and captioning. The task the training data most directly rewards.
Question answering about content. What is happening, what is unusual, what is the mood.
Reading substantial text. Documents, slides, signs — provided the text is large enough to survive resizing. See Document and OCR Understanding.
Interpreting charts and diagrams. Often surprisingly good at trend and structure, less reliable on exact values.
Code from mockups. Screenshot to markup works because both sides were well represented in training.
What they do badly
The failures cluster, and most trace back to the patch representation.
Counting. Especially many small similar objects. The patch grid does not individuate things, and there is no counting mechanism.
Precise spatial relationships. Rough layout is fine; exact offsets and alignments are not.
Small text and fine detail. Frequently destroyed by resizing before the model sees anything. This is a preprocessing loss, not a comprehension failure — and it explains why the same image at higher resolution can work.
Exact values from charts. It reads the shape, then estimates the numbers. Treat quoted figures as approximate.
Negation and absence. “Is there no cat in this image?” is unreliable, for the same reason negation confuses embeddings.
Practical notes
Resolution matters more than anything else. If detail is being missed, send a larger or cropped image before trying prompt changes. Many “the model cannot read this” problems are resizing problems.
Crop to what matters. A full screenshot spends its patch budget on irrelevant chrome. Cropping to the region of interest is the single most effective intervention available.
Images cost tokens. A high-resolution image can consume more context than several pages of text, and providers bill accordingly. Multiple images multiply this.
Ask for reasoning first. Step-by-step helps on visual reasoning for the same mechanical reason it helps on text.
Hallucination applies. A model asked about something not in the image will frequently describe it anyway. Every dynamic from Why Do AI Models Hallucinate? carries over — grounding in an image reduces invention without eliminating it.
What to remember
- Vision-language models put image patches and text tokens in one sequence, so attention works across both.
- Architecture is vision encoder + projection layer + language model; staged training means only the bridge is trained first.
- Strong at description, question answering, large text, and chart structure.
- Weak at counting, precise spatial relations, small detail, exact chart values, and negation.
- Resolution and cropping matter more than prompting; images consume substantial context.