How Do Models See Images?
An image becomes a sequence of patch tokens, and from there the machinery is the same one that reads text.
On this page
A transformer operates on a sequence of vectors. Text becomes that sequence through tokenization. Images need their own route in.
The trick that made vision transformers work is almost disappointingly simple: cut the image into a grid of squares and treat each square as a token.
Patches as tokens
Take a 224×224 image. Divide it into 16×16 pixel patches. That gives 14×14 = 196 patches.
Flatten each patch into a list of numbers — 16 × 16 × 3 colour channels = 768 values — and project it through a learned linear layer into an embedding. You now have 196 vectors.
That is a sequence. Attention can run on it unchanged. A patch of sky attends to other patches of sky; a patch containing an eye attends to the patch containing the other eye.
Two consequences follow immediately.
Patch count scales with resolution. Double the image dimensions and you quadruple the patches. Since attention cost is quadratic in sequence length, high-resolution images get expensive fast. This is why vision models resize aggressively, and why fine detail — small text in a screenshot — is often lost before the model sees anything.
Position must be injected, exactly as in text. Attention is order-blind, so without positional encoding the model would see a bag of unordered patches. Vision models add 2D position information so “above” and “to the left of” are recoverable.
Why patches rather than pixels
One token per pixel would give 50,176 tokens for a small image, and quadratic attention makes that unworkable.
One token per whole image loses all spatial structure — you could not ask where anything is.
Patches land in between: few enough for attention to handle, small enough that spatial relationships survive. The same compromise subword tokenization makes for text, for the same reason.
What convolutions did differently
Before vision transformers, image models were convolutional. A small filter slid across the image detecting local patterns, and stacked layers built up from edges to textures to objects.
Convolutions bake in two assumptions: nearby pixels are related, and a feature means the same thing wherever it appears. Both are true of images, which is why CNNs worked well with relatively little data.
Transformers assume neither. Any patch can attend to any patch from layer one — no built-in notion of locality. That flexibility costs data: vision transformers need far more training images than CNNs to reach the same accuracy, because they must learn that nearby patches are related rather than being told.
At sufficient scale they overtake CNNs, which is the same pattern seen throughout deep learning: fewer built-in assumptions wins once data is abundant.
Resolution handling
Fixed patch grids create a real problem: images come in every size and aspect ratio, and a model trained at 224×224 has positional embeddings for exactly 196 patches.
Approaches in current use:
Resize and pad. Simplest, and it destroys detail in large images while distorting unusual aspect ratios.
Tiling. Split a large image into several model-sized crops, process each, and combine. Preserves detail at the cost of more tokens and a lost global view — often paired with one downscaled full-image pass to recover overall layout.
Dynamic resolution. Interpolate positional embeddings to accept variable patch counts. More flexible, and increasingly standard.
Knowing which one you are dealing with explains a lot of observed behaviour. A model that cannot read small text in a screenshot is usually resizing it away, not failing to understand text.
What this enables
Once images are token sequences, they can be mixed with text tokens in the same context. That is the whole basis of vision-language models — one sequence containing both, with attention operating across the boundary.
It also explains a limitation worth knowing: the model sees a grid of patches, not objects. Counting many small items, reading tiny text, and judging precise spatial offsets are all genuinely hard, because the patch grid is a coarse representation and fine detail was discarded before the first layer.
What to remember
- Images become tokens by cutting them into patches — typically 16×16 pixels — each projected into an embedding.
- Patch count grows with the square of resolution, and attention is quadratic in patch count, so detail is expensive.
- 2D positional encoding is required, since attention is order-blind.
- Convolutions assume locality and get by on less data; transformers assume nothing and win at scale.
- Fine detail is often lost to resizing before the model sees anything.
Next: Vision-Language Models