Document and OCR Understanding
Extracting text is solved. Extracting structure is not, and structure is usually what you actually need.
On this page
Getting characters off a page is largely a solved problem. Getting the document off the page is not.
The gap matters because almost every real task needs structure, not characters. Which cell of which table. Which heading this paragraph sits under. Which of the three amounts on the invoice is the total. Reading order across two columns.
This is the stage where most RAG pipelines quietly lose their quality, before chunking ever runs.
Three approaches
Traditional OCR detects text regions and recognizes characters, outputting text plus bounding boxes. Fast, cheap, mature, and excellent at what it does. It gives you words and positions, and no understanding of what the document is.
Layout-aware models take OCR output plus coordinates and classify structure — this block is a heading, these cells form a table, this is a form field with that value. Purpose-built for the structure problem and generally the most reliable option for high-volume repetitive documents.
Vision-language models process the page image directly and answer questions about it, or emit structured output. Enormously more flexible: you can ask for exactly the fields you want in the shape you want, with no per-document-type engineering. See Vision-Language Models.
The tradeoff is honest. VLMs are far more capable and far less predictable. They can also hallucinate values that resemble what should be there — a genuinely dangerous failure on financial documents, and one that OCR cannot make because OCR does not generate.
Resolution is the first thing to check
A vision model resizes before it sees anything. Small text disappears in that step.
So “the model cannot read this document” is usually a preprocessing problem, not a comprehension problem, and the fixes are mechanical:
- Send higher resolution, if the provider accepts it
- Crop to the region of interest rather than the full page
- Split dense pages into sections and process each
- For dense text, run OCR and pass the text — do not make the vision model do character recognition it will do worse
That last point is worth emphasizing. Combining OCR text with the page image often beats either alone: OCR provides reliable characters, the image provides layout the text stream lost.
Tables
Tables are the hardest common case, and worth calling out specifically.
A table is a 2D structure flattened into a 1D text stream, and the flattening destroys the thing you needed. Merged cells, multi-row headers, tables spanning pages, and borderless tables held together only by whitespace alignment all break naive extraction.
What works, roughly in order of reliability: dedicated table-detection models that output cell grids; asking a VLM for a structured format directly — markdown or JSON — since it can reason about layout in a way a text stream cannot; and passing both OCR text and the image so alignment information survives.
What does not work is extracting a table as plain text and hoping. It will look approximately right and be wrong in ways that are hard to detect.
PDFs specifically
PDFs are not a document format so much as a page-drawing format, and this causes concrete problems.
Digital PDFs contain extractable text — no OCR needed. Always check for this first, since extraction is faster and more accurate than recognition. But the text is stored as positioned fragments, not paragraphs, so reading order is a reconstruction and multi-column layouts frequently interleave.
Scanned PDFs are images and need OCR.
Mixed PDFs contain both, sometimes on the same page. Detect per page rather than per document.
Two-column academic papers are the canonical failure: naive extraction reads across columns, producing text that is locally fluent and globally scrambled. Nothing downstream can recover from that.
Validate the output
Extraction failures are quiet. Text comes out, it looks plausible, and it is wrong. A few cheap checks catch most of it:
Arithmetic. If line items should sum to a total, verify it. This single check catches a large share of table extraction errors.
Format validation. Dates, currencies, and identifiers against expected patterns.
Confidence scores where OCR provides them — flag low-confidence regions for review rather than accepting silently.
Spot-check reading order on a sample of multi-column documents. Do not assume.
For anything consequential, keep a human in the path on low-confidence extractions. See Human-in-the-Loop Design.
What to remember
- Character extraction is solved; structure extraction is not, and structure is what you need.
- Three approaches: OCR (reliable, no understanding), layout models (best for repetitive high volume), VLMs (flexible, can hallucinate values).
- Resolution and cropping fix most “cannot read it” problems; combining OCR text with the page image beats either alone.
- Tables are the hardest case — request structured output, never plain text.
- Check for extractable text in PDFs first; multi-column reading order is a common silent failure.
- Validate with arithmetic and format checks, because extraction fails quietly.