There are two ways to get structured fields out of a scanned document. The familiar one: run OCR, get text, hand the text to a language model. The newer one: hand the page image to a multimodal model and skip the text stage entirely.

The literature has started to favour the second — a 2025 comparison (arXiv:2509.04469) found multimodal models often beating the OCR→text chain on invoices, because table structure and the position of a signature carry meaning that OCR discards. We run both paths in the same product, chosen per document type, and our own measurements are less one-sided than that. This post is what we found, including the part that cost us the most: the check that decides which path a document takes is easy to get wrong, and wrong in a way that looks fine.

What each path actually loses

The distinction is not "accuracy". Both paths read characters well. The distinction is what happens to everything that is not characters.

OCR flattens a page into a text stream. A table becomes a sequence of cell contents; which column a value sat in becomes an inference from ordering. A signature becomes nothing, or a garbage token. Two boxes side by side become one line. Everything the layout was telling you has to be reconstructed by the model from word order.

The vision path keeps the page. The model sees that a value is in the third column of a table, that a stamp overlaps a field, that a box is empty rather than absent. For a form — and an official document is a form — that is a lot of signal.

What the vision path costs is tokens. A page rendered at useful resolution is expensive in a way a page of text is not, and that cost scales with how many pages you send.

Where we measured no difference at all

Our cadastre extracts — Russian property registry documents, five pages, issued electronically with a real text layer — used to be forced down the vision path by a flag.

We ran both branches over the same file and compared leaf by leaf: 48 values, zero differences. The cadastral number, the cadastral value, the area, the address, and both 32-character signature certificate hashes — that last pair being exactly the fine print the flag existed to read — all matched.

So we dropped the flag. Two facts made that safe rather than merely cheaper:

A scanned extract still goes to vision. The router asks the PDF whether it carries a text layer of its own, which is the direct question.
The image budget never covered the document anyway. On a five-page extract, the per-request image allowance only ever reached pages 1–2. Pages 3–5 — the rights and encumbrances — arrived as text whether the flag was set or not.

That second point is worth pausing on. We had been paying for vision and receiving, for most of the document, exactly the text path.

Where the difference was decisive

The same comparison on a Chinese customs declaration went the other way, and not by a small margin. On the text branch:

both company codes vanished from the shipper field — the line contains (...)(...), two codes, and OCR's flattening left one;
an export date was invented for a cell that was blank on the page.

That second failure is the one that should decide your architecture. A missing value is a gap you can see. A plausible value for an empty box is a defect that survives every check you have, because nothing about it looks wrong. The vision path did not do this, because it could see that the box was empty.

Dense tabular forms are where vision earns its cost: many small cells, positional meaning, and empty boxes that must stay empty.

The probe that lies

Here is the finding that cost us the most, and the reason this section exists.

The natural way to route is: does this PDF have a text layer? If yes, text path; if no, vision. It is cheap, local, and correct — until a document arrives that has a text layer somebody else put there.

We hit one: a marriage certificate scan carrying an embedded OCR layer from whatever tool the sender had used. Six hundred characters of confident mojibake — Cyrillic mixed with stray Korean and Chinese glyphs, words fragmented into nonsense. To a has_text_layer check, that document looks like a clean digital PDF. It is a photograph with a bad transcription stapled to it.

Two lessons:

"Has a text layer" is not "has usable text." Score the text you found — script consistency, dictionary hit rate, ratio of alphabetic to garbage — before trusting it.
The cheapest check is the one most likely to be answering a different question than you think. Ours was answering "did anyone ever run OCR on this" and being read as "is this a digital document".

Cost, and the part that surprised us

Vision cost is dominated by how you render pages, not by the model. We had been sending each page as a set of tiles plus an overview — the standard trick for letting a model read fine print.

Turning tiling off and sending a single higher-quality overview per page cut image tokens by 89% on a customs declaration and 73% on a company registry extract, while increasing coverage from two pages to five, because the budget went further. Same bill, more document.

The general shape:

	OCR → text	Page → vision
Cost driver	Pages through the OCR service	Image tokens, set by render profile
Loses	Layout, table structure, emptiness	Nothing structural
Fails by	Dropping values, merging cells	Cost, and slower responses
Best on	Prose, long documents, digital PDFs	Dense forms, scans, tables

How we route, in practice

Three questions, in this order:

Does the file carry its own text layer, and is that text plausible? Both halves. The second is what we learned to add.
Is this document type dense and tabular? Customs declarations and multi-column forms go to vision regardless, because the failure mode of the text path on them is invention rather than omission.
Does the budget cover the document? If your image allowance reaches page 2 of 5, you are not running the vision path on that document — you are running a hybrid and calling it vision. Either raise the budget, change the render profile, or admit which path you are on.

Everything else takes the text path, because it is cheaper and, where we measured it, indistinguishable.

What we would tell someone choosing today

Do not pick a path for the whole pipeline. Pick per document type, and pick with a comparison you ran yourself on your own documents — leaf by leaf, not by an aggregate score. Ours took an afternoon and changed two decisions, one in each direction.

And measure emptiness. Every quality metric we know about rewards producing a plausible value. The failure that matters most on forms is producing one where the page had nothing.

Key Takeaways

The choice is about layout, not accuracy. OCR flattens tables, positions and emptiness; vision keeps them. Both read characters fine.
On a digital extract we measured 48 values with zero differences between the two paths — the vision flag there was pure cost.
On a dense declaration the text path invented a date for an empty cell. A plausible value where the page had none is the failure no downstream check catches.
"Has a text layer" is not "has usable text." A foreign OCR layer of mojibake passes that probe and routes a photograph down the text path.
Render profile dominates vision cost. Dropping tiling cut image tokens by 89% and 73% on two document types while covering more pages, not fewer.

FAQ

Is a vision model always better than OCR for document extraction?

No. On documents with a genuine text layer and simple structure we measured no difference at all across 48 extracted values — the vision path was pure added cost. It wins decisively on dense tabular forms and on scans, where layout carries meaning and OCR flattens it away.

How do I decide which path a document should take?

Ask three questions in order: does the file carry plausible text of its own, is the document type dense and tabular, and does your image budget actually cover the whole document. The third is routinely skipped and quietly turns "vision" into a hybrid.

What is the biggest risk of the OCR-then-LLM path?

Invention. When OCR flattens a form, an empty cell can disappear rather than arrive as empty, and the model fills the gap with something plausible. We saw exactly this — a fabricated export date on a blank cell — and a fabricated value passes every check that looks for missing data.

Why did dropping image tiles reduce cost without hurting quality?

Because the tiles were spending the per-request image budget on the first pages instead of the document. A single higher-quality overview per page cut image tokens by 73–89% and let the budget reach page five instead of page two.

Can I trust a PDF's text layer?

Only after scoring it. A scan can carry an OCR layer added by someone else's tool, full of mojibake, and it will satisfy any "does this have text" check. Look at script consistency and the ratio of real words to garbage before routing on it.

Conclusion

The interesting question was never "OCR or vision". It is which parts of a page carry meaning that is not characters, and whether your pipeline can tell the difference between a document it can read and one it merely has text for.

Run the comparison on your own documents, compare values rather than scores, and check what your image budget really covers. In our case that produced two opposite decisions on two document types — which is the honest outcome, and the one a single blanket choice would have hidden.

If you want to see field-level extraction with the routing decision made per document type, try KTTC.