A clumsy sentence in a translated customs declaration costs a re-read. A wrong digit in the HS code costs a rejected shipment. The two failures are separated by orders of magnitude in consequence, and by nothing at all in any quality metric you are likely to be running.

BLEU does not know a number from an adjective. Neither does COMET, and neither does an LLM judging fluency and adequacy — a translation that dropped a company registration code can score 100, because as a piece of language it is fine. If you translate documents, numeric integrity has to be measured separately, deterministically, and reported on its own.

This post is how we do it, including the two ways we got the check itself wrong before it worked.

Why the general metrics cannot help

Quality metrics are built to reward meaning-preserving language. A number is not language. It has no synonyms, no acceptable paraphrase and no partial credit: 2537143416 and 2537143415 are equally wrong and equally fluent.

Worse, the failure is invisible from the output side. A declaration with 27 of its 28 numbers intact reads exactly like one with all 28. Nobody notices until the receiving officer does.

So the check has to be a comparison, not a judgement. Comparisons are cheap, deterministic, and cannot hallucinate a clean bill of health.

The check, and the first way we got it wrong

The naive implementation tokenises both texts and compares the tokens that look numeric. It reports losses that are not there.

The reason is punctuation. In the source a value sits mid-sentence as 2537143416, and in the target it ends a line as 2537143416. Token-wise those are different strings, so the check reports a missing number and a spurious extra one. On a dense form that noise buries the real findings, and a check nobody trusts is a check nobody runs.

The fix is to compare maximal runs of digits, extracted with a regular expression, rather than tokens:

source: "код 2537143416, дата 02.07.2026"
runs:   2537143416, 02, 07, 2026

Strip everything that is not a digit, take the longest uninterrupted sequences, and compare those multisets. Punctuation, quotes, brackets and line breaks stop mattering, which is what you want — none of them are part of the number.

The second way we got it wrong: dates

Once digit runs are compared, dates immediately produce false alarms — because dates are the one class of number a document translation is supposed to change.

20260702 in a Chinese declaration becomes 02.07.2026 in the Russian rendering. That is correct behaviour: the target locale writes dates differently. To a digit-run comparison it is one number disappearing and three appearing.

So dates need their own path: recognise them, normalise both sides to a canonical form, and compare them as dates. Everything else is compared as raw digit runs, where any change is a defect.

The general principle is worth stating, because it generalises past dates: a check must know which transformations are legitimate. A comparison that flags correct behaviour trains people to ignore it, which is worse than not having it.

What the numbers look like on a real document

One full run, a Chinese export customs declaration translated to Russian:


Source numbers tracked	28
Preserved unchanged in the output	26
Deliberately reformatted (a date)	1
Genuinely lost	1

The single real loss is the interesting one. The shipper line in the source read (91330206MAD62XJF0Y)(33029600GQ) — two company codes, not one. The extraction schema declared one field. The first code was kept, the second had nowhere to go, and the output looked complete.

That is the shape of most numeric loss on template-filled documents, and it is worth being precise about it: the model read the value correctly and the form had no box for it. Numeric integrity is therefore not only a translation property. It is a property of the schema and the form, and the check catches all three failure modes at once — misread, mistranslated, or nowhere to put it.

The third way we got it wrong: the ruler itself

The most expensive lesson was not about numbers. It was about the tool measuring them.

We built an evaluation harness to score extraction against ground truth, and it reported 24 missing values on a batch of documents. Not one of them was real.

The harness compared the model's raw answer against the reference. The production path does something the harness did not: it aligns and normalises the payload before validating it — flattening grouped output, mapping names — so values the harness saw as absent were, in production, delivered. The measurement was scoring a pipeline that does not exist.

The lesson, and it applies to any check you write: the ruler has to run the same path as production. A check that skips a normalisation step production performs will report failures nobody can reproduce, and the team will learn to disregard the report — which is exactly when a real defect slips through it.

Where to put the check

Three placements, and they answer different questions:

After extraction, against the source text. Did we read every number on the page? Catches OCR drops and vision misreads.

After translation, source values against target values. Did every number survive translation? Catches the classic mistranslation and the dropped digit.

After rendering, extracted values against the finished document. Did every number reach the output? This is the one that catches "the form had no box", and it is the one most pipelines omit.

If you can only afford one, take the third. It is closest to what the customer receives, and it subsumes the others in the sense that a number lost anywhere upstream is missing here too.

Reporting it so it gets acted on

A numeric check produces a count, and a count is actionable in a way a score is not. Three rules we settled on:

Report it separately from quality. Not folded into a composite score, where a 100 on fluency can mask a lost code. Two numbers, side by side.
Name the values, not just the count. "One number lost" is a metric; "33029600GQ present in the source and absent from the output" is a bug report.
Distinguish reformatted from lost. They have different fixes — one is a locale rule, the other is a defect — and merging them makes the metric useless in both directions.

Key Takeaways

No general quality metric sees numeric loss. A translation missing a customs code reads perfectly and scores accordingly; the check has to be separate and deterministic.
Compare maximal digit runs, not tokens. Attached punctuation makes token comparison report losses that are not there, and a noisy check is an ignored check.
Dates are the legitimate exception and need their own normalised comparison; a check that flags correct behaviour teaches people to ignore it.
Most real loss on forms is "nowhere to put it". The value was read correctly and the target form had no field — so numeric integrity is a property of the schema too.
Make the ruler run production's path. Ours reported 24 missing values, all of them artefacts of a normalisation step the harness skipped and production performs.

FAQ

Why doesn't BLEU or COMET catch a wrong number?

Because they measure meaning-preserving language, and a number is not language. It has no synonyms and no partial credit — a digit changed is as fluent as a digit unchanged, so the metric sees nothing.

How should numeric integrity actually be checked?

By comparing maximal runs of digits between the source and the target as multisets, with dates recognised and normalised separately because they legitimately change form. It is a deterministic comparison, costs nothing, and produces a count plus the specific values.

Why compare digit runs instead of tokens?

Because punctuation attaches to numbers. 2537143416, and 2537143416 are different tokens and the same number, so a token comparison reports a false loss and a false addition for every number near a comma. That noise buries real findings.

What if a number is legitimately reformatted?

Handle dates on their own path — recognise, normalise both sides, compare as dates — and treat everything else as exact. Report reformatted and lost separately, because one is a locale rule working correctly and the other is a defect.

Where in the pipeline should the check run?

Ideally three times: after extraction, after translation, and after rendering. If you run it once, run it last — comparing the extracted values against the finished document catches everything upstream plus the case where the target form had no field for a value.

Conclusion

Numeric integrity is the rare quality property that is both cheap to measure and expensive to get wrong, which makes it the first thing to instrument on any document pipeline. It is a comparison, not a judgement: no model call, no threshold, no correlation study.

The hard part turned out not to be the check but the discipline around it — comparing the right thing, allowing the transformations that are legitimate, and making sure the measurement runs the same path as production. Get those wrong and you produce a report full of phantom failures, which is how a real one goes unnoticed.

If you translate declarations, certificates or registry extracts and want numeric integrity reported as its own number, try KTTC.