Judging a Translation With No Reference: What Comes After GEMBA-MQM

In production there is no reference translation. There is a source, a target, and a decision about whether the target is good enough to ship. Every metric built on comparing against a human translation — BLEU, chrF, COMET in its reference-based form — is unavailable at exactly the moment you need a number.

Reference-free quality estimation is the answer, and it is now good enough to act on. This post covers what changed in 2025–2026, what is reproducible on paid APIs versus what needs your own infrastructure, and the failure we hit ourselves: quality estimation that returned a plausible number for months while doing nothing at all.

Where GEMBA-MQM left things

GEMBA-MQM asks a language model to play MQM annotator: find the errors in this translation, classify each by category and severity, and let the score fall out of the weighted error count. It works because the model is doing the thing annotators do, rather than estimating a similarity.

Its real virtue is that the output is auditable. A number without an error list is an opinion; GEMBA-MQM gives you "major accuracy error at this span", which a reviewer can agree or disagree with. For document translation that matters more than a decimal place of correlation, because a reviewer is the next step regardless.

What 2025–2026 added

GEMBA V2 (WMT25) keeps the prompting and attacks the variance. A single LLM judgement is noisy — run it twice, get two scores. V2 samples the judgement about ten times and aggregates with a rank-based weighted average rather than a mean, so one wild sample cannot drag the result. Better correlation, at roughly ten times the cost.

MetricX-25 is a trained metric rather than a prompted one, and it scores better on the benchmarks. The catch is operational: it is a model you host, with the GPU, the deployment and the version management that implies.

The practical split, if you run on paid APIs and do not want to operate a model:

Our own conclusion: V2's sampling is worth it selectively, on segments a cheaper tier already flagged, and not as a default — because ten judgements of a segment nobody was going to change is ten wasted calls.

The two-tier setup that actually pays

The pattern we run, and would recommend:

Tier 1: rule-based, on everything. Length ratios, untranslated-source detection, number and glossary-term presence, script consistency. Costs nothing, runs in milliseconds, and catches the defects that matter most on official documents — a dropped number is not a style question.

Tier 2: LLM judgement, on what tier 1 flags. GEMBA-MQM on the segments the cheap tier scored below a threshold.

The arithmetic is the whole argument. If tier 1 clears 90% of segments, you pay for LLM judgement on 10% of the volume. Whether that is the right threshold is an empirical question about your content, and it is the single most consequential parameter in the setup.

The failure that makes QE look like it works

Now the part worth the price of admission.

Our GEMBA-MQM tier called the model with a token budget of max(4096, n * 150) — that is, exactly 4096 for any batch under 27 segments, which is most batches. The provider had moved to a reasoning model. A reasoning model spends the same budget on thinking as on answering, so the response came back with finish_reason=length, completion=4096, reasoning=4096, and no content at all.

The empty response went to a fallback that returns a flat score of 75, tagged estimator="fallback".

So quality estimation returned 75. Every time. For months. Nothing errored, nothing alerted, and the number was plausible enough that nobody looked — 75 out of 100 is exactly what a QE system is expected to produce.

The database told the story once we asked it: 1,581 segments scored rule_based, 70 scored gemba-mqm, 6 scored fallback. The LLM tier had all but stopped running, and the shape of those counts was the only evidence.

After the fix — one flag telling the model not to think — on 14 machine-translated segments of a real property registry extract:

Two lessons, and the second is the general one.

A fallback that returns a plausible value is worse than one that raises. If your QE tier cannot run, you want to know. A number meaning "we did not measure this" must not be shaped like a number meaning "we measured this and it was fine".

Tag every score with the estimator that produced it. That one field turned an invisible outage into a five-minute diagnosis.

What the same measurement said about the cheap tier

A side finding, and a useful one. The rule-based tier scored those same 14 segments between 75 and 85 — and let none of them through to the LLM.

The LLM then scored thirteen of them at 100.

So the prefilter is conservative, not wrong: it under-rates good translations but does not pass bad ones. Nothing defective was shipping. Two consequences followed anyway: users saw pessimistic scores on good work, and the expensive tier never got a word in — which is how the outage above stayed hidden.

Calibrating a prefilter is not about accuracy in the abstract. It is about two asymmetric costs: a false "bad" wastes an LLM call and depresses a displayed score; a false "good" ships a defect. Set the threshold with those costs in front of you, and measure what fraction escalates. If nothing escalates, your second tier is decorative.

What to measure, if you are starting

1. The escalation rate. What fraction of segments reach the expensive tier? If it is 0% or 100%, the threshold is doing nothing.
2. The estimator distribution. How many scores came from each tier, fallbacks included. This is the health check that would have caught our outage on day one.
3. Agreement on the flagged set. Of the segments the LLM tier scores badly, how many does a human agree with? This is the only number that says the tier is worth its cost.
4. Numbers separately. Numeric integrity is not a quality score. Check it deterministically and report it on its own — a translation can score 100 and have lost a customs code.

Key Takeaways

• GEMBA-MQM's real advantage is auditability, not correlation: an error list with categories and spans is something a reviewer can act on.
• GEMBA V2 buys variance reduction at about 10× cost. Use it on flagged segments, not as the default tier.
• MetricX-25 scores better and needs your own infrastructure — a real trade if you are API-only.
• A fallback returning a plausible score hides an outage. Ours returned a flat 75 for months; the tag on the score is what exposed it.
• A prefilter that escalates nothing makes your expensive tier decorative. Measure the escalation rate and set the threshold against the asymmetric costs.

FAQ

What is reference-free quality estimation?

Judging a translation using only the source and the target, with no human reference to compare against. It is the only kind of measurement available in production, where references do not exist by definition.

Is GEMBA-MQM still the right choice in 2026?

For teams on paid APIs, yes — it is reproducible without hosting anything, and its output is an auditable error list rather than a bare score. MetricX-25 scores better on benchmarks but requires you to operate a model, which is a different kind of commitment.

What does GEMBA V2 change?

It keeps the prompting and reduces variance by sampling the judgement about ten times, aggregating with a rank-based weighted average instead of a plain mean. Cost scales with the sampling, so it is best applied selectively to segments a cheaper tier already flagged.

Why would quality estimation return the same score for everything?

Because the LLM call is returning empty and a fallback is filling in. In our case a reasoning model spent the entire token budget on thinking, and the fallback returned a flat 75 that looked like a real score. The fix was one flag; the lesson was that a fallback must be distinguishable from a measurement.

Should quality estimation cover numeric accuracy?

No — check numbers deterministically and report them separately. A segment can score 100 on quality and still have dropped a customs code, because a missing digit is not a fluency problem and no quality metric is built to notice it.

Conclusion

Reference-free estimation has become good enough that the interesting questions are operational rather than methodological: which tier runs on which segments, what a score is tagged with, and whether the number in front of you was measured at all.

The methodology literature will keep improving correlation. In our experience the bigger wins were elsewhere — noticing that the expensive tier had silently stopped running, and discovering that the cheap tier was never letting anything reach it.

If you want document translation with quality estimation you can inspect segment by segment, try KTTC.