What Is GEMBA?
GEMBA (GPT Estimation Metric Based Assessment) is a family of machine translation quality evaluation methods where the evaluator is a large language model, rather than a network purpose-trained for this narrow task, like COMET. Instead of a separate training step, GEMBA simply frames evaluation as a prompt to an LLM: show the model the source and the translation, ask it to score it.
It was the first widely recognized approach to show that an off-the-shelf language model, with no additional training, can score translation quality at the level of specialized metrics or better.
Three GEMBA Variants
- GEMBA-DA — the model outputs a single numeric quality score (Direct Assessment), similar to how a human expert would rate a translation on a scale
- GEMBA-ESA — the model annotates specific problem spans (Error Span Annotation) and derives a final score from that annotation
- GEMBA-MQM — the model classifies the errors it finds by type and severity using MQM logic, producing structured annotation compatible with human evaluation methodology, not just a number
GEMBA-MQM is the most informative of the three: it doesn't just say "this translation is mediocre" — it explains what specifically is wrong and how serious it is.
How GEMBA-MQM Works in Practice
The model receives the source text, the translation, and an instruction to classify errors by MQM categories (accuracy, fluency, terminology, style) with a severity for each. The result is a list of found issues that can be aggregated into a numeric score using the same formula used in manual MQM scoring — which lets you compare LLM annotation directly against an editor's work.
GEMBA and the Segments It Misses
Segments GEMBA(-MQM) couldn't process, or that timed out, need an explicit policy: an unscored segment is not the same thing as a scored segment with no issues found. Blending those two cases into one "great score" is a common and dangerous reporting mistake, because the resulting average starts reflecting the skip rate instead of actual quality.
GEMBA vs. COMET: Which One to Use
| GEMBA | COMET | |
|---|---|---|
| What it needs | LLM access and a prompt | A purpose-trained model |
| Cost to run | Higher — a full LLM call per segment | Lower |
| Output detail | Can give error-type annotation (GEMBA-MQM) | Usually a number only |
| Flexibility | High — adjust via the prompt | Low — the model is trained once |
Combining them is common: COMET quickly filters out clearly good segments, and GEMBA-MQM examines what's left in detail.
Limits
LLM-based scoring inherits all the model's own blind spots: if the model handles a rare language or a narrow domain poorly, its translation scoring in that area will be unreliable too. An LLM judge can also be inconsistent between runs unless temperature and model version are pinned — worth keeping in mind when comparing scores over time.
FAQ
How is GEMBA different from QE in general?
GEMBA is a specific approach within the broader term QE, using an off-the-shelf LLM as the expert instead of a purpose-trained metric.
Which GEMBA variant is most useful?
GEMBA-MQM, because it gives not just a score but an explanation of which errors were found and how serious they are.
Does GEMBA need a reference translation?
No, GEMBA usually runs reference-free, scoring the source-target pair directly, though the format also allows a reference-based mode.
Is a skipped segment the same as a low score?
No, and reporting this wrong is an easy mistake: a skip (not scored) and a low score (scored, and issues were found) are different states that need to be counted separately.
Can GEMBA's score be trusted as the only criterion?
For monitoring a stream, yes. For decisions where a mistake is costly, GEMBA's annotation is usually double-checked by a person, especially at the boundary between a "major" and a "minor" error.