What Is LLM-as-a-Judge?
LLM-as-a-judge is an evaluation method in which a large language model is asked to score somebody else's output — usually another model's, sometimes a person's — against a rubric written in plain language.
The shift it represents is small to describe and large in consequence. Classic metrics compare a candidate against a reference answer and count overlap; BLEU does this with n-grams, COMET with learned embeddings. A judge model is given no reference. It is given the source, the output, and a description of what "good" means here, and asked to apply that description.
In translation, the best-known instance of the method is GEMBA — a specific protocol that asks a model to mark errors in the categories of the MQM typology and assign a penalty to each.
Why the Method Spread
Three properties explain its adoption, and each has a cost attached.
It needs no reference translation. Keeping a human reference for every document is what made reference-based metrics impractical outside research. A judge model removes that requirement, which is the same reason MTQE exists as a category.
The rubric is editable. Changing what a judge rewards means rewriting a paragraph, not retraining a model. A team that decides terminology consistency matters more than fluency can say so in words, the same afternoon.
It gives reasons, not only a number. A judge can name the span it objected to and the category of the objection. That output can be routed to an editor. 0.71 cannot.
Where It Fails
The failures are systematic and documented, and they are not evenly distributed — which is what makes them dangerous.
Position bias. Asked to compare two candidates, judge models prefer the first one presented at rates well above chance. Any A/B evaluation that does not swap the order and average is measuring presentation, not quality.
Verbosity bias. Longer, more elaborate answers score higher than concise ones of equal correctness. In translation this rewards padding and explanatory insertion — precisely what a good translation avoids.
Self-preference. A model tends to score text produced by itself, or by a model of its own family, above equivalent text from elsewhere. Evaluating a system with a judge from the same family measures kinship as well as quality.
Scale compression. Judges cluster their scores. Asked for 0–100 they will produce 70–95 and almost nothing below, which flattens the distinction between "acceptable" and "excellent" and hides small regressions inside noise.
Fluency is mistaken for accuracy. A fluent translation carrying a factual error — a wrong number, a reversed negation, a swapped name — reads to a judge like good work, because the surface signals of quality are all present. This is the failure that matters most in document translation, where the numbers are the content.
How the Method Is Used Responsibly
A judge is a triage instrument. The practices that make it a useful one:
- Swap the order in every pairwise comparison, and average the two runs.
- Use a different family of model for judging than for producing, where the choice exists.
- Score against a rubric with named categories, not a bare "rate this 1–10" — the categories are what make disagreements inspectable.
- Calibrate against human judgments on a sample before trusting a number, and re-calibrate when the prompt changes.
- Never let a judge gate what a person would have caught. Numbers, names, dates and legal quantities need a deterministic check, not a probabilistic reader.
At KTTC, GEMBA-MQM quality estimation runs over translated segments (app/services/quality_estimation.py). Segments it does not score are auto-passed rather than failed, so a top score records "not objected to", which is not the same claim as "verified".
FAQ
Is LLM-as-a-judge the same thing as GEMBA?
No. LLM-as-a-judge is the general method — a model evaluating output against a rubric. GEMBA is one named protocol built on it, specific to translation, which asks the judge to mark errors in MQM categories rather than emit a single number.
Is a judge model better than COMET?
They answer different questions. COMET is fast, cheap and stable, and returns one number. A judge is slower and costs a model call, and returns categorized reasons. Current practice runs COMET across everything and a judge over what COMET flags, rather than choosing between them.
Can a judge model replace human review?
Not where an error is expensive. A judge approximates an averaged human reading, and its documented biases run in the direction of confidence rather than caution. It decides where human attention goes; it does not substitute for it. See LQA.
How much agreement with human reviewers should be expected?
That is a question to measure per deployment, not to quote. Agreement depends on the rubric, the language pair, the domain and the model, and a figure produced in one setting says little about another.
Does using the same model to translate and to judge invalidate the score?
It biases it. Self-preference is measurable, and the safe assumption is that a system scoring its own work scores it generously. Where a second family of model is available, use it for the judging half.