What Is BLEU?
BLEU (Bilingual Evaluation Understudy) is a machine translation quality metric introduced in 2002 that became the industry standard for decades. BLEU compares a machine translation against one or more human-written reference translations and counts the share of matching n-grams — sequences of one, two, three, and four consecutive words.
The resulting score is a number from 0 to 100 (or 0 to 1): more overlap means a higher score.
How BLEU Is Calculated
In simplified terms: the metric takes a translation, breaks it into n-grams of different lengths, checks how many appear in the reference translation, and applies a separate brevity penalty to short translations, so a model can't inflate its score by producing short fragments that happen to overlap.
Why BLEU Stopped Being Enough
BLEU's core problem is that it scores literal overlap, not meaning. A correct translation phrased differently from the reference gets a low score, while a poor translation that happens to overlap with the reference's word choice gets a high one. This shows up especially with translations from modern LLMs, which tend to rephrase — BLEU systematically underrates their quality compared to a human's judgment.
An added problem: BLEU needs a reference translation for every text, and one often simply doesn't exist.
BLEU vs. COMET
| BLEU | COMET | |
|---|---|---|
| Year introduced | 2002 | 2020 |
| What it compares | Literal word overlap | Semantic similarity |
| Needs a reference | Yes, always | Not necessarily (COMETKiwi) |
| Correlation with humans | Moderate | High |
| Relevance in 2026 | Historical, for comparing against old benchmarks | Primary |
BLEU hasn't disappeared entirely — it stays useful for a quick rough comparison, and anywhere comparability with years of historical data reported in BLEU matters.
When BLEU Is Still Used
- Comparing against academic benchmarks that historically reported scores in BLEU
- A fast, rough check during development, before more expensive scoring via COMET or GEMBA
- Teaching and demo purposes — the metric is simple to explain
FAQ
Is BLEU still used in the industry?
As the primary acceptance metric — rarely. As one signal among several when comparing systems, or for comparability with historical data — yes.
What BLEU score counts as good?
It depends heavily on the language pair and text type, and there's no single threshold; structurally similar languages with good translation typically score higher than language pairs with very different grammar.
Why does BLEU underrate good LLM translations?
Because modern LLMs rephrase more often than they translate word for word, and BLEU counts exactly that literal overlap — a correct but differently worded translation gets penalized.
How is BLEU different from chrF?
chrF counts character overlap rather than word overlap, which makes it more robust to morphological differences in highly inflected languages — but it still measures literal overlap rather than meaning, unlike COMET.
Is BLEU needed to evaluate KTTC's document translations?
No: evaluating document translation is better served by methods that don't need a reference for every unique document — which is exactly why the industry moved from BLEU to reference-free approaches like MTQE.