What Is COMET?
COMET (Crosslingual Optimized Metric for Evaluation of Translation) is a neural network for scoring translation quality. Unlike BLEU, which simply counts word overlap with a reference, COMET is trained on a large set of translations paired with human ratings and learns to predict that rating based on semantic similarity rather than literal text overlap.
Two Modes
COMET-22 is the base variant, which still needs a human-written reference translation to score against, though the comparison is semantic rather than word-for-word.
COMETKiwi is the reference-free variant, working with no reference at all: the model scores the source-target pair directly. This is specifically the part of COMET people usually mean when they talk about MTQE.
Why COMET Beats BLEU
BLEU penalizes any rephrasing, even when the meaning is preserved perfectly — its match is counted word by word and phrase by phrase. COMET is trained on semantic similarity and doesn't punish a translation for saying the same thing differently. That matters especially for languages with flexible word order and for scoring translations from modern LLMs, which often rephrase rather than translate literally.
How to Use a COMET Score
A COMET score typically falls somewhere around 0 to 1 (sometimes expressed on a 0–100 scale) and is meant to be read relatively, not absolutely — what matters is comparing systems, or comparing versions of the same system, on the same set of texts, not the single number in isolation. Typical uses:
- Comparing several translation systems against each other on one test set
- Tracking a single system's quality over time
- Prioritizing segments for post-editing when no reference exists (via COMETKiwi)
COMET vs. GEMBA
| COMET | GEMBA | |
|---|---|---|
| Model type | Purpose-trained scoring network | General-purpose LLM acting as a judge |
| Output | A numeric score | A score and/or MQM-style error annotations |
| Needs a reference | Not necessarily (COMETKiwi) | Not necessarily |
| Cost to run | Lower | Higher (a full LLM call) |
The current 2026 approach doesn't pick one over the other — COMET gives a fast, cheap first signal, and GEMBA or another LLM judge adds finer-grained annotation where that's worth the cost.
Limits
Like any trained model, COMET inherits the blind spots of its training data — its predictions are less reliable on rare language pairs or narrow terminology. A COMET score also doesn't replace MQM-compatible error annotation where a client needs to know not just "how good" but "what specifically is wrong."
FAQ
Does COMET need a reference translation?
In the base COMET-22 variant, yes. In the reference-free COMETKiwi variant, no — the model scores the source-target pair directly.
How is COMET better than BLEU?
COMET scores semantic similarity rather than word overlap, so it doesn't penalize a correct rephrasing and correlates better with human judgment.
Can COMET be used as the only metric?
For monitoring and comparing systems, yes. For a final call on text where an error is costly, COMET is typically paired with human review or an LLM judge that provides error annotation.
How should a specific COMET number be read?
As a relative value: what matters is comparison across systems or over time on the same set of texts, not an absolute "good/bad" threshold.
Are COMET and GEMBA competing approaches?
No, they're complementary: COMET is cheaper and faster as a first signal, while GEMBA (or another LLM judge) adds error-type annotation where a finer picture is needed, not just a number.