COMET score explained: what translation teams need to know about MT quality metrics

The COMET score translation teams see in evaluation reports tends to get treated in one of two ways: unquestioning trust ("0.86, looks solid") or quiet dismissal ("the client doesn't care about numbers"). Neither tells you whether the translation is actually good.

We've watched this play out enough times that it's worth working through properly. What COMET measures, how it compares to BLEU, where it reliably reflects quality and where it doesn't, and what you can actually do with the score in a production workflow.

What the COMET score actually measures

COMET (Crosslingual Optimized Metric for Evaluation of Translation) was developed at the University of Lisbon and first presented at WMT 2020, where it outperformed BLEU, TER, and chrF in correlating with human quality assessments across multiple language pairs.

The mechanics are different from older metrics. Rather than counting n-gram overlaps between an MT hypothesis and a reference translation, COMET uses a pretrained cross-lingual language model as its backbone. It takes three inputs: the source sentence, the MT output, and a human-written reference. The model encodes all three and computes how well the hypothesis aligns with both the source and the reference in a shared multilingual embedding space.

The output is a score, typically in the 0–1 range for quality estimation models, though variants like wmt22-comet-da can go slightly above 1. The absolute number matters less than what the model learned to predict. COMET was trained on direct assessment (DA) scores — numerical ratings collected from professional translators evaluating segment-level MT quality across thousands of sentence pairs. It learned to predict what professional translators consider acceptable output, not what string similarity looks like.

That distinction matters in practice. If "la réglementation applicable" becomes "the applicable regulations" in one translation and "the relevant regulatory framework" in another, BLEU penalizes the second because the n-gram overlap is lower. COMET sees both as semantically close to the source and scores them similarly. Paraphrase, which LLM-based translation produces constantly, no longer gets automatically penalized.

COMET score translation evaluation vs. BLEU: the actual difference

BLEU was the dominant MT evaluation metric from roughly 2002 to the mid-2010s and it's still used everywhere — partly because it's fast, reproducible, and requires only a reference and a hypothesis. It counts how many 1-, 2-, 3-, and 4-gram sequences in the hypothesis appear in the reference, applies a brevity penalty, and produces a number between 0 and 100 (or 0 and 1, depending on implementation).

The problem is well-documented but worth stating plainly. BLEU penalizes valid paraphrase. It doesn't capture meaning. A segment that inverts a negation can score well on BLEU if the surrounding words match the reference. A segment that paraphrases correctly can score poorly for the same reason. Research from WMT shared tasks has consistently shown COMET correlating significantly better with human assessments than BLEU across most tested language pairs. The gap is largest for high-resource pairs like English-German and English-French. For low-resource pairs, COMET still tends to outperform BLEU, though its reliability drops when training data and references for those pairs are sparse.

TER (Translation Error Rate) counts the edit operations needed to transform the hypothesis into the reference; chrF uses character n-gram overlap. Both share BLEU's dependency on surface-level reference matching. COMET's advantage is that it introduces semantic understanding into what is otherwise a string comparison problem.

One practical constraint: standard COMET variants like wmt22-comet-da require a reference translation to compute. Quality estimation models like CometKiwi score hypotheses without references, which changes which use cases are actually feasible in production.

Why COMET thresholds break down across contexts

Here's where teams run into the most trouble. A QA process sets an "acceptable COMET score" — say, 0.80 — applies it uniformly across all projects, and then produces confusing results. Some segments that score 0.79 are fine. Some that score 0.85 contain a mistranslated technical term. The threshold feels systematic but behaves inconsistently.

This happens because COMET scores are relative in ways that matter for production use.

Average scores vary systematically by language pair, independent of translation quality. English-Portuguese tends to score higher than English-Finnish not because one is better translated, but because reference translation patterns and model training data differ across pairs. A 0.78 on a rarely evaluated language pair might represent solid output; the same score on English-Spanish might warrant closer review.

Domain sensitivity compounds this. Legal and medical text typically scores lower than general content because terminology-dense material has less acceptable paraphrase. A COMET score that's fine for a marketing document can indicate a real problem in a pharmaceutical regulatory filing, even if the number looks identical.

Model version drift is a third variable. Upgrading from wmt20-comet-da to wmt22-comet-da shifts your average scores even if your MT quality hasn't changed. Teams that set thresholds without pinning their COMET model version, then upgrade mid-project, run into this: the scores move; the translations didn't.

If you're building COMET thresholds into a production gate, calibrate them per language pair and per domain. Take a set of segments your human reviewers have already rated as acceptable in a given domain, run COMET on them, find where the distribution sits, and use that as your reference point. A threshold established on English-German legal content isn't transferable to English-Korean marketing text without re-calibration. If you want a broader framework for measuring translation quality in your agency, that's a useful starting point before introducing metric-based gates.

How translation teams actually use COMET

The most reliable use cases in production fall into several categories.

MT engine selection is the most straightforward. Before committing to a new translation engine for a specific language pair and domain, run both candidates on a representative test set with professional reference translations and compute COMET for each. This gives a more principled basis for comparison than reviewing sample segments manually. You want at least 100 sentence pairs per condition — smaller samples have too much variance.

Post-editing prioritization works differently. Quality estimation models like CometKiwi score MT output without reference translations, making them usable in real production environments where references aren't available. A post-editor working through 500 segments can use COMET-QE scores to sequence their work — identify the bottom 20% by score, review those first, spot-check the rest. This doesn't reduce review volume. It allocates effort more efficiently.

Tracking average COMET per job, per engine, and per language pair over time also surfaces quality drift before it shows up in client feedback. A sudden drop in average COMET for a specific language pair can indicate an engine update, a domain shift in the content, or a change in post-editing coverage. Steady decline over multiple jobs is harder to spot without a time-series view.

Translation memory quality filtering is a less common use case but a legitimate one. If you're building training data or curating a TM, COMET can help identify segment pairs where the hypothesis and reference align well enough to treat as reliable. Pairs that score poorly become candidates for removal or re-review before they enter a reference corpus.

What COMET misses

This is the part that matters most for deciding how much to trust the metric.

COMET doesn't evaluate factual accuracy. It compares a hypothesis to a reference. If the MT output hallucinates a date, a number, or a proper name, and the surrounding text is semantically similar to the reference, COMET may score it well. We've seen translations that swapped one product name for a competitor's, scored above threshold, and cleared automated QA without any flags. The error only surfaced during human review. This isn't an edge case — it's a predictable failure mode for any reference-matching metric applied to fluent LLM output.

Terminology adherence isn't captured. If your glossary specifies "adverse event" in clinical trial documentation rather than "side effect," a hypothesis using either term can score identically on COMET. The metric has no access to your termbase. For regulated industries, terminology errors are the most expensive category of mistake, and COMET won't catch them. That gap needs to be filled by dedicated terminology QA tools.

Document-level consistency isn't evaluated because COMET operates at the segment level. A translation that uses three different renderings of the same technical term across a 30-page document can score well on every individual segment while failing a basic consistency check across the document. Segment-level evaluation and document-level consistency check different things.

Fluency inflates scores for modern LLM output. Large language models produce fluent text. A segment that reads naturally but softens a regulatory obligation, inverts a conditional, or mistranslates a product specification can score well on COMET because its surface semantic profile is close to the reference. The more fluent the model, the more COMET overestimates quality in these cases.

None of this makes the metric useless. It makes it one layer in a process. We looked at what Xbench, Verifika, and other tools catch that automatic metrics don't in our comparison of translation QA tools in 2026.

COMET vs. human review: how to think about the tradeoff

There's a question that comes up in almost every workflow discussion: given a fixed post-editing budget, do you run COMET on everything and apply human review only to low-scoring segments, or do you apply human review uniformly?

The answer depends on what errors you can afford to miss.

For general content where fluency is the primary concern and the consequences of a mistranslated phrase are manageable, COMET-guided review is a reasonable starting point. Flag the bottom-scoring segments for closer attention; spot-check the rest.

For legal, medical, or technical content where a single missed terminology error or factual inaccuracy has real consequences, COMET can sequence review but can't meaningfully reduce its scope. The cost of one undetected error in a regulatory filing exceeds any savings from reviewing fewer segments.

The pattern we see across agency workflows is that automated scoring hasn't replaced human judgment — it's changed where that judgment gets applied. Review becomes more targeted, not shorter. That's a reasonable outcome for volume workflows. It's not a reason to reduce review coverage in high-stakes domains.

If you're working with AI translation output and want a structured view of quality before delivery — including formatting checks, quality ratings, and a QA report alongside the translated file — SnapIntel includes this as part of its workflow. It doesn't replace systematic COMET benchmarking for MT engine evaluation, but it gives teams something concrete to work from before the file reaches the client.

What to actually do with a COMET score

Pin your model version before doing anything else. The wmt22-comet-da checkpoint, available through the comet Python package from Unbabel, is widely used and stable. If you switch models later, re-baseline before applying old thresholds to new results. This takes a few minutes to document and prevents significant confusion when you revisit QA criteria later.

Build baselines per domain and per language pair before setting thresholds. Take 200–300 segments that human reviewers have rated as acceptable in your domain, run COMET on them, and find where the distribution sits. The median and the 10th percentile together tell you what "acceptable" looks like in your specific context. A threshold calibrated on English-German legal text isn't valid for English-Thai marketing content.

Track COMET at the job level over time. Average segment COMET per engine, per language pair, per month. Normal variation is expected. Sudden shifts are worth investigating before they become patterns.

Don't report COMET scores to clients without context. A number without a model version, a baseline, and an explanation of what the metric measures creates more questions than confidence. Use it internally as a process diagnostic; report human-validated outcomes externally.

COMET is a better predictor of professional translator judgment than BLEU. Used with an honest understanding of what it can and can't catch — hallucinations, terminology errors, document-level consistency — it helps translation teams allocate review effort more effectively. That's a narrower claim than "ensures quality," but it's the one the metric actually supports.