What is quality estimation in machine translation and can you trust the score?

A client asks how good the machine translation is, and nobody has read a word of it yet. That is the gap machine translation quality estimation was built to fill: predict how good a translated segment is using only the source and the target, with no human reference to compare against. The score comes back in seconds and looks authoritative. In our experience working with agencies and freelancers who run AI translation at volume, that score gets trusted about twice as much as it deserves, and used for maybe half the things it is genuinely good at.

So it is worth being precise about what the number is, where it comes from, and what it cannot see.

What a quality estimation score actually predicts

Reference-based metrics need a correct translation to exist before they can say anything. BLEU compares your output to a human translation. chrF does the same at character level. COMET, in its standard configuration, scores output against a reference using a neural model trained on human ratings. All of them answer one question: how close is this to the translation we already have?

QE answers a different question. Given this source and this target, how much work is left?

That difference changes how the number should be read. Most QE models are trained to predict either a Direct Assessment score, which is a human rating on a 0-100 scale, or HTER, the edit rate a post-editor actually produced while fixing the segment. The HTER-trained ones are more honest for production use, because the output maps to something you can act on. A high score means the model expects light touching up. A low score means it expects heavy editing or a rewrite.

What the number does not mean is "correct". A segment can be fluent, grammatical, terminologically plausible and flatly wrong about a quantity, and a QE model will frequently score it well. That is where most of the real damage happens, and we come back to it below.

Granularity matters too. Sentence-level QE gives one score per segment, which is what almost everyone uses. Word-level QE tags individual tokens as OK or BAD, which is more useful inside an editor and considerably noisier. Document-level QE exists mostly in research papers, because the typical model never sees more than one segment at a time and therefore has no way to notice that segment 40 contradicts segment 12.

How QE models learn what good looks like

QE is not magic and it is not rule-based. A model is shown pairs of source and machine-translated target along with a human judgment of how good that target was, and it learns to predict the judgment. The WMT quality estimation shared task has been running this exercise since 2012, and most of what is deployed commercially descends from it. Unbabel's COMETKiwi is the reference-free line of the COMET family. TransQuest came out of the same research community. More recently, teams have been prompting large language models to produce a quality judgment directly, which works surprisingly well and is much harder to reason about.

The training data explains the blind spots. Annotated QE data is heavily weighted toward news, parliamentary proceedings, and encyclopedic text, in high-resource European pairs. A lease agreement, a safety data sheet, or a bill of materials sitting in an XLSX workbook is out of distribution for the model in a way that is invisible from the outside. The score still comes back with two decimal places.

There is a second issue that we ran into before we understood what we were looking at. QE models were largely calibrated on the output of neural MT engines, which fail in characteristic ways: literal renderings, awkward word order, obvious dropped agreement. LLM output fails differently. It is fluent, idiomatic, and confident even when it has invented something. A QE model tuned on NMT error patterns tends to be generous toward LLM output for exactly the reason a careless human reviewer is generous toward it. The text reads well.

The LLM-as-judge approach has its own quirk. Asking a large model to rate a translation produces a number that correlates decently with human judgment and varies between runs on identical input, because the same sampling behaviour that makes the translation non-deterministic makes the rating non-deterministic too. If you are going to use one, pin the sampling settings and run it more than once on a sample before you believe the spread.

None of this makes QE useless. It makes QE a tool with a known shape, which is a different thing.

Where machine translation quality estimation earns its place

The single best use is triage, and it is genuinely good at it.

Take a batch of forty files, sixty thousand words, delivered as a mixed set of DOCX manuals and XLSX registers. Nobody is reading all of that carefully. Score every segment, sort ascending, and hand the reviewer the bottom fifteen percent first. The reviewer spends their first two hours where the probability of finding something is highest instead of starting at page one and losing attention by page nine. We have watched this change the outcome of a delivery more than once, and the mechanism is boring: attention is the scarce resource, and QE is a cheap way to point it somewhere.

The second use is effort estimation before you quote. If you are pricing post-editing work, the distribution of QE scores across a file tells you more than the word count does. Two 20,000-word files can carry completely different amounts of work, and a score distribution skewed low is a reason to quote a higher per-word rate or to decline. This connects directly to how you structure the post-editing pass itself, which we have written about separately in our guide to post-editing AI translations efficiently.

The third use is stratified sampling for QA. Most agencies sample ten percent of a file at random. Random sampling is a defensible statistical choice and a poor practical one, because errors are not uniformly distributed. Sampling ten percent weighted toward the low-scoring end finds more problems for the same reviewer time.

There is a fourth use that gets overlooked, and it is the one we would defend hardest. QE can tell you a file should not have gone near machine translation in the first place. When the whole score distribution for a document sits low rather than showing the usual long tail, that is rarely a translation problem. It is usually a source problem: an OCR layer full of garbage, a scanned form where the text extraction merged two columns, a spreadsheet where half the cells are fragments with no sentence around them. Catching that before the post-editor opens the file saves an afternoon and an awkward email.

What all three of the earlier uses have in common: QE is being asked to rank, not to judge. Ranking is what it does well. The moment somebody proposes using a QE threshold as an automatic pass/fail gate on client deliverables, the tool is being asked for something it cannot supply.

The errors QE is structurally bad at catching

Numbers are the big one. A tolerance in a technical spec that reads "≤ 0.5 mm" in the source and "≥ 0.5 mm" in the target is fluent, grammatical, and catastrophic. Most QE models score it fine, because nothing about the sentence looks broken from a language-modelling perspective. The same applies to decimal separators moving between locales, currency codes, dates flipping between day-first and month-first conventions, and units silently staying imperial.

Omission is the second. If the translation drops a subordinate clause, the remaining sentence is often perfectly well-formed. Some QE models pick up on the length mismatch, many do not, and none of them reliably notice that the missing clause was the one containing the liability limit.

Terminology is the third, and it is structural rather than accidental. A QE model has no access to your client's glossary. It cannot know that this client requires one specific German term for a manual and treats the common synonym as a defect. In an AI translation workflow the glossary is the thing carrying most of the client-specific quality burden, and QE is completely blind to it.

Then there is consistency. Because sentence-level QE evaluates segments in isolation, the same source term rendered three different ways across a 200-page document produces three respectable scores. Register and formality slip through the same hole: a QE model rarely knows whether this audience gets du or Sie.

We saw a version of this on a price list in an XLSX workbook where a thousands separator shifted in about forty cells. Every one of those cells scored well. The workbook was fluent, well-formed, and wrong by a factor of a thousand in places, and a QE gate would have waved it straight through to the client.

Calibrating a QE score against your own reviewers

The absolute number a QE model gives you is close to meaningless on its own. What matters is whether it ranks segments in roughly the order your reviewer would. That you can measure, and it takes about a day.

Pull 200 to 300 segments from real work in one language pair and one domain. Have a reviewer grade each one on a three-band scale: deliverable as it stands, needs a light edit, needs a rewrite. Then line those bands up against the QE scores and look for the point where the rewrite band starts to dominate. That is your threshold. It belongs to your pair, your domain, and your engine, and it will not transfer to your neighbour's.

Recalibrate whenever any of those three change. Switching translation engines is the one people forget, and it is the one that shifts the distribution most. A threshold that meant "send this to a reviewer" under last quarter's engine can mean "this is fine" under the new one, and the scores give you no warning that the ground moved.

One more thing the calibration exercise buys you: it tells your reviewers how much to trust the ordering. If a grader disagrees with the QE ranking on a third of the sample, that is worth knowing before you build a queue on top of it. We would rather find that out on 250 segments than on a client delivery.

This works best when you have steady repeat work in a handful of pairs. It does not apply well if every job is a different language and a different subject, because you never accumulate enough graded segments in one bucket to see a pattern. For that kind of shop, QE is still useful for within-file ranking, and the cross-file thresholds are not worth chasing. If you want the wider picture on how these scores relate to reference-based metrics, we covered that in our piece on what the COMET score actually tells you.

Pairing QE with checks that catch what it misses

The practical arrangement is layered, and the layers go in a specific order.

Deterministic checks run first: number and date matching, placeholder and tag integrity, untranslated segments, glossary term adherence, forbidden terms, length ratio outliers. These checks are unglamorous and mechanical, and they catch precisely the class of error QE cannot see. Any decent QA tool does them.

QE runs second, over what survives, and its only job is to order the reviewer's queue.

Human review runs third, starting at the top of that queue and stopping when the return on attention flattens out. The reviewer, not the score, decides what ships.

This is roughly the shape of what we built into SnapIntel. A finished project returns a QA report and a quality rating next to the translated DOCX, XLSX or PPTX, plus a neutral source and target XLSX export you can pull into any CAT tool for the review pass. The rating exists to tell a reviewer where to look first. It is not a sign-off, and we would not want anyone treating it as one. The workflow is at snapintel.io if you want to see how the pieces sit together.

What to do with this next week

Start with one file you have already delivered and already know the problems in. Run QE over it and check whether the low scores land on the segments your reviewer actually fixed. If they do, you have something worth building a process on. If they scatter, your domain is far enough from the model's training data that you should treat the scores as decoration for now.

Then write down, explicitly, the list of things your QE model cannot check, and make sure something else in your pipeline checks them. Numbers, dates, units, placeholders, glossary terms, cross-document consistency. That list is short and it is where the expensive failures live.

The score is a good instrument for pointing a human at the right paragraph. It is a bad instrument for deciding whether a human is needed.