What GPT-4 gets right and wrong in professional translation

We have been running the same set of test documents through each new model release since 2023, and the picture of gpt4 translation quality that comes out of it is narrower than either the marketing or the backlash suggests. The model writes fluent target text. It does not reliably write correct target text. Those two things come apart in specific, repeatable places, and once you know where, the question stops being "is it good enough" and becomes "good enough for which part of the job".

That distinction is the whole argument of this piece. GPT-4 class models changed the economics of a first draft. They did not change who is responsible for the delivered file.

What gpt4 translation quality looks like on a real document

Here is a case we keep going back to because it is so ordinary. A supplier quality manual, German into English, 38 pages, numbered clauses, six tables, about 11,000 words. The model returns a full draft in a few minutes. Read it cold and it reads like English written by someone who knows the subject. No garbled syntax, no obvious machine-translation smell, headings that sound like headings.

Then a reviewer goes through it properly and logs about a dozen real errors. On 11,000 words that is a low rate, and the temptation is to call it a win and move on. The problem is what those twelve are. Three involve numbers. One flips an obligation: a clause that says the supplier shall not substitute materials without written approval comes back saying the supplier shall substitute materials upon written approval. Everything around it is clean, so nothing draws the eye to it.

This is the pattern that matters. Older segment-based machine translation produced many errors, most of them visible. A fluent model produces few errors, and the surviving ones are camouflaged by good prose on either side. Reviewers who learned their habits on statistical MT tend to skim, because skimming used to work: the bad segments announced themselves. Against an LLM draft, skimming is how you ship an inverted obligation.

So the honest summary of the quality level is this. Fewer errors than you expect, placed worse than you expect, and a draft that actively discourages the kind of reading that would catch them.

What the model gets right, and this part is genuinely new

The gains are real, and they are mostly about looking beyond the sentence.

Document-level coherence is the clearest one. A pronoun in paragraph four that refers to a subject introduced in paragraph one usually resolves correctly. Defined terms stay consistent with the definition given earlier in the same text. Segment-by-segment MT could not do this at all, because it never saw paragraph one.

Structural rewriting is the second. German and Russian technical writing leans heavily on nominal constructions, and a literal rendering produces the stiff, noun-stacked English every translator recognises. GPT-4 class models unpack it without being asked. Russian "осуществляется приёмка товара" comes back as "the goods are accepted" rather than "acceptance of the goods is carried out". A long German sentence with three subordinate clauses gets split into two readable English sentences with the logic preserved. That is editing judgement, not substitution, and it is the single biggest reason post-editing an LLM draft feels different from post-editing MT output.

Register tracks the document type reasonably well. A warranty clause comes out formal, a product description comes out marketing-adjacent, and the model rarely mixes them inside one file.

Lexical ambiguity resolved from context is the fourth. German "Anlage" can be a plant, an annex, an attachment, or an installation. In a maintenance procedure the model picks the mechanical sense; in a contract with numbered appendices it picks the annex sense. Not always, but often enough that the error rate on this category has dropped sharply since 2023.

None of this makes it a translator. It makes it a competent first drafter with no accountability, which is a specific and useful thing to have.

The failure modes that actually cost money

Four categories account for nearly everything we see.

Numbers and units come first and stay first. Decimal commas and thousands separators move between locales, and the model handles the conversion correctly most of the time, which is worse than handling it badly. A German "0,5 kg" is usually "0.5 kg" and occasionally "5 kg". A Russian tender clause reading "не позднее 10 рабочих дней" came back as "no later than 10 days", dropping "working". Five calendar days of difference, in a document where the deadline is the point.

Negation and modality come second. Shall, shall not, may, must not, is not required to. These are short, structurally similar, and carry the entire legal weight of a clause. The model gets them right at a high rate and wrong at a rate that is nowhere near low enough for contract work.

Omission comes third. In long sentences with multiple conditions, a subordinate clause occasionally vanishes. There is no gap, no artefact, no awkward seam. The output is a grammatical sentence that says less than the source did, and the only way to find it is to compare against the source line by line.

Silent repair of bad source is fourth and the most interesting. Source documents contain contradictions, broken cross-references and leftover text from a previous version. A human translator queries those. The model smooths them into something plausible, and the problem disappears from the deliverable while remaining in the client's original. We have had a reviewer catch a contradiction in an English draft only to find the contradiction was not in the English at all; the model had resolved it, and the German still said two different things.

Mapped onto the MQM error typology, this is a model that scores well on fluency and poorly on accuracy. That is the worst combination for human review, because fluency is what reviewers notice and accuracy is what they have to work for.

Terminology drift is the error you will see every single time

If you take one thing from our test runs, take this one. Terminology is not an occasional problem with LLM translation. It is the default behaviour.

The mechanism is simple. A long document is processed in pieces, and the model has no persistent memory of what it chose earlier unless you give it one. So a term that appears forty times across thirty pages gets rendered two or three different ways, each of them defensible in isolation.

A German construction contract we used as a test case contains "Mangel" 31 times. Across one unguided run it came back as "defect", "deficiency", "fault" and "shortcoming". Every one of those is a dictionary-correct translation. In a contract where "defect" triggers a specific remedy clause, three of them are wrong, and a reviewer reading linearly will accept each as it passes because each reads fine where it sits.

The second half of the problem is client preference, which no model can infer. A client may insist on "non-conformity" where the industry default is "defect", because that is the word in their quality system. That lives in a glossary or it lives nowhere.

Which is why terminology work belongs before translation, not in QA afterwards. A glossary supplied as part of the request constrains the choice at generation time. Fixing it afterwards means a find-and-replace pass over a document where the same term legitimately differs by context, and that pass introduces its own errors.

This works best when your terminology is already written down. It does not help much on a first job for a new client in an unfamiliar domain, where you do not yet know which terms are load-bearing. On those, expect drift and budget review time for it.

What context changes, and what it cannot touch

Giving the model more to work with does help, in an uneven way that is worth understanding.

Domain context, a glossary and a clear instruction about audience and register produce the largest gains on exactly the things that are knowledge problems: specialist vocabulary, house style, the difference between a patient-facing and a regulator-facing version of the same medical text. Supply a domain and a term list and the lexical error rate drops noticeably.

Those same inputs do almost nothing for numbers, negation and omission. Not because the instruction is ignored, but because those failures are not knowledge gaps. The model is not unsure whether "shall not" means "shall not". It produced a fluent sentence and the negation fell out along the way. No amount of context fixes a mechanism like that, which is why the checks described in the next section are not optional extras.

Language pair matters more than most comparisons admit. English paired with German, French, Spanish or Portuguese sits at the high end. Japanese, Korean and Chinese into English sit lower, with more restructuring errors and more lost cohesion, because the source does less of the grammatical signalling the model relies on. Low-resource pairs are a different conversation entirely.

This works best when the document is internally consistent, sits in one domain, and you can state plainly who will read it. It applies much less well to mixed-domain files, to heavily templated legacy text full of client conventions nobody wrote down, and to documents where the source itself is a mess.

How to check the output when nobody on the team reads the target language

This comes up constantly, and the answer is a layered one rather than a single tool.

Start with the checks a machine does better than a person. Number and date matching between source and target, unit consistency, placeholder and tag integrity, terminology compliance against the glossary, and segment length ratio outliers. These catch most of the expensive category, numbers, and they catch omission when a target segment is suspiciously short. A QA report that lists flagged segments turns a 38-page review into a review of 40 lines.

Quality estimation scores help for triage. They are a prediction of adequacy, not a verdict, and they are least reliable on exactly the fluent-but-wrong cases that worry you most. Use them to rank which segments a human looks at first, not to decide whether a file ships. We wrote about where those scores hold up and where they do not in what quality estimation can and cannot tell you.

Back-translation has a narrow but real use. It surfaces inverted negation and dropped clauses, because both change the meaning in a way that survives the round trip. It tells you nothing about register, and it generates false alarms on any sentence the model legitimately restructured.

Then put a native reviewer on the flagged subset plus a sample of everything else. For work delivered under ISO 17100, revision by a second person is a requirement of the standard regardless of how the draft was produced. An AI first draft changes the cost of that revision. It does not remove it.

What to do with this

Decide post-editing level per document, not per client. The same client sends you internal memos where light post-editing is appropriate and supply contracts where it is not, and treating them the same is how margin disappears. Our breakdown of light versus full post-editing covers how to make that call per file.

Then run your own pilot, because general benchmarks will not tell you about your content. Take 500 to 1,000 words of real work in your actual language pair, with your actual terminology. Translate it with the model, hand it to the reviewer who normally handles that client, and have them log every error by MQM category rather than just fixing it. Two hours, once.

What you are looking for is the shape of the error profile. If the errors cluster in fluency and style, light post-editing is viable and you can price accordingly. If accuracy errors show up in numbers, modality or terminology, you need a full post-edit with an automated number and terminology pass in front of it, and the quote has to reflect that. The count matters less than the distribution, and the distribution is stable enough per client and document type that one pilot tells you how to handle the next twenty jobs.