How to vet and approve a new AI translation tool before rolling it out

A project manager forwards a link to a new AI translation tool, says the output on two test files looked clean, and asks whether the agency can use it on Thursday's delivery. Somebody now has to vet an AI translation tool for agency use, and in most of the agencies we talk to, nobody owns that decision. So it goes one of two ways. The tool gets waved through because one sample read well, or it gets refused because refusing is safer than being the person who said yes. Both are expensive. Here is the approval process we would run instead, in the order we would run it.

Why an agency must vet an AI translation tool, not just test it

Testing answers whether a tool produces usable output on your content. Vetting answers whether your agency is permitted to use it, who carries the risk when something goes wrong, and what has to change in your workflow once it is in. Those are separate questions, and the second set is the one that gets skipped.

We watched a small agency run a careful two-week quality comparison, pick a winner, and put it in front of four project managers. Six weeks later they found out that one of their larger clients had a clause requiring written notice before any new third-party service processed that client's documents. Nobody had read the contract, because the trial had felt like a technical exercise. Sorting it out cost more than a year of the tool's subscription: a disclosure letter, a call with the client's counsel, and a promise to keep that account on a separate workflow until the notice period ran out.

The opposite failure is quieter and more common. An agency decides nothing can be approved without a full security review, has nobody with time to run one, and so approves nothing. Meanwhile project managers translate on personal accounts with whatever is free that month. You end up with exactly the exposure you were trying to avoid, minus any record of it.

So the point of a vetting process is not thoroughness for its own sake. It is to produce a decision, on a date, that someone is willing to put their name on. Three weeks is usually enough. Three months means you have replaced the decision with a committee.

Decide who owns the decision before anyone opens a trial account

Pick one owner. Not a working group. The owner sets the decision date, collects the three inputs below, and writes the approval or the refusal. In an eleven-person agency we work with, that is the operations director, and the whole thing fits in two weeks.

The inputs that matter are a quality read, a contract read, and an operations read. The quality read belongs to whoever already signs off on deliveries, usually a senior reviewer, because that person knows what your clients complain about. The contract read belongs to whoever signs client agreements, because they are the one who will have to explain it. The operations read belongs to a project manager who will use the tool daily and can say what it breaks in the current handoff.

What goes wrong here is almost always the quality read being done by the person who found the tool. Enthusiasm is not a controlled test. We ask the reviewer who did not find the tool to run the pilot files, and we do not tell them which engine produced which output when there is more than one candidate.

One more thing to settle before the trial account exists: what happens if the answer is no. If a refusal means the project manager who proposed the tool quietly keeps using it, your process is theatre. Write down the alternative you will offer instead, even if the alternative is "we keep doing it the current way until the next review in six months."

Read your client contracts before you read the feature list

This is the step agencies skip, and it is the cheapest one to run. Open your five or ten largest client agreements and search them for the obvious terms: artificial intelligence, machine translation, automated processing, subprocessor, third party, confidentiality, data transfer. Most agreements say nothing about AI, which is itself an answer. Some require notice before a new processor touches the data. A few prohibit it outright for specified document types.

Under GDPR Article 28, a processor needs the controller's authorisation to engage another processor, and many agency agreements implement that with a named list plus a notice period. If your client's documents contain personal data, adding an AI translation vendor is usually adding a subprocessor, whatever the vendor's marketing calls itself. Our rule is to treat it that way by default and let the lawyers narrow it, rather than the reverse.

Out of this you get a three-column list that outlives the vetting exercise: clients whose contracts permit it, clients that need written notice first, and clients that are excluded. Keep the list where project managers can see it, because the list is what they will actually use in six months when nobody remembers the review. Our data privacy walkthrough covers the disclosure side of this in more detail, including what to send a client who asks where their files go: what agencies need to tell their clients about AI translation data privacy.

Two cases sit outside this logic. Sworn and certified translation is governed by whatever the receiving authority accepts, not by your contract. And work under legal privilege tends to come with instructions from counsel that override your general policy. Check both separately.

Run the pilot with the pass rule written down first

Write the pass rule before anyone uploads a file. Not a vague intention to "see if quality is acceptable". A sentence with a number in it, agreed by the owner and the reviewer, such as: post-editing time at or below sixty percent of our from-scratch rate on this content type, with zero accuracy errors in numbers, units, and safety instructions.

Then pick files that look like your real work rather than your easiest work. A DOCX with tables, footnotes, and tracked changes left in by the client. An XLSX workbook with merged cells and formulas, if workbook translation is part of what you sell. A PPTX deck with text in grouped shapes and speaker notes. Two language pairs, one of them a pair your current process struggles with, because a pilot run on English into German proves nothing about English into Japanese.

A 9,000-word EN to DE maintenance manual told us more in one afternoon than a month of demos. The prose came back fine. What the reviewer flagged was a table of torque values where two cells had been re-rendered with a decimal comma in one row and a decimal point in the next, and a footnote marker that had migrated to the wrong sentence. Neither problem appeared in the sample text the vendor supplied. Both are the kind of thing a QA report can be set up to catch once you know to look.

The second example comes from an XLSX price register. The tool translated the unit abbreviations in column D, turning "pcs" into the target-language equivalent in some rows and leaving it in others. The client's ERP import rejected the file. That is a five-minute find in a pilot and a two-day incident in production.

Have the reviewer log post-editing time per thousand words and sort what they fix into your existing error categories, so the pilot output is comparable to the QA data you already keep. If you do not keep that data, your baseline is a guess, and the pilot can only tell you whether the output is tolerable, not whether it is better.

Test how content gets out, not just how it goes in

Every vendor demo is about import. The question that decides lock-in is export. Before approval, confirm you can get three things out of the tool in a format your own stack reads: the translated file, a bilingual source and target table, and whatever terminology you fed in.

A neutral source and target XLSX export is the practical test, because any CAT tool will take a two-column spreadsheet and turn it into a TM you own. TMX is better when the tool offers it. If the only output is a finished DOCX, your TM stops growing the moment you start using the tool, and a year later a repeat client costs you as much to serve as a new one. We have watched a reviewer re-align a 40,000-word translated manual against its source by hand to recover a TM that should have come out of the tool in one click.

Ask the awkward questions while you still have negotiating room. What happens to our files and our glossary if we stop paying next month. Can we delete a project and get confirmation that it is gone. Where is the data processed. How long are uploads retained by default, and is that configurable. Is there an export of our glossary in CSV or TBX rather than a proprietary format.

None of this requires a security certification from the vendor, and we would not treat the absence of one as disqualifying for a small tool. What matters more is whether the answers are specific and written down somewhere you can point to later. A vendor who answers retention questions with "we take security seriously" has told you something useful.

Approve in writing, with scope limits and a review date

A narrow approval gets adopted. A broad one gets argued about for months and then ignored. So write the smallest approval that covers the work you actually have, and schedule the next review rather than trying to anticipate everything now.

What to put in the one-page approval record

The document types and language pairs covered. The clients excluded, drawn from the contract list. Who is allowed to run jobs. What the workflow requires: in our case a glossary prepared before translation starts, human post-editing on everything client-facing, and a QA pass before delivery. What is never sent through the tool: sworn translation, anything under an explicit client prohibition, and document categories your sector rules exclude. The review date, ninety days out. And the conditions that end the approval early, such as a client complaint traced to the tool or a change in the vendor's data terms.

Be specific about the post-editing level you require rather than saying "the translator will check it". ISO 18587 distinguishes light from full post-editing, and the difference between them is roughly the difference between a document that reads as translated and one that reads as written. Name the level you require in the approval record rather than deciding it per job, because deciding per job is how the standard drifts downward under deadline pressure.

Roll it out narrow, then write it into your process

Give the tool to two or three people for thirty days, with one shared place to log anything odd. Not a chat thread that scrolls away. A single document or issue list that the owner reads at the end of the month.

At thirty days, look at four things: whether any client complaint traces back to the new step, whether post-editing time is holding at the pilot's number or creeping up, whether terminology is staying consistent across the people using it, and whether anyone has quietly stopped using it. That last signal is the most informative and the least reported, so ask directly.

Only then does it go into the written process: the intake step where a project manager decides whether a job is eligible, the glossary preparation step, the post-editing requirement, the QA check, and the client disclosure wording if your contract list says notice is needed. If your process is not documented anywhere today, that is the real project, and a new tool will not fix it. We have written separately about documenting an agency's translation process so it survives staff turnover, and it is a better first investment than any tool evaluation.

This approach works best when you have at least a rough QA baseline and contracts you can actually find. It works poorly if every project is a different domain with a different reviewer, because then the pilot tells you about one project rather than about the tool. In that situation, run the vetting per client instead of per agency and accept that you will do it more than once.

Start here

Name the owner today and put the decision date in the calendar, fourteen days out. Tomorrow, open your five largest client agreements and search for the terms listed above, then build the three-column list. Before anyone touches a trial account, write the pass rule as one sentence with a number in it. Pick the pilot files from last month's actual jobs, not from the vendor's samples. At the end, the artefact you keep is one page: scope, exclusions, workflow requirements, review date. If you cannot fit the approval on one page, the approval is too broad to enforce.