Today, I want to explore a use case that we are seeing more and more in the localization space. Content is not always localized from a source (usually English) into a set of target languages. Instead, we are getting content generated directly in the target languages.
This can come from multiple scenarios (LLM chat-style responses, information about a topic generated directly in the target language, etc.) that have something in common: there is no English content to evaluate it against.
This is a great opportunity, since it opens the door to experiences in languages other than English that truly resemble the needs of global users, but it also comes with challenges.
The first one is how to evaluate the quality of that content. The tools we have (Translation Management Systems [TMS]) are ready to evaluate quality in relation to the source version the content was generated from (usually English), but they were not made to evaluate the quality of target content per se. There are tricks you can implement, like setting up a Language Quality Assessment (LQA) schema based on Multidimensional Quality Metrics (MQM) that only takes the target language into consideration, but the main challenge remains: we also need to provide “an English version” for that target content.
The main question, especially for those who are not familiar with how a TMS works, is why we cannot simply pair the target-language content with a source version and send it through the usual flow. The reason is that these tools were built for reuse. Content is split into segments (usually sentences), and each source segment and its translation are stored and automatically reused wherever the same source segment appears. If we create an artificial English source for content that was generated directly in the target language, that pairing can be reused elsewhere as if it were a validated translation. On top of that, someone has to generate that English pair for every piece of target content, which most likely means an LLM doing a back translation, adding time and cost (tokens). We would then need to double-check every translation the system provides, which is not scalable, and the artificial pairs would pollute the Translation Memory (TM) every time the operation is repeated.
If this content needs to be reused, a safer approach is to keep it out of your main TM and store it in a separate one used as reference material only, after running basic automated checks.
A second challenge is that TMS platforms allow us to review and edit content at the segment level, including adding comments and making suggestions. However, we cannot currently use these inputs to improve how AI-generated content is created. It would be beneficial if we could provide feedback or refine prompts based on our edits and comments. That way, the AI could incorporate this input into both current and future content generation.
A third gap is that we currently lack functionality that shows which models perform better when a certain content type is generated in a specific language. The same goes for knowing whether the quality is good enough to generate content directly in a target locale instead of relying on pure AI translation.
What I am seeing is that the line between content management systems (CMS) and TMS is getting blurrier and blurrier. There are certain flows that humans have always handled (like generating locale-specific marketing content) that are being automated more and more. To do that, we need a tool that ensures glossaries, style guides and quality requirements are met.
For that reason, I wouldn’t be surprised if in the future CMS companies were acquired by TMS companies, or if TMS companies started their own endeavor, allowing customers to choose between different approaches:
- Generate content directly in a particular locale following certain style guides and glossaries (flagging style discrepancies with previously generated content).
- Compare LLMs to generate that content in a specific locale for a certain content type.
- Use a standard localization workflow (AI translation with human review, or human curation) if the quality of the generated content doesn’t meet a defined threshold or its visibility is too high to take risks with just AI-generated content.
As localization evolves and less human translation is needed, I see a growing need for experts who can evaluate the quality of the content and adjust the models to deliver the output a specific scenario requires. Beyond that, we will also need people who make sure the user experience in a particular language and market is the best it can be. Ideally, the same experts will take on both responsibilities.
Leave a Reply