This site uses cookies.
Some of these cookies are essential to the operation of the site,
while others help to improve your experience by providing insights into how the site is being used.
For more information, please see the ProZ.com privacy policy.
Multi-modal AI is revolutionising language translation, enabling more accurate and nuanced communication across sectors like business, healthcare, and diplomacy.
In today’s interconnected world, language barriers are becoming increasingly significant as businesses and individuals seek to collaborate globally. The natural way to communicate isn’t through reading or writing; it’s through seeing, listening, and talking. Multi-Modal AI, which integrates text, audio, and visuals, is revolutionising real-time translation and interpretation. This technology can empower society by making knowledge and resources accessible to all, regardless of education or literacy. This article explores how multi-modal AI is revolutionising real-time translation, its impact on overcoming traditional language barriers, and the challenges it faces.
Introduction to Multi-Modal AI
Multi-modal AI combines diverse types of data inputs like text, images, and sounds to generate responses or translations. Unlike traditional AI models that rely solely on one form of input, multi-modal systems leverage multiple data types, allowing for more nuanced and accurate translations. Multi-modal AI not only helps with interpreting spoken languages, but also with contextualising the non-verbal cues such as body language or environmental factors. The convergence of these different data types makes multi-modal AI significantly more effective in fields like language translation, medical diagnosis, autonomous driving, and even creative arts.
According to a report by MarketsandMarkets, the global AI market is expected to grow from $150 billion in 2023 to $1.59 trillion by 2030, and multi-modal AI will account for a significant portion of this growth due to its diverse applications.
The six-title shortlist for this year’s Great Britain Sasakawa Foundation Translation Prize highlights classic and contemporary fiction and nonfiction translated from Japanese into English.
The shortlist for the Great Britain Sasakawa Foundation Translation Prize was announced on December 2, highlighting classic and contemporary fiction and nonfiction translated from Japanese into English. The prize, launched last year by the foundation in association with the Society of Authors, considers books published in Britain between April 1, 2023, and March 31, 2024. The results are due to be announced on February 12, 2025; the translator of the winning title will receive £3,000 and the runner-up £1,000.
Last year’s winning translator Alison Watts appears on the shortlist again for What You Are Looking for Is in the Library, translated from a novel by Aoyama Michiko about a librarian who transforms a series of visitors’ lives through her perfectly pitched book recommendations. The 2023 runner-up David Boyd is also shortlisted in 2024, again for a translation of a book by Oyamada Hiroko. The Factory zooms in on the absurdity of the workplace via three characters with mundane jobs in a surreal setting.
Orange wants to bring manga to as many readers as possible—but some fans are not happy.
A Japanese publishing startup is using Anthropic’s flagship large language model Claude to help translate manga into English, allowing the company to churn out a new title for a Western audience in just a few days rather than the two to three months it would take a team of humans.
Orange was founded by Shoko Ugaki, a manga superfan who (according to VP of product Rei Kuroda) has some 10,000 titles in his house. The company now wants more people outside Japan to have access to them. “I hope we can do a great job for our readers,” says Kuroda.
Orange’s Japanese-to-English translation of Neko Oji: Salaryman reincarnated as a kitten! IMAGES COURTESY ORANGE / YAJIMA
But not everyone is happy. The firm has angered a number of manga fans who see the use of AI to translate a celebrated and traditional art form as one more front in the ongoing battle between tech companies and artists. “However well-intentioned this company might be, I find the idea of using AI to translate manga distasteful and insulting,” says Casey Brienza, a sociologist and author of the book Manga in America: Transnational Book Publishing and the Domestication of Japanese Comics.
There will now be 7 languages besides the official Italian, for the Pope’s reflections and the mediation on a biblical text.
“Next week will begin the translation into Chinese, here at the audience,” Pope Francis announced with visible joy, at the general audience on November 27, 2024. The first translation into Chinese will therefore take place on December 4, the beginning of the liturgical year and first Sunday of Advent.
Every Wednesday, the Pontiff meditates on a biblical text during the general audience, an event open to the public. The audience takes place either in St. Peter’s Square or in the Paul VI Hall inside the Vatican.
Before he speaks, a short extract from the Bible is read out in several languages. After his catechesis, a summary of his speech and the translation of his messages addressed to the faithful in a specific language are also translated.
These translations, carried out by Vatican employees, are currently available in six languages in addition to the official Italian: French, English, German, Spanish, Portuguese, Polish, and Arabic.
Arabic is the latest language to be permanently added by Benedict XVI on October 10, 2012.
Occasionally, languages may be added in response to a special occasion, such as the presence of a group of pilgrims speaking another language – Ukrainian and Slovakian translations were heard recently.
The Connexion speaks to Adriana Hunter as she prepares to tackle the poetic bestseller Son odeur après la pluie
Adriana Hunter has translated over one hundred books and won multiple awards
Adriana Hunter is one of the most experienced British translators of French books with more than a hundred under her belt. She is the recipient of multiple literary prizes and awards.
She has been the official translator for the English version of the Asterix series of graphic novels since 2018, when she replaced Anthea Bell, and has worked on the last four of them.
She loves ‘juggling with words’ and has been honing her craft to bridge the gap between both languages.
This can mean anything from unravelling the intimacy of Amélie Nothomb’s bestselling novels or conveying the play-on-words and puns in Asterix.
The Connexion interviewed Cédric Sapin-Defour for Son odeur après la pluie, an unexpected bestseller telling the intimate relationship between him and Ubac, his Bernese mountain dog, who died in 2017.
He was asked for his opinion about whoever would be chosen for the English translation.
He was not sure how that person would accomplish what he considered an almost impossible task.
Vasco Pedro, CEO of the Lisbon-based startup Unbabel, delivered a provocative forecast at the Web Summit in Lisbon when he said human translators may no longer be needed within three years. The statement came in parallel with the launch of Widn.AI, Unbabel’s new AI-powered translation service built on its proprietary large language model, Tower. Capable of handling translations in 32 languages, Widn.AI represents a significant shift from the company’s earlier hybrid model, which paired AI technology with human editors.
“The advantage humans have in translation is razor-thin,” Pedro said, asserting that AI has reached a stage where it can handle all but the most complex translation tasks. This advancement aligns with a broader trend of generative AI boosting enterprise innovation, as companies increasingly leverage AI for tasks once deemed exclusively human.
Implications for Jobs and Industry
Unbabel’s innovation comes as AI’s potential to replace jobs is sparking heated debates. While Unbabel foresees growth fueled by a surge in translated content, Pedro admitted that the revenue per word is likely to drop. This mirrors broader predictions about AI’s disruptive potential, such as Vinod Khosla’s claim that AI could perform 80 percent of tasks across 80 percent of jobs.
The award will be the biggest of its kind in Europe and aims to celebrate the work of an overlooked and underpaid profession facing an existential threat from AI
Norway is launching a new translation price that is one of the most highly endowed of its kind in Europe, in an attempt to boost a “partly invisible” and often poorly paid profession increasingly under threat from machine translation.
Named after the Norwegian novelist and playwright who won the 2023 Nobel prize in literature, Jon Fosse, the Fosse prize for translators will reward one author every year with 500,000 NOK (£36,000) for making “a particularly significant contribution to translating Norwegian literature into another language”.
Funded by the Norwegian government and managed by the National Library in Oslo, the prize is exclusive to those translating from Bokmål and Nynorsk, the two official written standards of the Norwegian language.
“For a small language like Norwegian, the work of dedicated translators are crucial,” said Aslak Sira Myhre, director of the National Library of Norway. “It is a strenuous, creative and partly invisible work that brings literature to people and cultures closer together.”
Lin King ’22 has won the 2024 National Book Award in Translated Literature for her work translating Yáng Shuang-zi’s Taiwan Travelogueto Englishfrom its original Mandarin Chinese.
Competing in an original pool of 141 entrants in the category, King’s translation was named to the longlist in September, a finalist in October, and finally the winner on Wednesday night at the 75th National Book Awards Ceremony in New York City.
The novel follows the unlikely relationship of two women in 1930s Taiwan, a Japanese writer and her Taiwanese interpreter, as they tour the island nation under Japanese rule. The Translated Literature prize is particularly fitting for the book’s exploration of language, culture, and interpretation.
King accepted the award with Shuang-zi, who delivered remarks in Mandarin, which King then translated for the audience. “Some people ask me why I write about things from a hundred years ago,” King translated. “I always tell them, writing about the past is a means of moving toward the future.
“More than a century ago, some Taiwanese people began making the assertion, ‘Taiwan belongs to the Taiwanese,’” she continued. “Today, many Taiwanese continue to assert this, but now we are addressing it to a different audience. Before, we were saying it to the Japanese. Now, we are saying it to the Chinese.”
On November 14, 2024, the Screen Actors Guild-American Federation of Television and Radio Artists (SAG-AFTRA) announced an updated version of a previous agreement that covers the localization of video game projects produced in a non-English language.
With approximately 160,000 members, SAG-AFTRA calls itself the “world’s largest union representing performers and broadcasters.” The union also represents voiceover artists, including those who provide dubbing.
As comics and gaming website Bleeding Cool reported, the new Independent Interactive Localization Agreement is essentially an updated version of the base terms from the union’s Tiered Budget Independent Interactive Media Agreement, plus AI protections.
The new agreement is signed on a project-by-project basis by employers whose project was originally scripted in a language other than English, and whose intellectual property owner is based outside of the United States.
“Many brilliant, beloved games come to market in the U.S. from other countries, projects which need highly skilled localizing performers,” Interactive Media Agreement Negotiating Committee Chair Sarah Elmaleh was quoted as saying in the press release. Elmaleh added that “[m]any such companies have already signed Interim Localization Agreements”.
The Little Prince, Antoine de Saint Exupéry’s timeless masterpiece, has reached a historic milestone by becoming the world’s most translated book of fiction, with 600 translations to date!
On June 25, 2024, as part of the project “The Little Prince at the Bedside of the World’s Languages”, the 600th translation of this masterpiece was presented to the National Library of Panama, in Dulegaya, the language of the Indigenous Guna people of Northeastern Panama and Colombia. This event reinforces the role of the Little Prince as a universal work that unites peoples and contributes to the preservation of endangered languages.
Since its first publication in 1943 in New York, this philosophical tale, illustrated by Antoine de Saint Exupéry, has touched readers of all generations and cultures, offering a universal message of love, kindness, and humanity.
Today, with many languages on the brink of extinction, The Little Prince uniquely preserves languages and transmits cultures. Thanks to the passion of translators, this work has been translated into rare languages and endangered dialects, thus contributing to the preservation of the world’s linguistic heritage.
The 600th translation confirms the cultural and social impact of the Little Prince, which transcends borders and becomes a link between peoples. This story is more than a story: it is a celebration of cultural diversity. 1,500 copies of this Dulegaya edition, entitled Sagla Massi Bibbi, were printed in the spring of 2024 by the Panamanian publishing house El Hombre de la Mancha. They will be distributed in schools and libraries in the Guna Yala region as of 2025.
NEW YORK, NY, November 18, 2024 (EZ Newswire) — Pronto Translations, a leading translation service in New York, has been integrating generative AI technology such as ChatGPT into its workflows for the past 18 months to support its translation processes.
Following the initial report issued last April, which detailed common AI errors, continuous enhancements in deploying AI have necessitated an update due to the emergence of further drawbacks that impact AI translation processes. Despite ongoing improvements to AI engines, experiences at Pronto Translations confirm that while AI technologies like ChatGPT can assist with translation efforts, they are not capable of replacing human translators. Effective translation requires a collaborative approach between AI tools and skilled linguists. Below are the 10 most critical reasons identified by Pronto Translations:
Mistranslation Risks: ChatGPT generally excels more than many other machine translation tools in identifying the correct contexts for meanings. However, significant errors have been observed, such as confusing “nuts” meant for vehicle assembly with edible nuts, or misidentifying a washer as a laundry appliance in a car maintenance manual. These errors underscore the risks involved in relying solely on AI for translation.
Fabrication of Information: ChatGPT can occasionally generate inaccurate content, especially when dealing with less familiar or obscure terms and concepts. While it handles well-known information from its training data competently, it struggles in areas where the data is scant or the terms are not widely recognized. In such instances, ChatGPT may make educated guesses, leading to translations that are not only imprecise but also potentially misleading. This is particularly problematic in technical or specialized texts where each term has specific and significant implications.
On the first weekend of November, dozens of poets, novelists, translators and other literary aspirants descended on Futaba, Fukushima Prefecture, bringing an international presence to a quiet coastal town on the rebound.
This year marked the 18th edition of the Japan Writers Conference (JWC) and only the second fully in-person event since the COVID-19 pandemic. Across more than two dozen sessions over a day and a half, presenters and attendees grappled with a range of professional and aesthetic concerns in the domain of English-language writing.
Held in partnership with the Futaba Area Tourism Research Association, this year’s program took place in a town still vying to reinvent itself in the aftermath of the 2011 Great East Japan Earthquake. (The Fukushima No. 1 nuclear power plant is located between Futaba and neighboring Okuma and is still in the lengthy process of decommissioning.)
JWC sessions are notably brisk, typically consisting of single presentations followed by a Q&A within 50-minute blocks of time. C.E.J. Simons, a British Canadian senior associate professor of British literature and creative writing at International Christian University, helmed an early morning session on poetry and photography inspired by the post-disaster landscapes of Fukushima, interrogating the ethical stakes of such observation and engagement. The role of climate change in literary world-building and speculative fiction was the focus for Sara Ellis, 57, an American senior assistant professor at Meiji University, who concluded her session with a writing prompt for participants to consider the impact of environmental degradation on cultural memory.
In the wake of Han Kang’s Nobel Prize in Literature, the Literature Translation Institute of Korea has decided to actively pursue upgrading its translation academy to a graduate school. It will also strive to strengthen the global literary network by increasing exchanges with overseas writers, translators, and publishers.
“The Nobel Prize in Literature is not the end, but the beginning,” said Jeon Soo-yong at a press conference on her 100th day at the institute on Monday. “For Korean literature to become world literature, forming international discourse and building a foundation for critique must be strengthened,” she said, emphasizing the need to establish a graduate school of translation.
Currently, the Literature Translation Institute of Korea operates the Translation Academy, a non-degree program for students in seven languages. The plan is to upgrade it to a full-time master’s degree program, aiming to improve the quality of translations and create opportunities for local translators to take on roles in schools and other institutions.
It’s the latest offering from the German tech unicorn
German tech darling DeepL has (finally) launched a voice-to-text service. It’s called DeepL Voice, and it turns audio from live or video conversations into translated text.
DeepL users can now listen to people speaking a language they don’t understand and automatically translate it to one they do — in real-time. The new feature currently supports English, German, Japanese, Korean, Swedish, Dutch, French, Turkish, Polish, Portuguese, Russian, Spanish, and Italian.
What makes the launch of DeepL Voice exciting is that it runs on the same neural networks as the company’s text-to-text offering, which it claims is the “world’s best” AI translator.
As someone who’s just moved to a foreign country, I’m keen to try a voice-to-text translator that actually might work. All the ones I’ve tried so far aren’t real-time — there’s a lag that renders them pretty useless — and the translation quality is pretty poor.
For face-to-face conversations, you can launch DeepL Voice on your mobile and place it between you and the other speaker. It then displays your conversation so each person can follow translations easily on one device.
You can also integrate DeepL Voice into Microsoft Teams and video-conference across language barriers. The translated text appears on a sidebar as captions. It remains to be seen whether DeepL Voice will be available on platforms like Zoom or Google Meet anytime soon.
Award-winning writer, poet and translator Professor Makhosazana Xaba used the second annual AC Jordan commemoration lecture to take a stand for women translators, spotlighting the lack of value afforded to their intellectual labours, especially in respect of African languages.
The annual lecture, instituted in 2023 by the University of Cape Town’s (UCT) School of Languages and Literatures and the AC Jordan Chair in Africa Studies, provides a platform for critical reflections and engaging dialogues, highlighting African intellectual histories, scholars and scholarship across the continent, as part of efforts to advance decolonisation efforts at UCT.
The AC Jordan Chair was established at UCT in 1993, named for Professor Archibald Campbell Jordan, an academic pioneer of African scholarship, literature and linguistics, and renowned for his novel Ingqumbo yeminyanya (The Wrath of the Ancestors).
Taking to the stage to deliver her lecture at the end of October, Professor Xaba quickly dispensed with the original title of her speech, “On Translating The Wretched of the Earth into isiZulu: From Challenges and Pleasures to Epiphanies” – reflections on her recent translation into isiZulu of Frantz Fanon’s landmark text, Izimpabanga Zomhlaba (The Wretched of the Earth).
When it comes to solving climate change, every word counts. From the pitfalls of metaphors to the multiple meanings of the word “energy” – this is how translators at global climate negotiations navigate the language of global warming.
“I remember one morning we returned to our hotel at around 4:00am and slept for two hours. Then we were told that the final document was adopted so we had to rush back to the conference to translate the outcome documents,” says Jianjun Chen, a Chinese language translator at the United Nations, based in Geneva.
He is recounting the frantic hours before negotiators reached a deal at the 24th Convention of Parties to the United Nations Framework Convention on Climate Change (UNFCCC) – or COP24 – in Katowice, Poland, in 2018. Chen, who has worked at the UN for 14 years and translated multiple documents from the UN climate talks into Mandarin, isn’t fazed by the long hours or lack of sleep.
This year’s UN climate talks in Baku, Azerbaijan, will be his 13th. As world leaders gather for COP29 in Baku, Chen and 25 other translators are preparing for a slew of new climate vocabulary to enter the discourse – words that will dictate the ways countries and campaigners approach climate action.
The final text is the result of negotiations, sometimes very intense negotiations. So you have to be very careful about the wording – Jianjun Chen
Chen also translated important documents when the landmark Paris Agreement was signed at the UN climate talks in 2015 (COP21), pledging to try to prevent global warming to well below 2C, with a stretch target of a 1.5C limit. (Read more about why 1.5C is a critical threshold in this story by Martha Henriques). “I was called to start working in the middle of the night at 2 or 3am. Since there was always a tight deadline, we didn’t have the luxury to fall asleep,” he recalls.
Google has announced that it has extended AI voice capabilities to over a dozen new African languages across a range of Google services.
Google — which already supports typing with a custom keyboard in Gboard for approximately 200 African languages, and machine translations for over 60 African languages in Google Translate — now supports voice search, talk-to-type on Gboard, and dictation on Google Translate for 15 regional languages.
The development means that the company has more than doubled the number of African languages that enable speech-to-text in Google Translate and has doubled existing voice input support for Gboard and voice search in the region.
Daan van Esch, Technical Program Manager at Google, said that the update “will enable around 300 million more Africans to use their voice to interact with the web.”
Speaking recently to Slator about the challenges and opportunities of the language services market in Africa, Christian Elongue, Managing Director of Kabod Group said, “there is limited training data that many African languages are facing, [and there are multiple initiatives] contributing to creating data sets for various low-resource African languages.”
Translators have raised concerns over Simon & Schuster-owned publisher Veen Bosch & Keuning’s (VBK) “disastrous decision” to use artificial intelligence (AI) to translate some titles into English.
TheBookseller revealed last week that the Utrecht-headquartered publisher is “working on a limited experiment with some Dutch authors for their books to be translated into English language using AI”. It was later reported in the Guardianthat the “project contains less than 10 titles—all commercial fiction”, and does not include any literary books or titles to which English rights would likely be sold at any point. The story was also followed up on Radio 4’s “Today” programme.
The publisher, which was acquired by Simon & Schuster (S&S) earlier this year, explained that this would include “one editing phase, and [that] authors have been asked to give permission for this”.
Although the Dutch publisher is owned by S&S, The Bookseller understands that the two publishers’ editorial decisions remain separate.
Industry figures have voiced their concerns about the potential “reputational damage” of this move, and the inefficiency that AI could introduce into the translation process.
Louise Rogers Lalaurie, who has translated 15 novels from French and is the author of Matisse: The Books (Thames & Hudson/University of Chicago Press), explained that the “end result” of an AI-generated translation can cost “more than a good human translation first time round” due to the time-consuming “post-editing” process.
Simultaneous machine translation (SiMT) aims to deliver real-time translations as a source language, spoken or written. Traditionally, this requires models that control when to “read” more of the source and when to “write” the translation — decisions that rely on intensive model training, complex model designs, and significant computing power.
Now, researchers Libo Zhao, Jing Li, and Ziqian Zeng from Hong Kong Polytechnic University and South China University of Technology have introduced PsFuture, a zero-shot, adaptable read/write policy that enables SiMT models to make real-time translation decisions without additional training.
The researchers said they drew inspiration from human interpreters, who dynamically decide when to listen and when to speak based on evolving contexts. “Interpreters shift from listening to translating upon anticipating that further future words would not impact their current decisions,” they explained.
PsFuture allows translation models to make similar, context-aware decisions, leveraging “the model’s inherent linguistic comprehension and translation proficiency” and eliminating the need for further training.
Simulated Look-Ahead
Rather than relying on a fixed number of source words to determine the right time to start translating, PsFuture allows a model to anticipate what’s coming next. By using pseudo-future information — a simulated, brief “look-ahead” similar to how interpreters anticipate what might come next in a sentence — the model assesses if additional context would change its next translation output. If not, the model proceeds with translating. If more context is needed, it waits to “read” further.
Veen Bosch & Keuning, the largest publisher in the Netherlands, has confirmed plans to trial the use of artificial intelligence to assist in translation of commercial fiction
A major Dutch publisher plans to trial translating books into English using artificial intelligence.
Veen Bosch & Keuning (VBK) – the largest publisher in the Netherlands, acquired by Simon & Schuster earlier this year – is “using AI to assist in the translation of a limited number of books”, Vanessa van Hofwegen, commercial director at VBK said.
“This project contains less than 10 titles – all commercial fiction. No literary titles will nor shall be used. This is on an experimental basis, and we’re only including books where English rights have not been sold, and we don’t foresee the opportunity to sell English rights of these books in the future,” she added.
“There will be one editing phase, and authors have been asked to give permission for this,” a VBK spokesperson told the Bookseller. “We are not creating books with AI, it all starts and ends with human action.”
The fact that the publisher is planning to use AI translation only for commercial fiction, rather than literary titles, “assumes those books are purely formulaic and don’t contain many creative elements, which is rather insulting to the authors and readers involved”, said Michele Hutchison, who won the 2020 International Booker prize for her translation of Lucas Rijneveld’s The Discomfort of Evening.
“There’s only so far you can get” with machine translation post-editing – the process by which a human translator reviews an AI-generated translation. “The text might be superficially smooth but it is also likely to be very bland,” she added.
Comments about this article 
Share an article »
ProZ.com blog »
