Speech to Speech Translation for unwritten languages

Speech-to-speech translation (S2ST) consists on translating speech from one language to speech in another language, a significant step toward breaking down the global language barrier. In addition to enabling communication between people speaking different languages, Speech-to-Speech translation can also promote knowledge sharing internationally.

As a primary approach, S2ST systems were achieved by concatenating three different systems: Automatic Speech Recognition, Machine Translation and Text-to-Speech Synthesis, which could be error prone and poorly performing. Newer approaches were developed, where researchers have built one-stage S2ST systems that jointly optimize intermediate text generation and target speech generation steps or further remove the dependency on text completely.

The latest approach is beneficial for unwritten languages or with a poor documentation, but it remains a research area with little exploration mainly due to the lack of training data. A breakthrough innovation came from Meta (Facebook’s parent company), which announced the first AI powered speech translation system for an unwritten language, Hokkien, which is a language that is spoken in Taiwan and southeastern China.

Due to the lack of data, Meta used three different sources as training data, including: human annotation, where the people which could speak Hokkien and English were translating different materials, such as drama shows, mined data and weakly supervised data. Also, the advantage that they used was that the Hokkien language is somehow similar to Mandarin, whose resources where helpful in creating test and training data.

As an actual state, the work is still in progress, the system being able to translate just one sentence at a time. As a final scope, Meta announced that the newly build system will serve as a proof of concept for other future systems, hopefully uniting people in the future, regarding the language barrier between them.

 

Bibliography:

[1] https://research.facebook.com/publications/hokkien-direct-speech-to-speech-translation/

[2] https://learningenglish.voanews.com/a/meta-demonstrates-ai-powered-speech-to-speech-translation-system/6806486.html