WATCH | Is that Tom Hanks speaking Japanese? No, it's just AI

20 May 2021 - 08:36 By Reuters
subscribe Just R20 for the first month. Support independent journalism by subscribing to our digital news package.
Subscribe now
Start-up Flawless AI has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.
Start-up Flawless AI has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.
Image: 123RF/travelman

Bad lip-syncing in dubbing and subtitles can put off audiences and hurt box office takings of foreign films.

AI may be about to change all that.

Start-up Flawless AI, cofounded by film director Scott Mann, has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.

The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French.

Mann was inspired to come up with the tool when he saw how dubbing affected the narrative cohesion of his 2015 film Heist, starring Robert De Niro.

“I hate dubbing as it stands,” he said. “You have to change so many things to try and catch sync. You're changing words filmmakers and performers have thought about so deeply. They're thrown out to find a different word that fits, but it never really does.”

Mann decided to do something about this conspicuous mismatch. After some research, he discovered a white paper by Christian Theobalt of the Max Planck Institute for Informatics, laying out a new approach that uses artificial intelligence to recreate photo-real human visual effects.

The tool worked to translate a De Niro scene in Heist.

“It's able to essentially take an 'ooh' sound from De Niro 20 minutes earlier and place that in a different moment in the film,” explained Mann. “It measures at the same time and blends it so that the performance is the same but it's a different mouth movement.”

Flawless AI is now working with producers and studios to integrate the technology into post-production. The first films using this tool are around a year away.

Mann believes this technology will be a game-changer, helping to eradicate foreign-language categories at award shows and creating a more inclusive film industry with a real global stage.

“From a filmmaking point of view, you're going to see the rise of a much more diverse range of stars,” he said.

There might be a dissonance and a focus on the AI technology in the beginning when it is introduced, but eventually Mann hopes its use will be seamless.

“When it's a common thing, we're not going to think about it any more and we'll just enjoy the movies,” he said. 


subscribe Just R20 for the first month. Support independent journalism by subscribing to our digital news package.
Subscribe now