Using AI to enhance film dubbing

I’ve always had a soft spot for incredibly crappy film dubbing—especially this Bill Murray SNL classic that I hadn’t seen in 30 years but remember like it was yesterday…

…and not to mention Police Academy (“Damn you, wanna fight? Fight me!!“):

Not every example is so charming, however and now a company called Flawless plans to use neural networks to fit actors’ mouth movements to dialogue:

“The extraction of facial data” — a time-consuming computational process — “runs parallel with the production itself.” The technology strips actors’ faces off, converting their visages into a 3D model, according to Lynes. “This creates millions of 3D models, which the AI uses as reference points,” he says.

“And then, using an existing foreign-language recording of the dialogue, it studies the actor and generates a new 3D model per frame,” he adds. Finally, the imagery is converted back to 2D. Digital effects artists can then manually fix anything that seems off.

One thought on “Using AI to enhance film dubbing

Leave a Reply

Your email address will not be published. Required fields are marked *