AI Face Tracking Data for 3D Animation Takes Major Step Forward

While perhaps ultimately predictable and a reflection of where AI can meet current 3D character animation technology at where it is, it’s still quite creepy to see AI enter the space of face-tracking data and animation.

That’s because our faces, by definition, are very much a part of what makes us human. We’ve also seen how 3D character animation has struggled with trying to replicate how faces move and talk for decades now.

Still, a new AI study has been released that showcases how facial animation generation can evolve with multi-modality guidance, and that could be big news for those working in 3D character animation. Let’s what AI is unlocking for us next.

AI-Powered Face Tracking Data

With a focus on speech, this new Media2Face innovation aims to bring AI into the high-quality 4D facial data space to help animators and artists get better at recreating the nuances of facial animation.

This new technology makes use of multi-modality guidance to solve many issues such as limited realism and a lack of flexible conditioning. They’ve developed a new Generalized Neural Parametric Facial Asset (GNPFA) to help auto-encode the mapping of facial geometry with the goal of extracting high-quality expressions and accurate head poses for these future facial models.

You can see this new AI at work in the video below that showcases the Media2Face diffusion model which can accept rich multi-modality guidances from audio, text, and images.

The Future of 3D Character Animation

Now, this is exciting and terrifying for many different reasons. On the terrifying side, it of course looks uncanny and quite scary to see the lifeless so accurately recreate the lifelike. But we’ve seen versions of this before in all types of movies, shows, video games, etc… over the years.

It’s more exciting for those who might work in any sort of 3D character animation, whether that be for big game companies or animation studios, or like to tinker around with it themselves for fun on the weekends. AI has made short work of creeping into pretty much all areas of post-production and video editing, so it’s really no surprise to see it move into animation as well.

This new Media2Face model could end up being a breakthrough for facial animation and could help streamline projects by allowing animators to eventually send text-based scripts directly to their models and have them recite lines in perfect, natural animation.

We’ll keep our eyes on this technology as it evolves and updates you if (and perhaps just when) it makes its way to more mainstream programs and use cases.