Facebook develops new AI model that can fake dynamic character videos

Recently, Facebook published a paper on extracting characters from real videos, Vid2Game: Controllable Characters Extracted from Real-World Videos.

In this paper, Facebook proposed a new AI model that can locate and extract dynamic characters in original videos. The extracted characters can be controlled by developers, and new videos can be generated by adding any background.

1. How does AI combine real-life videos with game characters?

The team’s approach relies on two neural networks, or layers of mathematical functions that mimic biological neurons: Pose2Pose, a framework that maps the current pose and a single-instance control signal to the next pose; and Pose2Frame, which puts the current pose and the new pose (along with a given background) together on the output frame. These operations can be controlled by any “low-dimensional” signal, such as from a joystick or keyboard. The researchers say the system is powerful enough to localize extracted images in a dynamic background and generate video outputs by arbitrarily replacing existing backgrounds and fusing them into new images.

There are two specific steps to generate new videos with this AI model. First, an input video containing one or more images is fed into a Pose2Pose network trained on a specific area (such as dance), which isolates these images (plus an estimated foreground spatial mask) and their motion, with the specific area being the trajectory of their center of mass. (The mask is used to determine which areas of the background are replaced by the synthesized image information.)

Then, using this combined data, Pose2Frame separates character-related changes in the scene (such as shadows, held objects, and reflections) from changes that are not related to the character, ultimately forming a mixed video output of any desired background and action.

2. Extracting controllable characters through training comparison

To train the AI ​​system, the researchers collected three videos, each between five and eight minutes long: a tennis player outdoors, a person swinging a sword indoors, and a person walking.

Comparing the three videos to a neural network model fed a three-minute video of the dancers, the researchers report that their approach successfully compared dynamic elements such as differences in other people and camera angles, as well as changes in the characters’ costumes and camera angles.

"Each network solves a previously underserved computational problem while paving the way for generating video games with realistic graphics," the researchers wrote in their paper. "Furthermore, controllable characters extracted from YouTube-like videos can find their place in virtual worlds and augmented realities."

3. Using AI systems to optimize games has become a trend

Facebook isn't the only company working on AI systems that could help with game design.

Promethean AI, a startup AI technology company, uses machine learning to help human artists create artistic images for video games, using automation to eliminate repetitive modeling work.

Researchers at AI visual computing company Nvidia recently demonstrated a generative model that can create virtual environments using video clips.

Machine learning has also been used to optimize old game textures in retro games like Final Fantasy VII and The Legend of Zelda: Twilight Princess, as well as to generate thousands of levels from scratch in the game Doom.

Conclusion: AI technology has become a new breakthrough in the industry

AI has been increasingly widely used in various fields, and various industries are using AI technology to find new breakthroughs.

In the field of image processing, AI image technology is helping the gaming industry optimize or develop more interesting products, and is gradually becoming another important factor in competition in the gaming industry.