Li Zi is from Aofei Temple
Produced by Quantum Bit | Public Account QbitAI

How do you get a girl who is facing the camera and standing still to dance the dance you chose for her, showing her 360-degree body posture?

The Facebook team combined the DensePose model for multi-person pose recognition, which is responsible for perception , with the deep generative network, which is responsible for generation .

No matter whose touching posture it is, it can be attached to the girl, turning her single static state into rich dynamic state.

This research result was selected for ECCV 2018 .

Of course, it can't be just DensePose

The team combined the SMPL multi-person pose model with DensePose . In this way, a mature surface model can be used to understand an image.

This study used surface-based neural synthesis to render an image in a closed loop and generate various new poses.

△ The left is the source image, the middle is the source pose, and the right is the target pose

The dance moves that the person in the photo needs to learn come from another person ’s photo or a screenshot from a video.

The DensePose system is responsible for associating the two photos by mapping them in a common surface UV coordinate system .

However, if it is generated purely based on geometry , it will not look realistic because the data collected by DensePose is not accurate enough and there are self-occlusions in the image (for example, the body is blocked by the arm).

△ Texture extracted by DensePose (left) vs texture after restoration (right)

So, the team dealt with occlusion by introducing an image inpainting network in the surface coordinate system . The predictions of this network were combined with a more traditional feedforward condition and model prediction.

These predictions are made independently, and then a refinement module is used to optimize the prediction results. The reconstruction loss , adversarial loss and perceptual loss are combined to complement each other and obtain the final generation effect.

The complete network structure is as shown above.

Supervised learning

The supervised learning process of the model is as follows:

Starting from the input source image, each pixel is mapped to the UV coordinate system. This step is completed by the DensePose-driven migration network.

Then, the autoencoder responsible for repairing the image will predict what the person in the photo will look like from different angles . This prediction is also done in the distorted coordinate system.

Starting from the right, it is to generate the target , which also needs to be integrated into the UV coordinate system. Then use the loss function to process (the red part in the above figure) and input the result into the autoencoder to help the model learn.

Instead of rotating the human body 360 degrees , multiple static poses of the same person (with the same outfit) were used for supervision .

What are the results of the training?

Let's first look at the newly added image restoration step and the resulting effect:

Repairing the texture of DensePose still has obvious effects.

Let’s take a look at what a multi-person video looks like:

Although the face looked burnt, it was still very . I couldn't help but think of:

In addition, the team used the DeepFashion dataset to compare its own algorithm with that of other peers.

The result is that Facebook's algorithm outperformed its predecessors in three indicators: structural similarity, inception score, and detection score .

Please stay tuned for more amazing applications of DensePose.

Paper portal:
https://arxiv.org/pdf/1809.01995.pdf

By the way, there are many algorithms for teaching people to dance.

For example, Berkeley The advantage of turning a dancing idiot into a dancing king is that it is realistic, but the disadvantage is that it is impossible to achieve multi-person dance :



vs