The person in the photo is out! CloudWalk 3D human body reconstruction tops three lists, and a photo can generate a 3D image

Guo Yipu from Aofei Temple
Quantum Bit Report | Public Account QbitAI

The jumping girl was frozen in mid-air, becoming a flat photo.

Now, importing this photo into a specific program, it becomes like this:

Not only 360°, the 3D image is visible from all angles, as if the girl is about to jump out of the photo.

Previous 3D human body reconstruction often required multiple cameras or continuous multi-frame images to reconstruct a 3D model of the human body.

But now, CloudWalk Technology has refreshed the list in the field of 3D reconstruction, and it only requires an ordinary camera and a photo.

Moreover, since a photo can generate a 3D image, a continuous video can generate an animated film clip.

For example, a dynamic scene like Chen Peisi's "Eating Noodles":

Let's make it 3D and look at it from the front:

View from the side:

View from the back:

From the top of the head, eating noodles from a God’s perspective:

Just add some color and it can become an animated version of "Eating Noodles".

All three datasets ranked first

Yuncong's 3D human body reconstruction technology based on single-frame images ranked first in the three data sets of Human3.6M, Surreal and UP-3D, significantly reducing the original minimum error record by 30%.

△ Human3.6M dataset

△ Surreal dataset

UP -3D dataset

The error of each item on each data set is lower than that of previous studies, which means that the generated 3D model is relatively more accurate and closer to the real condition of the human body.

The surface error (Surface Error) of Yuncong Technology's 3D human body reconstruction technology was reduced from 75.4 mm to 52.7 mm on Surreal, and the 3D Joint Error was reduced from 55.8 mm to 40.1 mm. The 3D Joint Error on Human3.6M was reduced from 59.9 mm to 46.7 mm. The execution speed of the technology was reduced from hundreds of milliseconds to just 5 milliseconds.

Deployed on mobile phones, no 3D structured light required

Yuncong's technology infers the 3D shape of a human body or face by analyzing RGB images, and accurately predicts the position and orientation of each key point in 3D space through basic optical principles such as optical perspective and shadow superposition, thereby obtaining human body posture or expression information.

In addition to predicting the 3D shape and posture of the human body, this technology can also fully depict the human body with more than 60,000 points at a frame rate of 200fps.

In addition, an advantage in deployment is that ordinary optical cameras can be used as perception devices, without the need for continuous images or multi-view shooting.

In other words, if this technology is deployed on a mobile phone, 3D face recognition or the creation of 3D expressions can be achieved without 3D structured light.

Compared with traditional human key point detection, 3D reconstruction technology based on single-frame images can not only output skeletal joint point information, but also simultaneously predict a large number of human body surface key point information. The prediction results are richer, and the coordinates of each point are 3D, which can reflect the depth information of different torsos.