Your graffiti comes alive, CMU Chinese undergraduates make it come true | Code and demo

Guo Yipu from Aofei Temple
Quantum Bit Report | Public Account QbitAI

In the mortal world, paintings are static. But in the mysterious computer world, everything is different. If you don't believe it, just look at it.

Draw a unicorn and tap it.

The unicorn "comes alive", running and jumping, leaping onto the paper.

Give it a name, and then tap it lightly - the unicorn with a name will enter a three-dimensional world from the paper, and begin to climb over hills tirelessly...

There are all kinds of strange creatures living in this world.

For example, a snail dancing in the crowd.

Look, a monster with four legs:

The fisherman walking in the sky must be an immortal:

Ghost gentleman with hat

Even the grass and trees can become spirits:

Cartoon characters are also OK, you can see Doraemon:

The yellow-skinned mouse Pikachu, this time without hair:

And the magical big square face:

These little guys were all drawn by the spectators.

In this virtual world, with just a few graffiti strokes, everything you draw becomes real, which can be called the real magic brush. They run in the same world, ghosts and monsters entangled, just like a hundred ghosts parading at night.

This is the graffiti world created by Lingdong Huang, a Chinese boy who is currently studying at CMU. In this world, anyone can draw something like Ma Liang, the magic brush, and it will come true immediately.

Many people were amazed after seeing this:

This is fucking magic.

I love these doodles so much, I could look at them for days haha.

It's like a poem, full of flesh-and-blood images, floating in a natural and casual place.

How did you do it

They are clearly random scribbles, but they can automatically distinguish the "arms" and "legs" of the little monsters and move rhythmically. How do these graffiti come to life?

"Creator" Huang used three steps to complete the construction of this world.

Adding a skeleton to the doodle

Just like the key point detection of the human body, the doodles drawn casually also need key points to determine the position of the bones in order to depict its spirit.

Here, Huang used OpenCV's skeletonization tool and implemented it with Zhang-Suen's C++ code. However, this implementation traverses all the effects in the image, so the effect achieved on the web side is stuck and slow.

Huang had to use gpu.js again and rewrite the code using the JavaScript branch of WebGL shaders.

After the rewrite, it is much faster. For example, if we put a photo of Black Widow holding a gun, it can automatically outline the focus of the picture, Black Widow's torso and the two arms holding the guns.

Making machines understand skeletons

But now comes the problem. When seeing a picture like this, humans can tell that it is a person and his/her two arms, but machines don’t seem to be able to do that.

Therefore, Huang decided to use an 8×8 window to scan the entire image, find the parts with strokes and mark them as the root patch. Then, look at the four sides of the small square of the root patch and extend them to the side where there is a line. In this way, the entire skeleton can be found, and then an aggressive median-blur filter is used to reduce noise.

Now, machines can finally understand the shape of your doodle and its internal skeletal structure.

Make graffiti move

Now we need to find a way to make the graffiti move.

Here, Huang identified five major species: mammals, humans, birds, fish, and plants. Based on the skeleton of the graffiti image, he determined which species the drawn monster belonged to, and then assigned body organs to different parts of the skeleton: this is the head, this is the arm, this is the leg...

Afterwards, he set separate "walking" postures for different species. For example, plants should jump in a fixed place, mammals and humans should run on the ground, birds should flap their wings in the sky, and fish should swim in the water by wagging their tails.

Following this process, graffiti can be put into action.

By the way, the “sky, earth, and water” in the world where the graffiti monsters live also need to be defined.

Huang used a 2D Gaussian function multiplied by Perlin noise to generate a map similar to an island in the sea, with high in the middle, low around, and water on the outermost layer. In this way, flowers, birds, fish and insects can all find their place.

AI Funny Lovers

Huang will graduate from CMU next year. Before that, he was an intern at Disney. Born in 1997, he has created many magical projects.

For example, automatically generate landscape paintings:

There is also the Emoji villain war:

Automatic calligraphy generator:

Quite interesting content, you can click into the portal to experience it.

Professional "Ma Liang" company has been launched

The ability to make graffiti animated, which is similar to the magic brush of Ma Liang, has actually been commercialized by a company.

Xiaoxiaoniu Technology, founded by Dr. Cao Xiang, a former Microsoft human-computer interaction expert, is such a company. They use their self-developed Wonder Painter technology to turn various objects into animated characters.

For example, draw a girl in a skirt in the drawing interface:

Next, the girl can start dancing directly:

Xiaoxiaoniu's technology has been applied in some online marketing activities in the form of games. In the future, this kind of technology may also shine in the fields of art creation and children's education, which will greatly lower the threshold for artistic creation.

Therefore, "The Magic Brush of Ma Liang" is not only cutting-edge technology, but also has promising future business prospects.

Portal

Demo experience:
https://doodle-place.glitch.me/

I almost forgot to mention that because this project is so popular, sometimes you need to queue up like Didi to enter. The Pikachu and Doraemon at the beginning are in the second Demo server:
https://doodle-place-server-2.glitch.me/

Original blog:
http://golancourses.net/2019/ngdon/03/05/ngdon-drawingsoftware/

Skeletonization code:
https://skeletonization-js.glitch.me/