Exclusive Interview | The birth of the painting robot Andy and Meitu Xiuxiu’s ambition in artificial intelligence technology

Recently, Leifeng.com found that many celebrities and users have shared their own illustrations on Weibo and WeChat Moments. The outlines of these illustrations are almost the same as the real person, and the expressions are somewhat similar. But upon closer inspection, these illustrations seem to be from the same artist. Which artist can draw thousands of portraits at the same time?

It turns out that this painter is artificial intelligence.

These portraits are from Meitu's latest product, the AI ​​painting robot Andy. As a robot with painting talent, Andy can draw an illustration for the user based on the user's selfie.

What is the artificial intelligence technology behind Andy? Why does Meitu, a photo editing software, want to focus on artificial intelligence? Leifeng.com interviewed Jiang Han, the product manager of Meitu XiuXiu, and MTlab, which provides technical support for Andy.

The new technology behind Andy——MTgeneration

Leifeng.com learned that Meitu XiuXiu has been engaged in the research and development of artificial intelligence. In 2010, Meitu established the Meitu Imaging Laboratory (MTlab). In 2012, MTLab began to focus on artificial intelligence and deep learning.

Meitu CTO Zhang Wei once said in an interview with Leifeng.com: "I spend 70% of my time on it... We feel that if this company doesn't work on AI in three years, it may be empty."

Andy is the interim achievement of MTlab technology. MTlab personnel told Leifeng.com that the technologies implemented by Andy include MTlab's MTface (face technology) and MTsegmentation (image segmentation technology), as well as the latest image generation technology (MTgeneration). The core of image generation technology (MTgeneration) is based on the generation network Draw Net independently developed by MTlab. Through big data and deep learning, Draw Net can build painting models. These models include artistic styles and drawing rules at different levels, from composition to brushstrokes.

If we break down Andy's technology, we will find that Andy makes illustrations in this way: First, Andy studied a large number of illustration works, and based on this, he created a general painting model. After Andy saw the user's picture, through MTface technology, Andy can grasp the outline, position and characteristics of the face. In addition to MTsegmentation technology, Andy grasps the hair, clothes and background areas. Finally, the painting model is used to express the mastered features, and a finished picture is obtained.

From Andy's works we can see his grasp of the characteristics of the painting model. MTlab staff introduced, "Take the Garden style as an example. For the faces drawn by Andy, the eyes will not be deliberately drawn with double eyelids, and girls will have some blush under their eyes to look cute. This is a common feature learned from the previous data. Andy will combine these features to paint user photos to form an illustration style." Because of these settings, many users have also commented that Andy's brushes also have the usual beautification function.

Andy's technology and product form have been gradually accumulated. In early 2017, Meitu launched a "hand-drawn selfie" product. MTlab staff said that hand-drawn selfies initially used artificial intelligence, but users responded very well to this product at the time.

(Beautiful picture hand-painted selfie)

Hand-drawn selfies mainly use face technology and image segmentation to beautify the face, giving people comic-style magical luster hair, oversized shining eyes, and nearly transparent skin.

We also see that there are many ways to draw with computers: Google has launched AutoDraw, which matches the user's sketches with the existing image database to generate a complete painting; while Prisma mainly adds style filters to the user's photos. Andy is the first time that a computer is completely used to draw and copy a photo.

Andy is an application of the mature MT generation technology. Before Andy, Meitu also used MT generation technology to make a "mixed-race" application. Through big data and deep learning, Meitu has mastered the facial features of people from different countries, and then used Draw Net to generate a network to convert users' Asian photos into European photos, so that users can see what they would look like if they were European.

When asked about how Draw Net learns from large amounts of data, MTlab staff said that this part is still confidential. Meitu may publish a paper to explain it later, but it is still its technical barrier.

MT generation can be said to be the latest technology of MTlab. MTlab staff introduced that MT generation technology can also be used in other application scenarios. For example, "AI beautification" will directly help you adjust a perfect face that suits you, that is, it can learn the aesthetic habits of Asians and then directly optimize the face. Simply put, it is comparable to plastic surgery.

From new technology to implemented products, Andy's R&D process was particularly tortuous. He experienced many failures during the exploration process and the R&D cycle was also very long. There was almost no progress in the first few months. It was not until the past two months that he saw results and then optimized step by step.

This image shows Andy's early paintings compared to his current ones.

This also shows that AI painting is a long-term learning and iteration process. Andy learned for 6 months, while it takes about 2-3 years for humans to learn from scratch. At present, we can see that Andy can only roughly draw similar outlines when painting, which is not fine enough. Andy cannot recognize glasses, earrings, clothing details, etc. However, Jiang Han also said that users are currently more tolerant of new things. If Andy wants to continue, he needs to study harder.

How long will the painting robot Andy be popular?

There have been many phenomenal products in the beauty category: Face Meng, Prisma, "Your Name" filter, etc. However, we have seen that these products were popular in WeChat Moments for a while, but were quickly forgotten.

From the emergence of various hits to being forgotten, it reflects that the beauty industry needs to constantly pursue new technologies and new ideas. Jiang Han told Leifeng.com, "All Meitu products are centered around 'beauty', and 'beauty' is a fashion, and users always like new things and dislike old things. If Meitu stays at the current stage, it will be eliminated by the market sooner or later. This is a question we have been thinking about."

Jiang Han said, "Every user wants to have an illustration of their own. This is a good landing point for artificial intelligence. Andy was created to meet this universal desire."

After finding the application scenario, how can we make the Andy product last long?

Jiang Han told Leifeng.com: "Meitu currently has two directions: one is professional, and the other is entertainment. Hand-painted selfies and the painting robot Andy are both phenomenal works. But if you have good ideas and good results, you can also extend the vitality of this phenomenon indefinitely. Meitu has done a lot of explosive projects since the launch of the mobile version in early 2011, and we will continue to explore the tipping points. The professional path is to provide basic image functions to meet user needs in the long run."

Leifeng.com also found that Meitu Xiuxiu has launched a new "Meitu Black Technology" module, which includes "Hand-painted Selfie (Cross-Dimension Camera)", "Dream Doll Machine", "Girl Photo Gallery", and "Painting Robot Andy". These products are entertainment functions upgraded on top of basic functions.

Jiang Han introduced that Andy has great potential in business and can play a big role in Meitu's personalized customization. Meitu launched "Meitu Customization" in the first half of the year. After users use Meitu Xiuxiu to edit their photos, Meitu will automatically generate a variety of customized products: mobile phone cases, pillows, T-shirts, etc. Master Jiang believes that Andy's portrait illustrations are very suitable for customized products for users.

In addition, the technology developed by Meitu itself has also begun to take the B2B route, which is also a kind of commercialization. Andy's illustrations may take a long time to optimize, but the MT generation technology behind it can be applied to more scenarios. It is said that MTlab will focus on the research and development of Andy and MT generation technology for a long time. Their ultimate hope is that Andy can surpass illustrators and become a master painter in the AI ​​world.

It is a long process for artificial intelligence to understand semantics and images, and people have always been keen to invest time in language and image expression. Although it cannot really simulate human-to-human conversations, people are also keen to communicate with Siri, Cortana, and XiaoIce, and tease them from time to time.

Perhaps, teasing Andy's painting ability has become an interesting thing.