Why is Douyin so popular with so many short video effects?

Xiaoxiao sent from Aofei Temple
Quantum Bit Report | Public Account QbitAI

When it comes to short video special effects, you may think of "Tik Tok special effects" first.

Yes, Douyin’s special effects always seem to be popular, like using a video to make you “return to the age of three”:

△ Video source @Lai Meiyun

Or one move breaks the dimensional wall, and PDD also becomes handsome:

In fact, there are many APPs at home and abroad that are using AI-related special effects technology, but after running them on mobile phones, Douyin is often the most popular one.

For similar special effects, the AI ​​technology used behind them is actually similar, but the special effects produced by Douyin are more realistically rendered, run more smoothly, and have more popular products.

What is the secret behind the evergreen effects of Tik Tok?

Only 5M, the effect is comparable to Unreal Engine

The creativity of a special effect is unpredictable, but the rendering effect and computing time can be polished to the extreme.

Among them, the rendering effect depends on the improvement of two types of algorithms: CV (computer vision) + CG (computer graphics) , while the calculation time depends on the optimization of the model.

For computers, CV algorithms allow them to accurately "understand" the content of a photo. From the fit of special effects and the accuracy of real-time tracking to the resolution improvement and style change of the entire image, all fall within the scope of CV algorithms.

Special effects like synthesizing a pineapple to stick to the face, high-definition restoration of old photos, or overall style migration such as "returning to three years old" and "comic face" are all within the "responsibilities" of CV algorithms.

The CV algorithms used by Douyin are basically self-developed. At the CVPR 2021 conference alone, ByteDance had 14 papers selected for collaboration , covering areas such as NAS, image restoration, and behavior recognition. At the same time, it won championships in competitions in sub-fields such as mobile object detection, fine-grained image recognition, and video classification.

Speaking of CG algorithms , they allow computers to create lifelike special effects, and combine them with CV algorithms to fit them to pictures or videos.

Like the common PBR algorithm, it is just a basic rendering operation; the particle system commonly used in movies is also used by Douyin in mobile phone special effects. In addition, there are transparent materials, bloom effects...all are closer to real scenes, not just 2D stickers.

△ Comparison of 3D rendering and 2D procedural flame effects

However, whether it is the CV algorithm or the CG algorithm, the larger the model, the better the rendering effect.

There is even a saying in the film and television industry: "If a top-notch special effect of one second is achieved using a professional rendering algorithm, it may take 50 days of continuous rendering."

Such a special effects model absolutely cannot meet the user's real-time shooting needs, let alone smoothness.

But according to Volcano Engine technical expert Yang Hui, TikTok 's 5MB rendering effects can achieve results comparable to those achieved by Unreal Engine .

That’s right, in order to ensure that the special effects run smoothly, Douyin’s special effects model is strictly limited to 5MB to facilitate real-time use by users.

This 5MB includes all files including models, textures, animations, special effects and logic trigger mechanisms.

Behind the extremely compressed computing time is the combination of self-developed rendering engine + cloud rendering technology.

In terms of rendering tools, Douyin did not choose to use Unity or Unreal Engine for special effects production, but developed its own lightweight rendering engine for short videos and AR .

The self-developed engine focuses on enhancing the special effects algorithms required for short video shooting, discarding some special effects that are exclusive to games and not needed for short videos to ensure the rendering effect.

At the same time, in order to use better rendering algorithms for special effects, Douyin developed its own Cloud Rendering technology, which puts the special effects that were originally calculated by mobile phone hardware on the server side for calculation, and then transmits the effects to users in real time.

In this way, rendering effects that originally required GPU-level computation can now be completed in real time on mobile phones, and the resulting improvement is obvious.

For example, the particle system in the rendering algorithm can only play a better effect (fireworks special effects) if it is calculated by GPU ; in contrast, the particle system calculated by CPU has a weaker effect (rain control special effects) :

△ Fireworks effects in 2021 vs. rain control effects in 2018

However, cutting-edge CV+CG algorithms and self-developed tools are only part of the secret of Douyin's special effects. The reason why it can produce so many hits is inseparable from its mature production process and industrialized system covering all scenarios.

Volcano Engine: Technological Blowout

Whether a technology can be evergreen depends on whether the products it brings can be mass-produced and whether its applicable scenarios are comprehensive. The same is true for special effects. Douyin special effects already meet these two conditions:

On the one hand, according to the "Douyin Special Effects Data Report" jointly released by Douyin and Volcano Engine, in the first half of 2021, Douyin platform launched more than 100 special effects hits on average every day, and 1 out of every 5 submissions by users used special effects.

This means that Douyin already has the ability to mass-produce special effects. At the same time, the quality of these special effects is enough to make one-fifth of the submissions use them (DAILY users of Douyin have now reached 600 million) .

This also makes special effects design easier. Before Douyin turned special effects into a "creative production line", special effects designers were not a "low threshold" profession. According to Douyin's popular designer You Yang, he was originally a film and television post-production, but successfully became a Douyin special effects artist, and more and more colleagues around him are devoting themselves to this profession.

On the other hand, the application scenarios of Douyin's special effects technology are no longer limited to the Douyin APP, but have already touched all walks of life.

In the automotive industry, AR car viewing is achieved with special effects ; in the beauty industry, AR makeup trials are performed with special effects ; in the traditional media industry, virtual hosts and virtual customer service are created with digital human technology...

△ AR makeup trial

It is foreseeable that Douyin special effects are no longer the special effects originally used when shooting short videos, but are gradually becoming a kind of "creative digital content."

The popular TikTok special effects have become another indispensable part of ByteDance’s growth secret.

Now, ByteDance has made them all public — in the form of Volcano Engine .

That’s right, just like Feishu, the “boom” product of ByteDance’s management capabilities, the boom in ByteDance’s technical capabilities brought about Volcano Engine.

As a technology platform that ByteDance used internally, Volcano Engine not only "stores" the algorithmic technology capabilities behind all of Douyin's special effects, but also includes all the engineering technologies for special effects from production, quality control to implementation .

In terms of production , in order to speed up production and lower the technical threshold, ByteDance has transformed the algorithm technology behind Douyin special effects into various modular components.

For example, after Douyin made the commonly used CV and CG algorithms into components, designers only need to drag the reading bar or press a button to use this function in special effects.

For the entire algorithm production process, from training to inference, Douyin uses its self-developed inference framework and training platform to ensure the most suitable running speed for the APP.

Quality review Special effects designers will be rated for their effects, and only those with sufficient qualifications will be allowed to go online; at the same time, ByteDance has also established a special innovation team to ensure the cutting-edge and innovative nature of the algorithm and design more components that meet the needs of designers (many top conference papers come from here) .

On the ground , Douyin will also consider the user experience and the customized needs of customers in different scenarios.

For example, Douyin has adopted technical measures such as grading strategies for different mobile phone performance and optimizing operating speed to reduce the system crash rate to one in 100,000.

At the same time, technologies such as resource distribution are used to ensure that users can use new special effects without updating the software; when facing different scenarios, Douyin special effects can also adapt to different resolutions, interface types and models.

All of this, together with cutting-edge algorithm technology, constitute the secret to the success of Tik Tok special effects.

“Byte Technology has no secrets”

On July 28, Volcano Engine and TikTok jointly announced that tens of thousands of the same special effects as TikTok will be available to customers.

This means that ByteDance is providing technical services to the outside world with all the advantages of Douyin special effects, including algorithms and prop resources, so that more people can enjoy the results of this culmination.

At present, a number of enterprise products such as Dongqiudi and Xinxin APP are already using Volcano Engine as their business growth technology.

In the future, Volcano Engine also plans to further introduce the latest technologies such as end-cloud collaboration and multi-person interaction to enhance the special effects experience of users and customers.

It can be imagined that more sophisticated special effects and more interactive technologies will become the future development direction of Douyin.

At the same time, the development of special effects technology will further bring more surprises to various industries.

Taking the film and television and gaming industries as an example, these two technologies can greatly reduce video post-production and game art costs. For example, adding a mosaic to a character or making a smart sticker no longer requires manual adjustment frame by frame, as the algorithm can automatically complete these steps.

By publicly disclosing all of its growth technologies, is ByteDance afraid of competition from its peers?

Yang Hui said he was not afraid:

As time goes by, the entry threshold for any new technology will become lower and lower. What we can do is not to engage in price wars, but to expand and update technology to ensure our own competitiveness.

From this perspective, there is indeed no secret to ByteDance's growth technology.