Even if you don’t know how to program, you can still do such a cool video style transfer? This tool has become a hot topic on Reddit, and you can also try it online

Jia Haonan from Aofei Temple
Quantum Bit Report | Public Account QbitAI

In the past two days, a post on Reddit about video style transfer has become popular, and it has topped the hot list of the machine learning community just one day after it was published.

The bright and gorgeous demo amazed netizens.

The famous horror film "The Shining" has been processed to become colorful and comic-style:

The movie Pulp Fiction, on the other hand, looks a lot like the colorful glass windows of a Catholic church:

Everyone left a message asking "Have the layer filters of CNN been modified?"

But the poster said, "I'm not a programmer, and I don't know what CNN is..."

This is a bit incredible. How can a newbie who has never been exposed to machine learning achieve such a good effect of video style transfer?

"One-click" video style transfer tool

The author who reedited the popular post is a pure art practitioner and has never studied computers or programming.

However, he used a "one-click" fool-proof tool to easily complete the video style transfer.

The tool is called Deep Dream Generator .

Readers who are familiar with image style transfer may be familiar with Deep Dream, which was released by Google as early as 2015.

Deep Dream was originally developed for the 2014 ImageNet visual recognition challenge, and its main purpose was to recognize faces and other targets in images.

But then it was discovered that the trained Deep Dream could run in reverse, integrating the pixel features of a given image into the target.

Subsequently, Deep Dream began to become popular in the art creation circle. The style transfer images generated by it are quite dreamy, so they are called "Deep Dream".

The team that created this tool made Deep Dream easy to use, requiring no computer knowledge and can be used directly on the web.

It is very easy to use. You only need to upload the target image and "style" and generate it with one click.

The generated image effect is completely determined by the uploaded "style":

With this tool, even a novice who knows neither art nor programming can mass-produce works of art.

Two methods of video style transfer

Although there is no technical description on the Deep Dream Generator official website, Deep Dream has long been open source, and style transfer is already a familiar area in the application of deep neural networks.

Generally, there are two basic ideas for common style transfer algorithms: one is the optimization method , and the other is the feedforward method optimized on it .

Optimization method

In this method, no real neural network is used.

The neural network is not trained to do anything in this task. It just takes advantage of backpropagation to minimize two defined loss values.

The tensor that is back-propagated to is the image that we want to achieve, which we will call a "replica" from here on out. The artwork whose style we want to transfer is called a style image, and the picture to which we want to transfer the style is called a content image.

The “replica” is initialized to random noise. It is then passed through several layers of a pre-trained image classification network along with the content and style images.

The algorithm uses the output of each intermediate layer to calculate two types of losses: style loss and content loss. In terms of style, the closer the "reproduction" is to the style image, the closer it is in content.

△ Content loss formula

These losses are then minimized by directly changing the "fork".

After several iterations, the “replica” can have the style of the style image and the content of the content image. It is a stylized version of the original content image.

Feedforward method

The disadvantages of the optimization method are high computational cost and long processing time.

So is there a good way to directly utilize the characteristics of deep neural networks to reduce the burden on developers?

The essence of the feedforward method is to create an untrained image conversion network whose function is to convert the content image into the best guess of the "reproduction".

The output of the image conversion network is then used as the “copy”, together with the content and style images, through a pre-trained image classification network to calculate the content and style losses.

Finally, in order to reduce the loss, the loss function needs to be back-propagated into the parameters of the image conversion network, rather than directly into the "replica" result.

Arbitrary style transfer

Although the feed-forward method can generate stylized results immediately, it can only reproduce a given style image.

Is it possible to train a network that can take any style image and produce a stylized result from both images?

In other words, is it possible to create a truly arbitrary style transfer network?

A few years ago, researchers discovered that the instance normalization layer in an image translation network is the only important layer that represents style.

If we keep all convolution parameters unchanged and only learn new instance regularization parameters, we can represent completely different styles in one network.

A team from Cornell University first turned this idea into reality. Their solution is to use Adaptive Instance Normalization, which uses an encoder-decoder architecture to generate Instance Norm parameters from style images, and has achieved quite good results.

Of course, all three methods introduced above require a certain level of computer programming knowledge, but if you just want to try Deep Dream Generator, you can click the portal below:

https://deepdreamgenerator.com/generator

Reference link
: https://arxiv.org/abs/1703.06868