The father of LSTM has now joined the Ghost Animal All-Stars, "Why didn't he win the Turing Award? It's so unfair!"
Gui Lizi from Aofei Temple
Quantum Bit Report | Public Account QbitAI
Ever since the three giants of machine learning, Hinton, LeCun, and Bengio, won the Turing Award, one person has been ruined.
His name is Juergen Schmidhuber , and he is also known as the father of LSTM .
(LSTM, long short-term memory, is an important machine learning method that can learn long-term information.)
At that time, when the news of the Turing Award was announced, the whole building was shouting his name.
Things are different now. Someone wrote a LSTM song for him to better support him (wrong) :
The material comes from a TED X talk.
Maybe not as professional as the contestants in the ghost animal area of Bilibili, but in terms of brainwashing, it is not inferior. The lyrics are roughly as follows, please watch it with the video:
Has anybody never heard of the LSTM?
Has anybody never heard of the LSTM?
Has anybody never heard of the LSTM?
Human level intelligence
.The LSTM is a little bit like your brain
.learns a little bit here and learns a little bit there.
learns a little bit here and learns a little bit there.And my first publication on that dates back 30 years, 1987
.The long short term memory.The
long short term memory.
Another industrial revolution.Another industrial revolution .Many many many, many many many people are listening
.
Trillions of replicating humans make the universe intelligent. Trillions of replicating humans make the universe intelligent .Animal-like
AI.
Everything is going to change .Has anybody never heard
of the LSTM?Has anybody never heard
of the LSTM?Has anybody never heard
of the LSTM?LSTM working for Facebook .
Is anybody on Facebook?
LSTM working for Facebook .
Is anybody on Facebook?
The
long short term memory.The long short term memory.
A new type of life is going to be mandatory.The
long short term memory.The long short term memory.
A new type of life is going to be mandatory.
billion human brains, taken together .I have no doubt in my mind
.Is anybody doing that
?I have no doubt in my mind
.Is anybody doing that
?(I didn't hear the last brainwashing cycle clearly)
There is even an illusion that the protagonist in a Disney movie is singing while walking. This little song has received more than 300 likes on Twitter:
The author of the video said: "We think it's unfair that Schmidhuber didn't win the Turing Award. In order to cheer him up, we made this song."
So, there are so many big names in the world who have not won the Turing Award, why was the father of LSTM the only one to be ruined? And why did the lyrics only mention Facebook?
“My groundbreaking work”
In fact, in recent years, the father of LSTM has often attracted worldwide attention because of his fierce expressions such as " his pioneering work has not received enough respect ."
So, let me first describe here what a groundbreaking work LSTM is.
Just like the song goes, LSTM is the abbreviation of Long Short-Term Memory , which is somewhat similar to the memory of the human brain.
It is a special type of
Recurrent Neural Network
(
RNN
), first published in 1997. So, what makes it special?
Ordinary RNNs can apply previously learned information to current tasks, but the farther away the information is, the harder it is to associate it. Quoting an example from Christopher Colah’s blog:
To predict the last word “ sky ” in “the clouds are in the sky ” , there is little need to consider the context;
If you want to predict the " French " in " I grew up in France ... I speak fluent French ", ordinary RNN knows to fill in a language, but it is not easy to connect to the distant information " France ".
LSTM was created to solve this problem. The difference between it and ordinary RNN is:
The recurrent part of a normal RNN has only one layer ( tanh ), which changes only with the change of state h, so it is sensitive to short-term input:
△ Ordinary RNN, from Colah's blog
The LSTM loop has a c-state to store the long-term state. Therefore, the loop has 4 layers to determine how much the short-term state affects the long-term plot.
△
LSTM, from Colah’s blog
In this way, LSTM can learn longer sequences. This is how a problem that has long plagued the academic community was solved.
The paper published in 1997 was written by Schmidhuber (second author) and his student Hochreiter (first author).
Later, LSTM was developed and promoted by many parties, and now it has become an indispensable method in the field of machine learning.
Google uses LSTM in speech recognition and Google Translate; Apple's Siri and Amazon's Alexa are also based on LSTM.
Facebook uses LSTM to perform 4.5 billion automatic translations per year. This is the data from 2017.
△ From Reddit user @TheOverachiever
'Not getting enough attention'
But the more important this technology is, the more Schmidhuber, one of the inventors, feels that he is not getting enough attention.
After all, the three giants of machine learning have long been famous all over the world, but his name is far less famous than theirs.
Far from the center of the universe, he is co-director of the Dalle More Institute for Artificial Intelligence ( IDSIA ) in Manno, Lugano district, canton Ticino, southern Switzerland .
In 2015, to commemorate the 60th anniversary of artificial intelligence, the three giants co-wrote a review of deep learning and published it in Nature.
As everyone was reading it carefully, the article received an objection from Schmidhuber:
He listed nine issues, criticizing the Big Three's review for not respecting enough the achievements of their predecessors .
For example, there is no mention of Alexey Ivakhnenko, who, along with his colleagues, published the world’s first effective deep learning network in 1965 .
For example, when writing about Back Propagation (BP), they only cited their own papers and a few other papers, ignoring the papers of Bryson et al. in the early 1960s, from which BP was later developed.
For example, there is no mention of Schmidhuber's own work on RNNs .
Soon, LeCun, one of the Big Three, responded to his counterpart in Europe. The main reason was that the greater credit should go to those who applied the ideas to machine learning , rather than those who came up with the ideas:
Before 1986, many people had already used the chain rule, and many people had discovered that the Jacobian matrix could be multiplied inversely in the multi-step function.
But can we say that Gauss, Leibniz, Newton, and Lagrange all invented backpropagation?
No! They didn’t apply this idea to machine learning. They didn’t implement it.
LeCun said that many people have thought about using the chain rule to train a machine, but backpropagation was not actually invented until the 1980s.
He also said:
Juergen wanted recognition so badly that he attributed all kinds of things to him, many of which he did not deserve.
This resulted in him standing up after every talk and claiming what others had just presented as his own, and the way he presented it was often impolite.
△ He became a meme: "All the LSTMs you wrote are mine."
The opposition between the father of LSTM and the three giants was formed.
When the Big Three won the Turing Award together, the onlookers remembered this interesting battle and built one building after another.
Some humans are not satisfied with simple activities such as hanging out on forums and answering questions on Zhihu, and choose to take up the weapon of ghosts and beasts, which leads to the LSTM song that everyone hears .
The lyrics specifically mention Facebook, probably as a commemoration.
After all, the song title only refers to one side of the battle, and the main spokesperson for the other side, LeCun, is already the chief AI scientist of Facebook.
One More Thing
Let me tell you a secret, those who have tasted the sweetness of ghost animals will not be satisfied with just making one song.
Here's another one, with a more intense beat and some strange lyrics:
You may have heard sentences like this:
"Three people were sentenced to death."
"What are your last words?"
As if to say:
References are as follows.
Colah’s blog:
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Getting Started with Deep Learning (6) - Long Short-Term Memory Network (LSTM):
https://zybuluo.com/hanbingtao/note/581764
Watch Juergen Schmidhuber's debate with LeCun, Bengio, and Hinton:
https://blog.csdn.net/happytofly/article/details/80120442
The New York Times reported:
https://www.nytimes.com/2016/11/27/technology/artificial-intelligence-pioneer-jurgen-schmidhuber-overlooked.html
If you like it, click "Like"!