An uproar! It was revealed that 47% of ICLR 2020 reviewers had never published a related paper

Qian Mingyuyang from Aofei Temple
Quantum Bit Report | Public Account QbitAI

The person who decides whether your paper is selected for a top conference may not have published a related paper himself.

This is not a joke, this is what is happening.

This happened at ICLR 2020, the top conference in the field of deep learning.

Like other top conferences in the AI ​​field, the number of submissions to this conference continues to hit new highs:

It has increased from 1,500 articles last year to 2,600 articles.

The direct result is that there are not enough reviewers.

Then the scene at the beginning appeared. Among the expanded reviewers, 47% had never published relevant papers.

AI academic community is in uproar

This data comes from a tweet:

Of the 7,583 reviewers for ICLR 2020, 1,078 did not know much about the field; 2,484 had read many papers, 2,604 had published 1-2 papers, and 1,417 had been publishing papers for many years.

Calculated in this way, 47% of reviewers have never published relevant papers.

The person who posted this tweet is Horace He, a Facebook intern who is also followed by Jeff Dean.

Last night, Professor Zhou Zhihua of Nanjing University forwarded this data, saying that it was reported that 47% of the reviewers of ICLR 2020 had never published a paper in this field .

And commented that the number of submissions is far greater than what the qualified reviewers can handle (overload), which will cause the tops to collapse one by one, and perhaps they will eventually have to return to the journals.

He then gave further comments:

Open review is only effective when participants are experts of a certain level; otherwise, it is more likely to be misled.

Academic judgment cannot be "equal". The knowledge and judgment of ordinary practitioners are incomparable to those of high-level experts. The top conferences can be "top" precisely because they have high-level experts to review them, but this is no longer possible now.

This data also caused an uproar in the entire AI academic circle, and many people forwarded it and expressed their shock.

Not just ICLR

More worryingly, some point out that ICLR is not alone.

Earlier, NeurIPS revealed that it had hired undergraduates who had just graduated and had not had their papers accepted by NeurIPS as reviewers.

AAAI also has the same problem, and some people worry that this phenomenon will eventually backfire on the entire academic community.

And this incident has not only caused heated discussions in China, but also among netizens abroad.

One PhD student expressed angrily: “Thank you so much to the reviewers of ICLR. I can finally make up my mind to say goodbye to the academic world.”

Anima Anandkumar, director of machine learning research at NVIDIA, said: Don't take those useless review comments to heart. If you believe that your work is valuable, then keep working hard. Don't deny yourself because of the reviewers.
In this way, the review comment that became a legend before: What is Softmax? seems to have a clue.

Did you submit to ICLR? What do you think about it?