IBM's "AI Argumentative" made the cover of Nature and became famous after defeating top human debaters

Author | Fu Jing

When it comes to debating, besides the human BB King, there is also the "machine debater".

As early as June 2018, IBM's AI debater Project Debater defeated Dan Zafrir, president of the Israel International Debate Association, and Noa Ovadia, the 2016 Israeli national debate champion, and became famous overnight.

Three years have passed in a blink of an eye, and Project Debater, which claims to be "the first AI system that can debate with humans on complex topics," appeared on the cover of the latest issue of Nature magazine.

On March 17, 2021, IBM Research's paper titled An autonomous debating system was officially published in Nature.

AI is defined as "a machine that can perform tasks that intelligent creatures can complete." As we all know, arguing and debating are a basic way to reflect human intelligence, and are also necessary for many human activities and common to human society. Therefore, improving computers' ability to understand and process natural language and developing argumentation technology have become an emerging direction in AI research.

Years ago, even the most advanced AI was not very good at analyzing human discourse and determining how evidence was used to support conclusions (a process known as argument mining).

Later, with the advancement of AI technology and the increasing maturity of demonstration technology engineering, coupled with strong commercial demand, this field began to develop rapidly. Leifeng.com learned that there are currently more than 50 laboratories around the world studying this issue, including research teams of all major software companies.

Chris Reed from the Centre for Argument Technology at the University of Dundee in the UK believes that one reason why this field is showing rapid development is that AI systems can recognize patterns in language usage in large amounts of text, which has been transformative in many applications, but they have not been successful in mining arguments themselves.

A deeper look reveals why: argument structure is too varied, too complex, too subtle, and often too hidden to be easily identified in the way that sentence structure is.

Against this backdrop, IBM proposed Project Debater, an autonomous system that can debate with humans . This system scans and stores 400 million news reports and content from Wikipedia.

In the paper, IBM provides a complete description of its system architecture and a comprehensive and systematic evaluation.

It is worth mentioning that IBM emphasized the fundamental difference between AI and human debate and AI challenging humans in games.

IBM believes that the latter is the classic challenge that AI researchers have been pursuing for the past few decades, and it still exists within AI's "comfort zone" - and the debate between AI and humans obviously means that AI has stepped out of its "comfort zone". After all, humans still have the upper hand in the debate, and a new paradigm is needed to make substantial progress.

As Chris Reed comments: This paper shows how far research in this area has come.

In Chris Reed's view, Project Debater is a huge engineering feat.

Specifically, Project Debater covers:

• New methods for collecting, interpreting, and argument-relevant material in texts

• Methods to fix sentence grammar (mainly for the system to redeploy extracted sentence fragments during argumentation)

Around key topics, the above methods, combined with information prepared by humans in advance, will provide relevant knowledge, arguments, and rebuttals. In fact, this knowledge base will also be supplemented with sentence fragments written by humans in advance, so that it is not difficult to prepare and introduce presentations during debates.

The main process is:

• Look for sentences with high tendency to relevant evidence;

• Use a neural model to rank the probability that a sentence represents an argument;

• Combined with neural networks and based on knowledge, a classification is made for each argument position.

Chris Reed's comment on Project Debater:

Project Debater is very ambitious both as an AI system and as a major challenge in the field of AI.

The logic behind this is that almost all AI research has set its sights high, and a key problem is to obtain enough data and calculate effective solutions. In this regard, Project Debater solves this obstacle through a two-pronged approach: on the one hand, it only focuses on more than a hundred debate topics; on the other hand, it obtains materials from a large data set as support.

Undoubtedly, Project Debater is not perfect at the moment.

At present, perhaps the biggest weakness of this system is that it is difficult to imitate the coherence and fluency of human debaters. In fact, this is related to the level of its processor to select, extract and organize language.

This limitation is not unique to Project Debater, of course—despite two thousand years of research on arguments, our understanding of their structure remains remarkably poor. After all, the key features of a coherent model of argumentative reasoning will vary widely depending on the focus of the study of debate (whether it is language use, epistemology, cognitive processes, or logical validity).

Therefore, what is a good argument model is itself a big problem.

Those who have been closely following the development of the AI ​​field must have already heard about how Project Debater performs in actual combat.

First, on June 18, 2018, at IBM’s San Francisco office, Project Debater faced off against two top human debaters: Dan Zafrir, president of the Israel International Debate Association, and Noa Ovadia, the 2016 Israeli national debate champion.

In two debates with the process of "4 minutes opening speech - 4 minutes rebuttal - 2 minutes argument summary", the human debaters spoke first, and then Project Debater refuted.

Project Debater processed a large amount of text and constructed a well-structured speech on the topic with clear and explicit views. According to foreign media Engadget, Project Debater threw out a lot of its own opinions and also made fluent rebuttals based on the opponent's speech. In the end, Project Debater won 9 more votes than Dan Zafrir and won the game.

Then on February 11, 2019, Project Debater competed live against Harish Natarajan, finalist of the 2016 World University Debating Championship and champion of the 2012 European University Debating Championship, and the human debater emerged victorious.

On November 21, 2019, Cambridge University held a debate titled "Does the emergence of AI do more harm than good?", where Project Debater once again faced off against human debaters.

Judging from the results, Project Debater won with a support rate of 51%, winning by a narrow margin.

But regardless of the number of votes, many people believe that:

The performance of the IBM AI robot is far inferior to that of human debaters in many aspects. This debate clearly shows that there is still a huge gap between AI debaters and human debaters, such as incorrect sentence structure or unconvincing arguments.

However, IBM also made new improvements to Project Debater at the time, such as evaluating arguments with better quality, detecting arguments with redundant vocabulary of human debaters, and even making it humorous based on a joke library.

In ancient Rome, the famous politician and philosopher Cicero once said:

Eloquence is one of the most glorious virtues of mankind.

Since the time of Socrates, debate has been an important part of human life, shining with the light of reason and wisdom. We will wait and see how AI can better acquire this skill in the future.

Source:

• https://www.nature.com/articles/d41586-021-00539-5

• https://www.nature.com/articles/s41586-021-03215-w

This article is originally written by Leifeng.com, author : Fu Jing. Please reply "reprint" to apply for authorization. Reprinting without authorization is prohibited .