Autonomous driving executives grouped together to criticize the DMV: Using takeover frequency to measure technology level, isn't this forcing people to go to the suburbs to increase mileage?

Guo Yipu from Yunaofeisi Quantum
Bit Report | Public Account QbitAI

At the beginning of each year, the "Autonomous Driving Disengagement Report" released by the California Department of Motor Vehicles (DMV) is an important reference for all parties to evaluate autonomous driving companies.

Because this report contains specific road test mileage and specific takeover times, all of which come from first-hand data from each company and are endorsed by the government, the report is discussed every year to judge the strength of each company.

For example, in last year’s report , except for Waymo, which was far ahead, Zoox, Nuro, and Pony.ai were all ahead, and the machines drove for a long time before people took over, and the data looked good.

But Apple and Uber are in a worse situation. The machines have to be taken over manually after running for a short while, which seems very un-"autonomous driving".

However, the statistical method of this DMV report has always been criticized.

Not only are there doubts from outside the industry, but also the founders of well-known companies in the industry think that the statistical method is unreliable.

For example, Aurora co-founder Chris Urmson, TuSimple co-founder and CTO Hou Xiaodi, and Cruise co-founder Kyle Vogt all expressed their views directly.

The main point of doubt is whether it is reasonable to use "frequency of separation" as a core assessment indicator.

Disengagement refers to the transfer of control of an autonomous vehicle from a machine to a human. The machine stops and the human takes over driving, because there are always times when the machine may not be able to cope or when the safety officer watching the car feels that the situation is not good.

From a literal meaning, the more frequent the takeover, the shorter the time the autonomous driving system is in effect, and it seems that the autonomous driving company is less powerful.

However, executives of autonomous driving companies feel that this assessment method is unreliable.

Aurora co-founder Chris Urmson, who was also the former CTO of Google's autonomous driving, believes that there are two problems. On the one hand, there is no clear definition of what constitutes a disengagement, so the numbers are meaningless; on the other hand, companies may be developing new features or testing them under more complex road conditions, which will inevitably lead to a higher frequency of disengagement.

"The number of disengagements over 100 million miles of driving in a flat, dry area with no cars, no people, and no intersections is nothing compared to the number of disengagements over 100 miles of driving in a city like Pittsburgh with complex road conditions," Urmson said.

Therefore, Aurora internally measures technological progress by engineering speed.

TuSimple co-founder and CTO Hou Xiaodi also said earlier that the criteria for separation are very subjective, mainly assessing the company's moral standards rather than technical capabilities. "Zhang San is strict and will take over if he violates traffic rules. Li Si is radical and will take over only if he avoids major traffic accidents."

Moreover, because the standard is subjective, each company can play word games in the report to find loopholes. And, similar to Chris Urmson's point of view, Hou Xiaodi believes that mileage on complex urban roads is completely different from mileage on empty and deserted suburbs, which is not included in the report.

“If you have driven thousands of miles with this test version without having to take over, it means that the Feature has not encountered any problems, and then it is time to develop the next version. The meaning of testing is to find problems. Only by constantly finding and solving problems can we enter the stable version iteration period.” Therefore, the frequency of takeover is meaningless for autonomous driving in the testing phase, and the test ride experience should be compared instead.

Cruise co-founder Kyle Vogt also mentioned the issue of “mileage” in his blog post. He believes that disengagement often occurs in several different situations:

1. Emergency, such as when someone or a car suddenly rushes in front of the car;

2. The situation is not actually urgent, but the safety officer is too cautious and takes over, just like your mother always takes over when she sees you cooking;

3. Show courtesy to other cars on the road. After all, autonomous driving is different from human driving. If other drivers feel awkward or confused, the safety officer will intervene.

4. If the autonomous driving system makes a mistake and you don’t take over, something will go wrong.

Vogt believes that only the fourth situation is a case where the autonomous driving system really has problems, so it is unfair to include the first three.

Therefore, the frequency of disengagement cannot measure whether autonomous driving is ready for commercial implementation. Instead, we should compare the differences between autonomous driving cars and human driving to see whether autonomous driving is better than human driving.

After all, car accidents often occur when humans drive nowadays.

Finally, I don’t know if it’s because of too many complaints and complaints, but this year’s California “Autonomous Driving Disengagement Report” has not yet been released.

However, the company conducting road tests in California has already submitted its data, and now that they have heard the complaints and are making temporary changes, I'm afraid they are running out of time.

Reference links:
https://medium.com/aurora-blog/defining-progress-in-2020-5f5a4e241cb1
https://www.qbitai.com/2019/04/1805.html
https://medium.com/cruise/the-disengagement-myth-1b5cbdf8e239