What are the technical issues with ADAS?

The American Automobile Association (AAA) recently released a report that not only failed to endorse active ADAS features such as lane departure prevention and automatic emergency braking, which are claimed by automakers to "save lives," but also pointed out that the industry is far from safely operating ADAS and suggested that automakers should abandon plans to provide ADAS.

AAA issued the warning after testing a range of ADAS-equipped vehicles on real roads and closed courses, saying that the performance of many current ADAS vehicles is "inconsistent" and "far from 100% reliable." The report mentioned "dangerous situations", including "active driver assistance systems are often disengaged with only a little notice - almost immediately returning control to the driver."

"AAA recommends that manufacturers expand testing of active driver assistance systems and limit the rollout of such systems until their capabilities improve and provide a more consistent and safe driving experience," the report reads.

AAA pointed out two major problems: poor performance of lane departure assistance and failure of ADAS to adequately warn drivers. AAA researchers said: "On public roads, nearly 73% of errors involved leaving the lane or failing to maintain a stable position in the lane;" ADAS-equipped vehicles were too close to other vehicles or guardrails.

Depending on the specific nature of the incidents involving lane departure prevention, here are a few common scenarios, according to AAA:

Systems suddenly disengage (sometimes in critical situations);

failed to function;

The system erroneously disengaged due to inattention;

Drift left and right in the lane in a "ping-pong" style;

The side is too close to other vehicles or road safety barriers to an unacceptable degree;

Change lanes on curves, pavement transitions, or when exiting an uphill or downhill slope.

Those systems mostly performed as expected in AAA's closed-course tests, but when the vehicles approached a "soft test vehicle" (a dummy car), "overall, ADAS-equipped vehicles collided with it 66% of the time, with an average impact speed of 25 mph."

AAA also found that active driver assistance systems disengage in dangerous ways, with vehicles equipped with such systems typically returning control to the driver at a moment's notice -- often when the driver is paying only a little attention to the road or has become too dependent on the system.

“AAA has repeatedly found that active driver assistance systems perform inconsistently, especially in real-world situations,” Greg Brannon, AAA’s director of vehicle engineering and industry relations, said in a statement. “Manufacturers need to work to develop more reliable technology, including improving lane departure prevention and providing more adequate warnings.”

Car manufacturers deny responsibility

Unsurprisingly, most car manufacturers have responded to the questions raised in the AAA report. They did not specifically discuss the idea that ADAS functions are unreliable, but emphasized that ADAS is designed to "assist the driver, not replace the driver;" if the driver of the vehicle encounters danger on the road and is unable to handle it, it is the driver's failure to stay focused on driving.

In short, as Phil Magney, founder of VSI Labs, an autonomous driving technology consultancy, told EE Times, carmakers have abdicated responsibility: “There are two types of autonomous vehicles: supervised and unsupervised. Anything below Level 3 is considered supervised automation, which means the driver is fully responsible for the safe operation of the vehicle, paying close attention to what is happening and intervening when necessary.”

He is right. If the ADAS function in your car fails, such as hitting a parked car on the side of the road, the responsibility lies with you; if you are paying attention while sitting in the driver's seat, you can fully compensate for the imperfections of the ADAS function. Whether you think it is fair or unfair, after all, the car manufacturer developed a "less advanced" driver assistance system that cannot clearly distinguish whether it is a parked car or a garbage bag on the side of the road, and they have absolved themselves of responsibility.

Are there any test benchmarks?

However, the variability of ADAS performance will one day become a problem that automakers must face. Currently, a lot of work is underway to standardize the naming of ADAS functions. However, the different performance levels of ADAS functions will still confuse consumers, because the public cannot predict their performance based on the names of ADAS functions alone.

Is there a set of test benchmarks that car manufacturers can use so that they can develop ADA functions that meet high standards? Magney admitted that there is no such benchmark. "No testing agency has a test benchmark designed specifically for autonomous driving. They do have good guidelines to test ADAS functions, but self-driving car performance is not within their research scope."

Magney said he was somewhat surprised that so much work had not been done, but noted: "I think it's because no one can agree on what is safe or unsafe."

Others disagree. For example, Colin Barnden, chief analyst at market research firm Semicast Research, told EE Times that the test benchmark should play a role similar to the New Car Assessment Program (NCAP), where officials evaluate vehicle safety performance and provide consumer information.

However, in the United States, the National Highway Traffic Safety Administration (NHTSA) does not seem to be rigorous or proactive in promoting NCAP-related tests and standards. Some people even criticize NHTSA for giving almost 98% of new cars four or five stars, which is like the concept of "every participant in a kindergarten football game gets a prize."

Compared with the United States, Europe is far ahead of the United States in the development of ADAS test guidelines. Barnden said that EU NCAP and Thatcham Research, a British testing agency officially authorized by the EU, are the most active in related work, and if US NCAP cannot do the same thing, it can simply adopt EU NCAP's standards.

He cited an example, saying that the Automatic Emergency Braking (AEB) Vulnerable Road User (ARU) test criteria v3.0.3, which came into effect in June 2020, is the most stringent AEB test criteria currently available.

Any technical questions?

The vehicles used in AAA's testing included a 2019 BMW X7 SUV equipped with Active Driving Assistant Professional, a 2019 Cadillac CT6 sedan with Super Cruise, a 2019 Ford Edge SUV with Ford Co-Pilot36, a 2020 Kia Telluride SUV with Highway Driving Assist, and a 2020 Subaru Outback SUV with EyeSight.

“In my opinion, the Level 2 systems performed very poorly in this test,” said Magney of VSI Labs after reviewing the AAA test results. “I was surprised by the failure rate of the lane departure prevention system.”

He pointed out that VSI Labs has tested many lane departure prevention systems, and the most advanced solutions currently use AI and perform much better than the first generation; Tesla's vision-based lane departure prevention menu also performs well because they repeatedly train the system based on millions of miles of consumer usage data.

Why is it that such a high percentage of the vehicles tested by AAA hit the soft car used for testing? Magney said this result is not surprising, because lane departure prevention systems do not distinguish whether the lane is blocked, even the most advanced Tesla system; and for ADAS, "its sensors do not have enough confidence in the lateral position and lateral size (boundaries of objects) to take "aggressive" action." If it is too "aggressive", it will brake frequently and cause danger.

The test vehicle was stationary on the side of the road, which is also a reason; Magney pointed out that AEB often filters out radar information for this reason. And for AI systems, such extreme case training is not enough; "But this is actually no longer an extreme case. Hopefully, this is a known unsafe condition in every instance of intended functional safety."

What about the human factor?

For ADAS, car manufacturers can legally attribute the responsibility to human drivers for not paying full attention when an accident occurs; but shouldn't car manufacturers acknowledge the existence of "human factors"? For most drivers, it is not always completely clear what ADAS wants to do, and more importantly, it is uncertain at what point ADAS will decide to hand the steering wheel back to the driver.

Semicast Research's Barden sees a big problem with the latest AAA report: "AAA doesn't mention driver monitoring systems (DMS) at all in the entire report; that's where the safety issue is, and it's not so much about ADAS."

Indeed, the question of when ADAS should hand over driving to humans should not be discussed without considering DMS based on machine vision. Barden proposed the following dilemma, which highlights the problem of existing DMS:

ADAS is too proactive when the driver is alert and fully focused on the driving task.

In emergency situations where every second counts, especially when the driver is distracted, fatigued or injured, ADAS is too passive.

Barnden pointed out that the solution should be a driver assistance system that can make real-time changes to the reaction speed based on continuous monitoring of the driver's concentration state; "In the next decade, almost every light vehicle produced will have DMS as the main safety system to help ensure that the driver is focused on the driving task and issue an alert when they are distracted, fatigued or injured."

He believes that ADAS should be placed in the position of a second-line safety system to correct minor operating errors and provide longitudinal or lateral intervention when absolutely necessary; and if the precision and usage rate of DMS increase, the distraction or intervention problems mentioned in the AAA study will not exist at all.

In the meantime, whether you are in the new car showrooms of major brands with eloquent salesmen or in the driver's seat of your new car, buyers of ADAS vehicles please continue to be vigilant!