Cornell researchers use bird's-eye view to identify objects, making stereo cameras a low-cost alternative to lidar for self-driving cars

(Image source: Cornell University official website)

According to foreign media reports, currently, the laser sensors used to detect three-dimensional objects in the path of self-driving cars are large, ugly, expensive, and inefficient, but highly accurate. Such laser detection and ranging (lidar) sensors are usually installed on the roof of the car, which increases wind resistance, a disadvantage for electric vehicles, and will increase the cost of a car by about $10,000 (about 67,181 yuan). But despite its many shortcomings, most experts believe that lidar sensors are the only viable way for self-driving cars to safely perceive pedestrians, cars and other dangers on the road.

Now, researchers at Cornell University have found a simpler way to detect objects with nearly the same accuracy as lidar, using just two inexpensive cameras on either side of the windshield, at a fraction of the cost. The researchers found that when the captured images were analyzed from a bird's-eye view rather than from the traditional head-on perspective, accuracy was more than doubled, making stereo cameras a low-cost alternative to lidar.

“One of the fundamental problems with self-driving cars is recognizing objects around them, which is obviously key to being able to navigate the driving environment,” said Kilian Weinberger, associate professor of computer science at Cornell and senior author of the paper. “It’s widely believed that you can’t produce self-driving cars without lidar. We’ve shown that, at least in principle, it’s possible.”

Lidar sensors use lasers to create a 3D dot map of the surrounding environment, measuring the distance of objects at the speed of light. Stereo cameras, like the human eye, use two perspectives to determine depth and look very promising. But their accuracy in identifying objects is very low, and the conventional wisdom is that they are too imprecise.

Yan Wang, the first author of the paper and a doctoral student in the Department of Computer Science at Cornell University, and his partners carefully reviewed the data from the stereo cameras and were surprised to find that the information obtained by the stereo cameras was as accurate as that of the lidar. However, they found that differences in accuracy appeared when analyzing the data from the stereo cameras.

For most self-driving cars, data captured by cameras or sensors is analyzed using convolutional neural networks, which are machine learning algorithms that recognize images by applying filters to identify patterns associated with the image. Such convolutional neural networks have been shown to be very good at recognizing objects in standard color photos, but recognizing them from the front distorts the three-dimensional information in the photo. So Wang and his colleagues turned the images from a frontal view into a point cloud observed from a bird's-eye view, which more than doubled the accuracy.

Weinberger said that eventually, stereo cameras could become the primary method for low-cost cars to identify objects, or a backup method for high-end cars that already have lidar.