Tesla is also researching "pseudo lidar"

Recently, Andrej Karpathy, senior director of Tesla AI, revealed at a machine learning conference that he is also doing some research on pseudo-LiDAR technology. He believes that the gap between virtual LiDAR and real LiDAR is getting smaller and smaller.

pseudo-LiDAR

The earliest paper proposing the concept of "pseudo-LiDAR" data can be traced back to a technical paper from Cornell University in 2018. The authors Yan Wang and Wei-Lun Chao are both Chinese. The paper proposed a new method to shorten the performance gap between pure vision technology architecture and LiDAR.

This paper changes the 3D information presentation form of the stereo camera target detection system, converts the image-based stereo vision data into a 3D point cloud similar to that generated by LiDAR, and switches it to the final view format through data conversion. Although the visual effect is not comparable to LiDAR in terms of experimental results, this new low-cost method provides a way of thinking for the visual solution.

Two years ago, Cornell University and others published papers on visual depth estimation, target recognition, 3D Packing, etc. based on this method. Some researchers found that after adopting their new method, the camera's performance in target detection is close to that of LiDAR, and its cost is only a fraction of the latter. Analyzing the images captured by the camera with a bird's-eye view instead of a front view can increase the accuracy of target detection by 2 times, making stereo cameras a viable alternative to LiDAR, and its cost is much lower than the latter.

Tesla is also doing similar research

Musk's attitude towards LiDAR is self-evident. He believes that relying on LiDAR is like relying on crutches for walking. Recently, Tesla AI Senior Director Andrej Karpathy revealed at the Machine Learning Conference held this year that Tesla is also doing some research on "pseudo LiDAR".

Along the way, he shared a specific example of how Tesla can achieve the accuracy of traditional LiDAR with only a few cameras. The secret in Tesla's evolving solution is not the camera itself, but the advanced processing and neural network built by the camera to fully understand the range and quality of the input. Through the stitching of cameras in different directions, visual depth estimation is projected into a bird's-eye view and used as a local navigation map. At the same time, the depth of each pixel in the picture is estimated, just like the LiDAR point cloud, to form a 3D object detection.

Tesla's camera-based approach is much cheaper and easier to implement in terms of hardware, but requires extremely complex algorithms to convert raw camera input and vehicle telematics into useful information.

Fundamentally, computers can identify lane markings, signs, and other vehicles from a series of continuous static images (also known as video). Tesla takes computer vision to an unprecedented level, analyzing not only the image, but the individual pixels within the image.

“We take a pseudo-lidar approach where we basically predict the depth of each pixel, and we project that out,” Karpathy said. Doing this over time can replicate many of the capabilities of traditional lidar systems, but it requires a lot of real-time processing power to make the image deconstruction capabilities work.

Vehicles are driven in real time, so having a system that can make determinations or predictions based on images does no good if the results aren’t instantaneous. Thankfully, Tesla built its own hardware for the third major version of its Autopilot computer, and it’s designed specifically to run Tesla’s code.

The gap with LiDAR is narrowing

Tesla’s so-called pseudo-lidar solution is getting better. Karpathy showed off a series of lidar-like 3D maps of the world that looked an awful lot like the results from a cutting-edge lidar solution. Of course, the visualizations are more for humans than computers, so they don’t really convey how impactful Tesla’s advances in computer vision are. “Using only vision techniques and pseudo-lidar approaches, the gap will close very quickly,” Karpathy said.

However, some experts have raised doubts that the vision-based method is highly dependent on image clarity, camera pixels and fiber strength, which may be a difficult problem to solve at present. Therefore, it depends on whether the price of lidar drops faster or the visual algorithm improves faster.