Why is multi-sensor fusion necessary for autonomous driving?

Autonomous driving is becoming a key technology that will impact future industries. On-board sensors are the key to perceiving the outside world in autonomous driving systems. They are like the "eyes, ears, mouth and nose" of the vehicle, helping the vehicle perceive the outside world. Hearing and vision are all indispensable, and the collaborative performance of these perceptions directly determines the safety of autonomous vehicles.

At present, the industry mainly uses three types of sensors, including cameras, lidar, and millimeter-wave radar. Each sensor has its own advantages and disadvantages, so there are usually different tasks in the autonomous driving system.

The main reason for using multi-sensor fusion technology is to maximize the advantages and avoid the disadvantages, design redundancy, and improve the safety factor of the whole vehicle. The functions achieved by the multi-sensor fusion system far exceed the sum of the functions that these independent systems can achieve, which is equivalent to 1+1>2.

Using different sensor types can provide an additional level of redundancy in the event that all sensors of a particular type fail. This error or failure could be caused by natural causes (e.g., fog) or man-made phenomena (e.g., electronic or human interference with cameras or radars).

Currently, most autonomous driving solutions on the market include cameras, lidar and millimeter-wave radar, and use the same system to collect and process data. We need to unify the coordinate system and clock of these sensors in order to achieve three similarities and differences: at the same time, the same geographic coordinates, and the same target appearing in different categories of sensors.

Sensor calibration is divided into internal calibration and external calibration. Internal calibration solves the transformation between each individual sensor and the world coordinate system; external calibration solves the transformation between different sensors in the world coordinate system. Sensor external calibration depends on the accurate internal calibration of the sensor.