"Detailed Explanation of LiDAR Algorithms for Intelligent Driving" 4. External Parameter Calibration of LiDAR-Camera

戈壁滩上的辉煌

"Detailed Explanation of LiDAR Algorithms for Intelligent Driving" 4. External Parameter Calibration of LiDAR-Camera [Copy link]

In the field of intelligent driving, due to the high complementarity between camera data and lidar data, the data fusion of the two has become a research hotspot. One of the keys to achieve high-quality fusion is to accurately calibrate the external parameters of the two. This chapter will explore the external parameter calibration method between 3D lidar and 2D camera (LC) in depth. According to the motion state of the sensor during calibration, LC calibration can be divided into static calibration and dynamic calibration; according to the need for calibration objects, it can be divided into two methods: with calibration objects and without calibration objects (i.e. self-calibration). The difference in real-time performance and algorithm operation status divides the calibration algorithm into offline calibration and online calibration. In scenes with calibration objects, checkerboards, QR code calibration plates, etc. are often used to quickly locate feature points in the lidar coordinate system and the visual coordinate system to establish connections between sensors. In the absence of calibration objects, the self-calibration algorithm extracts line features, surface features, etc. in the environment and uses feature matching to estimate the LC external parameters. Using the correspondence between the 3D area in the point cloud and the 2D area in the image is also an effective self-calibration method. Online LC extrinsic calibration during dynamic driving of the vehicle faces higher challenges. The industry mainly estimates the calibration parameters by maximizing the registration information of the point cloud and the image, and constructs a nonlinear optimization problem to solve the extrinsic parameters. According to the different information extraction methods, the research is mainly divided into edge registration, mutual information, segmentation and other directions. When the lidar and the camera have no overlapping fields of view, the LC extrinsic parameters can also be solved by analyzing the motion trajectory and combining the hand-eye model. Huang et al. further applied methods such as Gauss-Helmert to optimize the calibration process.

The ILCC algorithm innovatively uses a checkerboard calibration plate and is verified by a Velodyne 32-line laser radar and a Ladybug 3 panoramic camera. The essence of the ILCC algorithm is to efficiently utilize checkerboard information. It is not limited to the four vertices of the traditional LCC, but to accurately capture the black and white intersections of the checkerboard in the image with the sparse laser point cloud to build a unique matching relationship. By combining the checkerboard size, grid size, laser reflection intensity and color contrast, a loss function is constructed and optimized to accurately locate the three-dimensional coordinates of the intersection. Subsequently, the corresponding two-dimensional coordinates are extracted from the panoramic image to form a complete 3D-2D feature point matching system. The ILCC algorithm first uses the UPnP algorithm to quickly obtain the initial value, and then relies on the nonlinear optimization capability of the LMU617 algorithm to accurately calculate the final external parameter estimate.

Feature intersection extraction, point cloud segmentation After obtaining the original 3D laser point cloud, the ILCC algorithm will first cluster it. The nCC algorithm combines flatness, laser point distribution, and edge size constraints to filter the checkerboard point cloud. The checkerboard road detection preliminarily filters the calibration plate point set by the number of laser points and coordinate values. The number of vertical calibration plate laser points is estimated according to the calibration plate size and lidar resolution. The ILCC algorithm selects point cloud clusters with plane features through clustering flatness features and principal component analysis methods. The plane point set that meets the conditions is further selected according to the boundary size of the cluster. The ILCC algorithm selects the point cloud clusters corresponding to the calibration plate by analyzing the uniformity of the point cloud distribution. After obtaining the 3D laser point cloud corresponding to the checkerboard calibration plate, the ILCC algorithm estimates the location of the black and white intersections of the checkerboard based on the intensity information of the point cloud.

The PESC algorithm, a static LC calibration without calibration objects, was innovatively proposed by C. Yuan and other scholars from the School of Mechanical Engineering of the University of Hong Kong in 2021. The algorithm cleverly combines Livox AVIA semi-solid-state laser radar and RGB camera technology to pioneer the development of a new pixel-level extrinsic self-calibration (PESC) pixel-level laser-vision external parameter joint calibration method. This algorithm breaks through the limitations of traditional reliance on calibration plates. By capturing and parsing universal features such as natural edge lines in the surrounding environment, it establishes an accurate feature mapping between the visual and laser coordinate systems to achieve autonomous calibration of external parameters between sensors. Its official implementation code has been open sourced to GitHub for academic research and industrial applications. The PESC algorithm carefully constructs three core modules: (i) Edge feature extraction: Efficiently identify and extract key edge information in images and point clouds; (ii) Feature association matching: Accurately match corresponding feature points in visual and laser data to build a cross-modal association bridge; (iii) Nonlinear optimization external parameter solution: Based on a powerful optimization algorithm, accurately calculate and optimize the external parameters between sensors. In the edge feature extraction process, the PESC algorithm avoids the common zero-value mapping problem in traditional methods. Even in complex scenarios where the camera's field of view is covered but the lidar cannot be observed due to occlusion, PESC can still work robustly, screening effective features through intelligent algorithms to ensure the accuracy and robustness of the calibration process. This innovation not only simplifies the calibration process, but also significantly improves the calibration accuracy, providing strong technical support for fields such as autonomous driving and robot navigation.

A new breakthrough in LC dynamic online calibration - AOCCL algorithm: Levinson and Thrun of Stanford University pioneered the AOCCL algorithm in 2013, realizing the automatic online calibration of cameras and laser radars without the need for calibration plates. The core logic of the AOCCL algorithm is exquisite and direct: under precise external parameter calibration, the edge features with deep mutations in the laser radar point cloud should accurately correspond to the edge contours in the image after being mapped to the image plane. This algorithm can not only track the changes in the external parameters of the camera and laser radar in real time while the vehicle is driving, but also complete the external parameter calibration using multi-frame point cloud and image data under static conditions, showing extremely high flexibility and practicality. It is worth noting that when implementing dynamic calibration, it is crucial to ensure the clock synchronization of the laser radar and the camera. At the same time, in order to eliminate the potential interference of vehicle movement on the laser point cloud, a specific method is also required to correct motion distortion. At the feature processing level, the AOCCL algorithm cleverly extracts edge features in parallel in the image and point cloud worlds. For the image part, the algorithm accurately locates edge feature pixels through two major steps: first, the image sequence (I1 to In) is converted into a grayscale image E*, where each pixel value of E* is determined by the maximum absolute deviation between the corresponding pixel of the original image and its 8 neighborhood points. This conversion greatly highlights the edge information in the image and lays a solid foundation for subsequent feature matching.