What is an automatic parking system? Analysis of automatic parking path planning and tracking technology

Projection Transformation

3.1 Principle of Projection Transformation The popular understanding of projection transformation is: assuming that the same camera is at two different positions A and B, shooting the same plane (the focus is on the shooting plane, such as the desktop, wall, and ground plane) in different postures, two images are generated. The relationship between the two images is called projection transformation. Professor Zhang Zhengyou's camera calibration method uses the projection model from the calibration plate plane to the image plane.

In the projection transformation model diagram, the camera shoots the same X plane from two different angles. The projection transformation matrix H (homography matrix) between the images captured by the two cameras is:

Among them, K is the camera intrinsic parameter matrix, and R and T are the extrinsic parameters between the two cameras. There are many ways to derive this formula on the Internet. You only need to know that the homography matrix H actually contains the pose relationship between the two cameras. This also explains why some AVM pipeline methods require calibrating the camera's extrinsic parameters, and then unify all four fisheyes to the vehicle body coordinate system through the camera installation parameters provided by the manufacturer, without this process, and only need to use the calibration cloth for joint calibration. In fact, the two methods are internally connected, and both cannot avoid calculating the camera extrinsic parameters. 3.2 Projection transformation to generate a bird's-eye view The process of generating a bird's-eye view can be understood as: projecting the image captured by the fisheye camera onto a plane of a camera that is parallel to the ground above the car. The specific value of this homography matrix H is determined by the coordinates of the checkerboard corner points detected in the dedistorted image and the coordinates of the checkerboard corner points in the jointly calibrated panoramic image. As shown in the figure, taking the rear camera as an example, the coordinates of the checkerboard framed in the known figure (2) are jointly calibrated. The coordinates of the checkerboard in figure (1) can be detected through the function of opencv, thereby establishing a solution model for the homography matrix H.

(1) Chessboard positions in the dedistorted image (2) Joint calibration of chessboard positions in the panoramic image (3) Bird's-eye view 3.3 Some experience 3.3.1 Try to select more corner points to calculate the homography matrix Solving the homography matrix is ​​a fitting process. If too few points are selected, it is easy to fall into the local optimal solution. The result is that only the points you selected can be projected correctly on the bird's-eye view, and the projection of other pixels may be incorrect. This is a bit similar to the problem of too few training samples in deep learning , which leads to overfitting.

Homography matrix has three forms (1) (2) (3) As can be seen from the above formula, a pair of matching points can provide two sets of equations. In theory, four pairs of matching points can solve the homography matrix. Opencv provides two functions to solve the homography matrix, findHomography and getPerspectiveTransform . The input of getPerspectiveTransform is four pairs of points, which inverts the matrix in (2). Ideally, this method is feasible, but due to the presence of noise, the errors of the corner points detected on the image and the errors of the calibration cloth chessboard make this method extremely inaccurate. The findHomography method for finding the homography matrix takes a lot of input points and solves an overdetermined equation (3). After a lot of derivation, the homography matrix is ​​the eigenvector corresponding to the minimum singular value in the singular value decomposition of the matrix in (3). This method uses more samples for fitting and has better final results. In addition, Opencv has many optimization algorithms, such as the homography matrix solution method based on the idea of ​​RANSAC. Of course, in order to improve the effect, you can DIY the calibration cloth . There are many such DIY calibration cloths on Taobao, and you can make as many grids as you want. As mentioned in 1.2, due to the incomplete removal of distortion, some straight lines are still curved. This phenomenon is particularly obvious after projecting onto the bird's-eye view. Through a large number of chessboard points for projection transformation, this problem can be forced to be corrected to a certain extent. At least the panoramic view near the car body can be made better, and the AVM system is most concerned about the part around the car body, and the part far from the car body will not be presented. The figure shows a schematic diagram of a factory's DIY calibration cloth.

DIY calibration cloth 3.3.2 Try to place the chessboard in the center of the image captured by the camera. The fisheye camera has less distortion in the center and more distortion at the edge. The result of de-distortion is usually better in the middle, with more residual distortion at the edge. Therefore, in order to make the homography matrix calculation more accurate, it is necessary to ensure that the chessboard is located in the center of the fisheye camera when the calibration cloth is placed. This is why the schematic diagram used for calibration cloth on Taobao is usually like Figure (2) instead of Figure (1). Obviously, the chessboard in Figure (2) is located near the left rearview mirror (the left fisheye camera is on the left rearview mirror), that is, in the middle of the camera image, while the chessboard in Figure 1 is on the edge of the camera image.

Bird's-eye view from the left fisheye camera (1) (2)

Splicing and Fusion

After the projection transformation in 3, four bird's-eye views with overlapping areas are obtained as shown in the figure. These bird's-eye views need to be spliced ​​and fused.

Take the left and front fisheye camera top view as an example and observe their overlapping areas:

The white area is the overlapping area, AB is the boundary of the front bird's-eye view, and CD is the boundary of the right bird's-eye view. The usual approach is to use AB and CD as boundaries, calculate the distance between the white area pixel and AB and CD, and then calculate a weight. The closer to CD, the greater the weight of the front bird's-eye view; the closer to AB, the greater the weight of the left bird's-eye view. However, there will be a boundary effect as shown in the figure:

The reason for the weight map of the front view is also easy to understand: as shown in the figure, extend AB and CD to point O. In the area of ​​EAOCE, the weight map calculated using the above method is a complete and continuous model. If the weight map is calculated in the area of ​​EABF DC E, it is equivalent to forcibly truncating a complete and continuous domain, and the calculated weight map must have traces of truncation.

A complete and continuous model diagram therefore requires a strategy to calculate weights on a continuous scope. Here is an idea: Calculate the weights in the continuous scope of EAFCE and get a very beautiful weight diagram:

Continuous Models and Weight Graphs

Bird's-eye view fine-tuning based on optical flow

In the entire AVM system, the distortion table, focal length, camera principal point position, and calibration cloth used for joint calibration provided by the manufacturer will introduce errors. These errors will cause some offsets in the generated bird's-eye view in the overlapping area. The stitching and fusion module in Section 4 is to make the bird's-eye view transition smoothly in the coverage area and avoid artifacts as much as possible. However, this pressure cannot be applied entirely to the stitching and fusion module. Therefore, it is necessary to fine-tune the bird's-eye view before the stitching and fusion module. This fine-tuning function is for customers or 4S shop personnel to manually adjust. During the survey, it was found that many AVM systems deployed on vehicles now include fine-tuning functions. But most of them have a problem: only one side can be guaranteed to be aligned, and the other side cannot be aligned. For example, the front is aligned but the back is not aligned. It is used to fuse the front and rear fine-tuning optical flow maps to obtain a smoothly transitioning optical flow map, taking into account the fine-tuning of the front and rear sides. Taking the left top view as an example, the algorithm flow is as follows:

Fix the front and rear bird's-eye views

Manually fine-tune the left bird's-eye view to make the overlapping area between the left bird's-eye view and the front bird's-eye view fit. Record the fine-tuning matrix M1 and calculate the optical flow map1 based on the matrix

Manually fine-tune the left bird's-eye view to make the overlapping area between the left bird's-eye view and the back bird's-eye view fit. Record the fine-tuning matrix M2 and calculate the optical flow map2 based on the matrix

The weight map w of map1 and map2 is calculated based on the pixel distance. That is, the closer the distance is to the previous bird's-eye view, the greater the weight of map1, and vice versa.

Use w to perform weighted fusion of map1 and map2

When the directions of the fine-tuning matrices M1 and M2 are exactly opposite, this idea based on optical flow can well fuse the two, because matrix transformation is a relatively "hard" mathematical method, while optical flow is as "soft" as water. Take a look at the before and after comparison of the fusion effect:

The left side shows the ordinary fusion effect, and the right side shows the effect after fine-tuning based on the optical flow idea. It can be seen that this method can take into account the fine-tuning effects before and after. It can be understood that the error of the stitching area is evenly distributed to the middle area, and there is no problem of stitching and fusion in the middle area. Moreover, in the process of parking, more attention is paid to the relative position of the car and the surrounding objects rather than the precise position of the surrounding objects. Therefore, it actually looks fine. The AVM algorithm for 2D texture mapping of three-dimensional models is based on projection transformation, which projects the fisheye image onto the bird's-eye view. The homography transformation has a premise: the plane assumption. It is to project one plane onto another plane. The problem is that all three-dimensional objects in the image, such as cars, pillars, and walls, are all treated as planes. These contents will be stretched very long on the bird's-eye view, as shown in the figure: