Learning 2D-to-BEV View Translation with Geometric Transformers

BEV perception is an important trend in autonomous driving. Conventional autonomous driving algorithm methods are based on detection, segmentation, and tracking in front views or perspective views, while the surrounding scene can be represented in BEV, which is relatively more intuitive, and representing the target in BEV is most ideal for subsequent modules. This study proposes a new BEV learning mechanism: Geometry-guided Kernel Transformer. GKT uses geometric priors to guide the transformer to focus on distinguishing areas and expand kernel features to generate BEV representations.

Background

For multi-view camera systems, the difficulty lies in how to convert 2D image representation into BEV representation. According to whether the geometric information is explicitly used for feature transformation, existing methods can be divided into two categories: geometry-based point-by-point transformation and geometry-free global transformation.

The point-by-point transformation based on geometry is shown in the figure below. Using the available correspondences, the 2D features are projected into 3D space and form a BEV representation.

However, this method is too dependent on calibration parameters. The external environment of a moving car is complex, and the camera may deviate from the calibration position during operation, which will cause errors in the system. In addition, point-by-point is complex and time-consuming, making it difficult to apply in actual environments.

The global transformation method takes into account the full correlation between the 2D image and the BEV view. This method is less affected by camera bias. But this also brings problems: the computing power required for global transformation is proportional to the number of pixels in the image. In addition, the model must globally mine discriminative information from all views, which makes the convergence process of the image more difficult.

In this study, researchers proposed a new 2D to BEV transformation mechanism to achieve higher efficiency and robustness, the algorithm is called Geometry-guided Kernel Transformer (GKT). The method uses coarse camera parameters to project BEV positions to obtain 2D positions in multi-view and multi-scale feature maps. Then, the model expands Kh×Kw kernel features around the previous position and interacts the BEV algorithm with the corresponding expanded features to generate BEV views.

Implementation

GKT uses geometric deep learning to guide the transformer to focus on the key areas, and expand the core features to generate BEV views. The framework of the proposed GKT is shown in the figure below. The upper part of the figure shows that the geometric information is used to guide the transformer to focus on the previous area in the multi-view image. The lower part is the core features of the expanded previous area, which interact with the BEV query to generate the BEV representation. In this method, a shared convolutional neural network extracts view features from the surrounding view, and the BEV space is evenly divided into grids, each grid corresponding to a 3D coordinate.

In order to make judgments quickly, the study introduced different indexing methods to get rid of the camera's dependence on calibration parameters at runtime. By comparing three methods: Im2col, grid sampling, and LUT indexing, it was determined that LUT indexing can enable GKT to achieve higher FPS. In this method, the core area of ​​each BEV grid is fixed and can be pre-calculated offline. At runtime, the system can obtain the corresponding pixel index of each BEV query from the LUT and obtain features more quickly through indexing.

test

The algorithm is trained on the nuScenes dataset and evaluated on the val dataset. In the figure below, the vehicle map view segmentation efficiency of GKT and other BEV-based methods under two settings is compared. The test uses a 1×7 convolution to capture horizontal image information, and then applies an effective 7×1 kernel in GKT for 2D to BEV view conversion.

In the test, GKT showed high robustness. Compared with point-by-point transformation, GKT only uses camera parameters as a guide, rather than relying on them completely. When the camera deviates, the kernel area moves accordingly, but can still cover the target. Since the arrangement position of the transformer remains unchanged, but the attention weight of the core area is dynamically generated according to the offset, GKT can always focus on the target, thereby reducing the impact of camera deviation.

At the same time, GKT is highly efficient. By using the proposed LUT index, the 2D-3D mapping operation required for point-by-point transformation can be eliminated at runtime, making the forward process compact and fast. Compared with the global transformation, GKT only focuses on the geometry-guided kernel region and avoids global interactions. In comparison, GKT requires fewer assumptions and converges faster.

Therefore, GKT combines the advantages of point-by-point transformation and global transformation to achieve efficient and robust 2D to BEV view learning. Tests on the nuScenes dataset also show that GKT is very efficient, running at 72.3 FPS using a 3090 GPU and 45.6 FPS using a 2080ti GPU, which is lower than all current methods.