Difficulties of Transformer-based object detection algorithm

When it comes to pure vision autonomous driving solutions, the first thing that comes to mind is Tesla. Indeed, as early as 2021, Tesla has already implemented a pure vision BEV detection solution, and the effect is very good.

Careful students may have discovered that the core component of this BEV solution that converts images from camera space to BEV space is Transformer.

Transformer originated from the field of natural language processing and was first applied to machine translation. Later, it was found that it also worked well in the field of computer vision and crushed CNN networks in major rankings.

In the field of object detection, the visual Transformer can not only realize 2D detection and 3D detection, but also multimodal detection. The performance of detection from the BEV perspective is also excellent.

Therefore, mastering Transformer-related knowledge and engineering basics has become a skill requirement for companies recruiting algorithm engineers, and is also a big plus on the resume.

However, there are three difficulties in mastering the Transformer-based object detection algorithm:

Understand the theoretical basis behind Transformer, such as self-attention, positional embedding, object query, etc. The information on the Internet is rather messy and not systematic enough, making it difficult to achieve a deep understanding and mastery through self-study.

Grasp the ideas and innovations of the Transformer-based object detection algorithm. Some Transformer papers involve many new concepts, and the wording is not so easy to understand. After reading the paper, I still don’t understand the details of the algorithm.

2

The Transformer code is not easy to understand because its working mechanism is quite different from CNN, so it takes a lot of effort to fully understand the code and put it into practice.

3