Introducing a design method for intelligent driving in pure memory mode

Function control logic

Similar to memory parking, the memory driving function mainly implements simple service-oriented application services. Typical application scenarios include: commuting routes, picking up children, supermarket purchases, etc. That is, the driver sets the destination he wants to reach on the system and activates the memory driving function. The intelligent driving vehicle can refer to the previously memorized driving route and drive to the destination autonomously. During this period, the system needs to remember the user's teaching route to realize full-scene pilot assisted driving from point A to point B, including automatic passing through intersections, U-turns, lane changes, etc.

The steps to realize the memory driving function are mainly divided into two steps:

1) First, the driver drives the vehicle from the starting point A to the end point B under manual driving. During this period, the memory driving system needs to complete the following tasks in the background:

The background constructs a real-time local map corresponding to the road section where the driver drove through real-time positioning and mapping;

The backend needs to combine the constructed map to synchronously record the environmental information (mainly road signs) in real time during driving;

The background records the driver's entire driving habits through a function similar to the shadow mode;

The background needs to record the corresponding constructed map, environmental information, vehicle driving control status, and driving habit control status information in real time;

2) Secondly, after activating the memory driving function, the system needs to implement the following sub-functions:

The background needs to confirm whether the vehicle's current position is on the previously defined memory driving route by identifying the road environment information and vehicle body posture;

If the route relocation is successful, the vehicle is controlled to drive forward along the predetermined route to the destination. During this period, the intelligent driving function call control of the entire vehicle can be completely controlled by referring to the previously stored driving data;

If the relocation is unsuccessful, the system will prompt that the memory driving cannot be activated, and when the driver controls the vehicle to drive manually to the destination, the current road section information will be rebuilt into a new map. The newly built map can be considered as a supplement to the previous instant map. Both routes can be included in the alternative routes before the memory driving control is performed later.

A more acute question is what are the similarities and differences between memory driving and traditional urban autonomous driving in terms of the control of the entire vehicle scene?


In fact, memory driving can be regarded as a narrow sense of urban autonomous driving NOA. The sensor units they use are exactly the same, and the only difference is the use of maps. For urban autonomous driving NOA, it is usually considered to directly import high-precision maps for precise positioning, and integrate navigation information to find the next driving path of the vehicle, while the memory driving here needs to be divided into two steps: user accompanying driving and system trial calibration.

① User driving/manual map creation

The user-accompanying stage is also called the manual mapping stage. The process is that during the manual driving of the vehicle, the system secretly calls the online perception module port through the background to perform real-time positioning and mapping, so that the temporary mapping can better replace the map built in the urban area/highway. This process is similar to crowdsourcing mapping in urban autonomous driving. It should be noted here that considering the country's issues on data security, information security and map qualifications, the map we build through memory is generally a single-vehicle type, and the map cannot be uploaded to the cloud through the network. Of course, if it is for the same type of vehicle developed by the same tier1, you can consider sharing the map package within the vehicle through software upgrades.

② Intelligent Mapping

For the memory driving function, full consideration will be given to the use of background records for intelligent mapping during the activation of the intelligent driving system function. This intelligent mapping process generally also needs to integrate some of the prior knowledge data transmitted by the traditional high-precision positioning map to effectively supplement the current instant mapping. For example, when a vehicle is driving on a certain section of the road, for memory driving, considering computing power, recognition ability, storage capacity, etc., only a fixed lateral distance (generally within 10m) will be considered to model the three lanes of the self-lane and the side lane. High-precision maps can consider a wider range of mapping (even up to 5-6 lanes). In this way, after the memory map and the high-precision map are matched, the data of the high-precision map can be supplemented to the middle of the memory map to expand the recognition range and mapping capabilities of the memory map.

Of course, the functions described in intelligent mapping require two levels of calculation at the same time, which may cause huge consumption of resources. This puts forward greater demands on the computing resources of intelligent driving domain control.

③Smart recommendation

Smart recommendation is the biggest difference between the memory driving function and the traditional intelligent driving system function. Under normal circumstances, the activation of traditional ADAS system functions is usually initiated by the driver, and the system is passively activated. Of course, some OEMs tend to make the entire function a recommendable activation method. The biggest feature of the memory driving function is "repositioning + smart recommendation". That is, when entering a path that has been mapped before, the posture and position of the vehicle will be positioned and matched, and then based on certain HMI interaction rules, it will make an intelligent recommendation on whether to turn on the memory driving function.

It should be noted that if the driver chooses different driving paths to reach the destination in two different time periods when traveling from point A to point B, there may be two segments of the map stored in the memory map.

For the processing of the most important perception sources in the intelligent driving domain (whether it is visual point cloud, laser point cloud or millimeter wave point cloud), it can be generally divided into four levels of processing: scene reconstruction, element recognition, key target reorganization and self-vehicle repositioning.

“Scene reconstruction” means inferring the geometry of the scene, including the position of the vehicle in the scene, from a video sequence. “Feature recognition” is a term used to attach semantic labels to individual recognized objects in a video image or scene, including various different hierarchical structures. “Ego relocalization” refers to the recognition and metric localization of the vehicle relative to its surroundings. “Key object reconstruction” is a method of reintegrating the three components of relocalization, recognition, and reconstruction into a unified representation.

The analysis process of how the above-mentioned perceptions are applied to actual perception scenarios can be analyzed in detail through the following examples.

Here we can explain a new function in the intelligent driving function - "memory parking". This kind of function requires the visual environment of autonomous driving to draw a map of the vehicle's surroundings in advance in the background during the driver's manual driving phase, and at the same time estimate the vehicle's current posture in the user map. One of the key tasks for mapping and relocalization is to relocate the driving trajectory based on the previously recorded vehicle driving path relocation.

1) Self-driving car mapping and relocalization

First of all, during the scene training stage, the vehicle needs to drive through the "closed section" for training (the closed section here is a broad concept), and use the map HDMap transmitted from the infrastructure as the base map for intelligent mapping. Of course, from the perspective of the vehicle, it is also necessary to combine a circle of cameras for scene BEV mapping. It should be noted here that in terms of priority, we generally consider using HDMap as the base map input to the vehicle when HDMap can guarantee a certain accuracy for normal transmission. BEV serves as a backup after the HDMap fails, providing a backup vehicle positioning map for the vehicle.

The figure below shows a classic feature-based retargeting pipeline .

2) Semantic recognition of environmental targets

Secondly, in the scene understanding stage, 3D reconstruction (i.e., restoring the 3D geometric elements of the scene) and element recognition are performed. This includes features such as scene drawing, obstacle avoidance, motor vehicle control, and even reflective lighting scene elements. Scene element recognition is actually a high-level reasoning about scene elements. That is, the elements such as cars, two-wheelers, and trucks in the scene are divided into subsets according to the spatial hierarchy, and each subset is annotated with information.

在图像识别的关键要素中，首要是需要提取图像中的显著特征。图像中的一个显著特征可能是像素区域，其中强度以特定方式变化，例如边缘、角落或斑点。要估计地标在世界坐标系上的位置，就需要执行特定目标跟踪，其中可以匹配相同特征的两个或多个视图。一旦车辆移动的足够远导致对于图像特征目标抓取不够清晰时， 算法 则会拍摄另一张图像并重新提取相应的特征。通过重建相应的特征信息来获得它们在真实世界中的坐标和姿态。然后，这些 检测 到的、描述的和定位的地标被存储在永久存储器中，以描述车辆轨迹的相对位置。如果车辆返回限定范围内的相同位置，实时特征检测将与存储的地标匹配，以恢复车辆相对于存储轨迹的姿态。

3) Post-fusion processing