Learn about autonomous driving trajectory prediction technology in one article

In fact, trajectory prediction is becoming more and more important and has become a core component in the architecture of autonomous driving systems. It is located between perception and decision-making planning and plays a bridging role. It is based on sensor data such as cameras and lidars to comprehensively analyze the dynamics of the surrounding environment, including the movement patterns of other vehicles, pedestrians and non-motor vehicles. This process involves not only capturing the current state, but more importantly, estimating future possibilities, that is, predicting the possible trajectories of these objects in the next few seconds to minutes. Its importance lies in its direct connection to the safety and driving efficiency of autonomous vehicles: through accurate prediction, the system can plan obstacle avoidance paths in advance, optimize driving speed and routes, effectively respond to emergencies, and reduce traffic accidents. In addition, the accuracy of trajectory prediction is also the key to achieving a smooth autonomous driving experience, helping vehicles to shuttle freely in complex traffic environments and improving passenger comfort and trust. It can be said that the level of intelligence and anthropomorphism of autonomous driving technology depends to a large extent on the ability of trajectory prediction.

#02

Physics-based methods use vehicle dynamics and kinematics theory, such as constant velocity (CV) models, constant acceleration (CA) models, and more complex constant turning rate and velocity (CTRV) models. These methods usually rely on well-established mathematical and statistical techniques to make predictions based on historical data and predefined models, and directly infer future trajectories from the current state by considering the vehicle's dynamic parameters. These models are concise and efficient, and computationally efficient, but they are unable to handle the complex interactions and uncertainties of real traffic, and are usually suitable for scenarios with prediction times shorter than one second.

Based on classic machine learning methods, such as support vector machines (SVM), Gaussian processes (GP), and hidden Markov models (HMM), future behaviors are predicted by learning historical trajectory patterns. These models can incorporate more contextual factors and handle more types of scene features, but their generalization capabilities are limited, especially when dealing with unseen driving strategies. For example, although dynamic Bayesian networks (DBNs) can model interactions to a certain extent, their strategy judgment capabilities are limited.

基于深度学习的方法，基于深度学习的方法在自动驾驶车辆（AVs）的轨迹预测方面受到了广泛关注。这些方法利用人工神经网络从大量数据中学习复杂的模式和关系，如使用循环神经网络(RNN)、卷积神经网络(CNN)和生成式对抗网络(GAN)，通过多层次特征提取，能捕捉更复杂的时间序列依赖和交互关系，生成多模态轨迹，显著提升了预测的复杂度和精度。RNN和其变种LSTM、GRU能有效处理序列数据，捕捉时间依赖；CNN擅长提取空间特征，两者结合则可同时考虑时间序列和空间交互。注意力机制聚焦关键信息，提高预测的针对性。生成式对抗网络(GAN)和变分自编码器(VAE)等生成模型则为多模态预测提供了可能。与传统方法相比，基于深度学习的方法在捕获复杂模式、处理不同场景和生成更准确的轨迹预测方面表现出更好的性能。缺点是它们需要大量标记的训练数据和计算资源来进行训练和推理。此外，学习模型的可解释性是一个挑战，因此验证预测并了解模型在现实场景中的局限性非常重要。

Reinforcement learning-based methods, especially inverse reinforcement learning (IRL) and deep inverse reinforcement learning (DIRL), learn reward mechanisms by simulating expert trajectories to predict future trajectories. IRL learns policies through reward functions and distinguishes the value of different behaviors, while DIRL uses deep networks to approximate complex reward functions. Combined with deep learning networks, these methods can better extract expert demonstrations and consider more factors. The disadvantage is that most of them are computationally intensive and require long training times. In addition, the trade-off between the complexity of RL algorithms and the availability of training data, as well as the problem of uncertainty in generalization to various driving scenarios and real-world environments, must be considered.

In short, each type of method has its own advantages and disadvantages, and they are often used in combination in practical applications to improve the accuracy and real-time performance of predictions and meet the needs of autonomous driving trajectory prediction and safety planning.

#03

The emergence of various datasets has promoted the performance of learning-based prediction algorithms. In order to evaluate the quality of trajectory prediction models, the predicted trajectories are usually compared with the true trajectories obtained from various datasets. These datasets are collected by sensors such as lidar and cameras, and are manually annotated or automatically generated vehicle trajectory sequences.

Most methods take trajectories as input, and some also use vehicle states or map information. Since most trajectories in these datasets are obtained by learning methods from images or point clouds, some models also directly use images or point clouds as input for end-to-end trajectory prediction.

(1) Root Mean Squared Error (RMSE): Calculates the square root of the mean of the squared prediction error. RMSE is sensitive to large trajectory prediction errors and is one of the commonly used metrics for trajectory prediction.

(2) Negative Log Likelihood (NLL): For the modeled trajectory distribution, RMSE is used to calculate the average error of the model, while NLL focuses more on determining the correctness of the trajectory in the maneuver-based model.

(3) Average displacement error (ADE): The average distance between the predicted trajectory and the true trajectory. For multimodal prediction, the minimum ADE (mADE) is usually used to indicate the minimum value of ADE over K predictions.

(4) Final displacement error (FDE): The distance between the final prediction result and the corresponding true trajectory position. For multimodal prediction, the minimum FDE (mFDE) is usually used to indicate the minimum value of FDE over K predictions.

(5) Miss Rate (MR): The ratio of the predicted trajectory to the actual trajectory that is not within 2.0 m of the actual trajectory based on the distance to the final position. When the prediction result is multimodal, assuming that the prediction result is K possible future trajectories, ADE, FDE, and MR will be determined based on the best future trajectory.

(6) Computation time: Computation time is very important for the in-vehicle performance of the method. Autonomous vehicles have limited computing power, but trajectory prediction models are usually complex and require huge computing resources. To achieve higher levels of autonomous driving, the calculation of each module must be relatively fast to minimize latency. Therefore, real-time performance or computational cost is very important for the model.

(7) Prediction horizon: refers to the future time step that the model can predict. Generally, the longer the prediction horizon, the lower the accuracy in dynamic or even random driving environments. However, in order to meet the requirements of planning and control systems, trajectory prediction results with a certain cycle time should be input into the system so that the prediction time should not be too short and be consistent with other modules.

#04

As autonomous driving technology continues to evolve, trajectory prediction, as a key part of ensuring road safety, faces challenges in many dimensions, including but not limited to the following aspects:

Real-time: Current complex interactions require that the model not only consider the dynamics and kinematics of individual vehicles, but also integrate environmental dynamics, such as road layout, traffic rules, traffic signs, and even the driving intentions of multiple other vehicles. This results in highly complex model design and difficulty in balancing accuracy and computational efficiency.

Uncertainty: The future trajectories of traffic participants are affected by many unpredictable factors, such as driver intentions and environmental changes, making it difficult to achieve absolute accuracy in predictions. This requires the model to be able to handle the uncertainty in the predictions.

Sensor limitations: Autonomous vehicles rely on a variety of sensors such as cameras, lidar, millimeter-wave radar, etc., which have limited coverage and are easily affected by occlusion and weather, resulting in inaccurate tracking.

Data shortage: In practical applications, insufficient or missing data, sensor failure, and historical data corruption will affect the accuracy of predictions.