Pedestrian detection and tracking method based on vehicle-mounted vision

Abstract: In order to improve the active safety of vehicles in urban traffic environments and ensure the safety of pedestrians, a pedestrian protection method based on vehicle-mounted visual sensors is proposed. The Adaboost algorithm is used to achieve rapid detection of pedestrians, and the Kalman filter principle is combined to track pedestrians to obtain their running trajectories. This method uses the discrete Adaboost algorithm to train sample class Haar features to obtain a cascade classifier for identifying pedestrians, and traverses the images collected by the vehicle-mounted vision to obtain pedestrian targets; combined with the Kalman filter principle, the detected pedestrian targets are tracked, and a dynamic region of interest for detecting pedestrians is established. The tracking results are used to analyze the running trajectory of pedestrians. Experiments show that: This method takes an average of about 80 ms/frame, and the detection rate reaches 88%; after tracking combined with the Kalman filter principle, the average time consumption is reduced to 55 ms/frame, and the real-time performance is good.

With the continuous increase in the number of cars, the number of road traffic accidents and casualties due to traffic accidents remains high. In order to meet people's increasing requirements for automobile safety performance, more and more advanced technologies are being applied to the field of active automobile safety. In road traffic accidents, the main victims are pedestrians and cyclists participating in the traffic system. According to the data of the National Highway Traffic Safety Administration (NHTSA), in 2008, 69,000 pedestrians were killed or injured and 4,378 pedestrians died in traffic accidents in the United States. The number of pedestrian deaths accounted for 11.7% of the total number of deaths in traffic accidents throughout the year. In 2007, the number of pedestrian deaths caused by traffic accidents in my country was 21,106, accounting for 25.9% of the total number of deaths in traffic accidents, and the number of pedestrian injuries was 70,838, accounting for 18.6% of the total number of injuries in traffic accidents. Compared with some developed countries, my country's traffic mode is mainly a mixed traffic mode, which leads to different causes of death and injury patterns in traffic accidents.

In recent years, in order to ensure the safety of pedestrians and improve the active safety performance of automobiles, some scientific research institutes at home and abroad have conducted research and discussion on pedestrian protection technology. While continuously improving the passive safety system of automobiles, they have gradually developed and applied active safety systems, combined with the introduction of pedestrian protection concepts and technologies, to improve the protection of pedestrians. For example, Bajracharya et al. established a binocular vision pedestrian protection integrated system to achieve the detection and tracking of pedestrians within a distance of 40 m in road traffic scenes.

Munder et al. established a pedestrian recognition classifier by integrating the point distribution shape model and texture features of pedestrians, and used the Bayesian method based on particle filtering to achieve pedestrian tracking. Enzweiler et al. in Germany conducted comparative analysis on wavelet features, linear support vector machines and neural network methods in statistical learning, and detected pedestrians in images by designing classifiers of different scales. Jiang Fan et al. from Tsinghua University proposed a pedestrian tracking algorithm based on model fusion, combining offline learning and online mutual learning to update the model. Cheng Youlong et al. from the University of Science and Technology of China integrated the prior knowledge of pedestrian detection into the self-learning process of the tracking model, and dynamically modeled the tracked pedestrians, thereby achieving the tracking of pedestrians with complex movements in real monitoring situations. Multi-sensor information fusion and the establishment of pedestrian models require a large amount of calculation and calculation parameters, which is difficult to meet the real-time requirements of systems such as vehicle active safety warning. This paper uses a vehicle-mounted monocular vision sensor and a trained pedestrian recognition cascade classifier to obtain pedestrians in front of the vehicle in real time, and tracks them to record their movement trajectories, thereby providing a technical reference for effective warnings for drivers and pedestrians.

1 Pedestrian Detection Based on Adaboost Algorithm

1.1 Principle of Adaboost Algorithm

The Adaboost algorithm obtains a strong classifier formed by superimposing weak classifiers with general classification capabilities through training, and then connects several strong classifiers into a cascade classifier to traverse the image. In order to quickly realize pedestrian detection and anti-collision warning, in view of the characteristics of the Adaboost algorithm, this paper selects the discrete Adaboost algorithm to train a cascade classifier for identifying pedestrians, so as to quickly exclude most non-pedestrian windows in the image. Its structure is shown in Figure 1. Among them, the training process of the strong classifier at each stage is shown in Figure 2.

Figure 1. Schematic diagram of the N-stage cascade classifier structure

Figure 2 Discrete Adaboost training algorithm

From its training process, we can see that the algorithm mainly adjusts the weights of training samples to strengthen the training of misclassified samples, and finally forms a strong classifier by cascading all weak classifiers through weight combination.

1.2 Offline training of samples and online detection of pedestrians

This paper uses the Adaboost algorithm to train a cascade classifier for identifying pedestrians, and loads the classifier through a program to achieve online detection of pedestrians. The detection process is shown in Figure 3.

The offline training module of the sample is mainly to obtain a cascade classifier for identifying pedestrians. The specific process is as follows.

(1) Sample acquisition and preprocessing: Pedestrian samples are obtained by manually segmenting pedestrian images of different distances, directions, clothing and sizes in front of the vehicle offline, and uniformly scaling them to 16 × 32 pixels; non-pedestrian samples are obtained by segmenting from background images that do not contain pedestrians. In order to reduce the intra-class differences of training samples and reduce the impact of different illumination on the training results, the histogram equalization method is used to preprocess the samples. A total of 3,060 training samples are selected in this paper, including 2,100 pedestrian samples and 960 non-pedestrian samples;

(2) Sample feature extraction: The Haar-like feature proposed by P. Viola et al. is used as the feature for pedestrian detection. This feature mainly describes the feature differences between adjacent regions of the image pattern. The integral image can be used to quickly calculate the feature value of the rectangular region.

(3) Training results: The trained cascade classifier consists of 20 levels of strong classifiers, each of which contains a different number of weak classifiers. Each weak classifier consists of a Haar-like feature, a threshold, and a direction indicating an inequality sign. As the number of classifier levels increases, the number of weak classifiers, i.e., Haar-like features, contained in the strong classifier increases. The number of Haar-like features and their number contained in the first 6 levels of strong classifiers obtained through training are shown in Table 1.

The pedestrian online detection module distinguishes each sub-window of the image to be inspected based on the trained classifier. The detection result is a series of target rectangles that show the position of pedestrians in the image. This paper traverses the image to be inspected by scaling the detection sub-window to segment pedestrian targets of unknown size in the image. The specific detection process is as follows.

(1) Obtain an image to be inspected of size 320 × 240 pixels, set the minimum detection window size to 32 × 64 pixels, and move the detection window from the lower left corner of the image to the right every other pixel until it reaches the edge of the image;

(2) Using the cascade classifiers obtained from the training module, the sub-windows to be inspected are judged according to the method shown in Figure 1 to complete the scanning of the size of the detection window at this level: If the sub-window to be inspected can pass all cascade classifiers, it is considered a pedestrian window; If any strong classifier cannot pass, it can be judged as a non-pedestrian window;

(3) Enlarge the size of the detection sub-window according to the proportional coefficient determined by the experiment, and rescan the image to be inspected according to the above steps to obtain a larger pedestrian target.

2 Pedestrian Tracking Based on Kalman Filter

Discrete Kalman filtering is used to predict the area where pedestrians may appear at the next moment, shortening the target search time and achieving rapid tracking and positioning of pedestrians. The pedestrian tracking results can not only obtain the movement trajectory of pedestrians, but also provide a reliable data source for pedestrian movement analysis.

2.1 Kalman filter algorithm principle

Kalman filtering consists of a dynamic process model and a feedback correction link. The dynamic process model realizes the prediction function, and the feedback correction link uses the product of gain and residual as a forcing function on the model. Assume that the state equation and observation equation of the system are as follows:

In the formula: Xk is the n-dimensional state vector at time k; Zk is the m-dimensional observation vector at time k; the n-order square matrix Фk-1 is the state transfer matrix; the m × n-order matrix Hk is the observation matrix; Wk-1 and Vk are two zero-mean Gaussian white noise sequences that obey the normal distribution, and their variance matrices are Qk and Rk respectively.

The Kalman filtering process can be expressed by the prediction equation:

And the filtering recursion equation:

represents, where:

is the filter gain;

is the prediction error variance matrix;

is the filter error variance matrix.

2.2 Pedestrian Tracking

According to the Kalman filter principle, this paper tracks the centroid position of the detected pedestrian in front of the vehicle and the height and width of the pedestrian's circumscribed rectangle. In each frame, the state of the pedestrian can be represented by the position of its centroid, the rate of change of position, and the size of the outer rectangular box. Assume that (xt, yt) represents the pixel position of the centroid of the pedestrian area in the t-th frame image; (Δxt, Δyt) represent the change of the centroid respectively; (ht, wt) are the height and width of the circumscribed rectangle surrounding the pedestrian; (Δht, Δwt) are the changes in height and width. Therefore, the state vector of the pedestrian in the t-th frame image can be expressed as:

Since pedestrians move slowly and the time interval between two adjacent frames is short, it can be assumed that pedestrians move at a uniform speed within a unit time interval. The state transfer matrix can be expressed as follows:

In order to observe the various state variables in the road area, take the observation vector Zt = (xt, yt, ht, wt) ^T and select the system measurement matrix as:

In order to apply Kalman filtering to track pedestrians in front of the vehicle, the initial values ​​of the state variables and the error variance matrix must be determined. This paper starts tracking based on Kalman filtering after successfully realizing the recognition and positioning of pedestrians in two consecutive frames of images. Assuming that the images in which pedestrians are detected are the tth frame and the t + 1st frame, the initial state vector X0 can be expressed as:

In addition, the corresponding covariance matrix P0 needs to be defined. Since Pt is repeatedly updated as more images are acquired, a larger initial value can be given to it. Assume that the predicted position has an error of ± 10 pixels from the actual position in the x and y directions, and the speed has an error of ± 5 pixels from the actual speed in the x and y directions. Due to the movement of the pedestrian's legs, the width of the pedestrian's outer rectangle changes more than the height, so it is assumed that the height of the outer rectangular window has an error of ± 5 pixels from the height of the actual pedestrian rectangular window, and the rate of change of the height has an error of ± 3 pixels; the error of the width is ± 10 pixels, and the rate of change of the width has an error of ± 5 pixels. Therefore, the error covariance matrix P0 can be defined as:

In addition to X0 and P0, the error covariance matrices Q and R of the system state equation and measurement equation need to be estimated. By observation, the system state noise can be considered as follows: The standard deviation of the system positioning error in the x and y directions is 5 pixels, which further indicates that the standard deviation of the speed is 2 pixels per frame. The standard deviation of the height of the pedestrian's external rectangular window is 3 pixels, and its change rate is 1 pixel. The standard deviation of the pedestrian's rectangular window width is 5 pixels, and its change rate is 2 pixels. Therefore, the state variable noise covariance matrix can be expressed as:

Similarly, the measurement equations are defined with a standard deviation of 3 pixels in the x and y directions, 2 pixels in height, and 3 pixels in width. Therefore:

Using the above state prediction and update equations and initial conditions, the state vector X^k (-) and the corresponding covariance matrix Pk (+) of each frame image can be estimated. The covariance matrix Pk (+) represents the uncertainty range of the predicted position of the rectangular window for detecting pedestrians in the next frame. The larger the Pk (+), the more unstable the estimated value and the larger the search area. At this time, the search area is automatically adjusted [17].

3 Experiments and analysis

The pedestrian detection and tracking recognition process is as follows:

(1) According to the online detection module shown in FIG3 , analyze whether there are pedestrians in the sequence of images collected by the CCD, and record the relevant information of the pedestrian target rectangle;

(2) If a pedestrian is detected in two consecutive frames in the sequence of images, the pedestrian tracking program is started and the detection results are used to update the Kalman filter so that it can predict the area where the pedestrian may be in the next frame;

(3) The predicted area is modified according to the following strategy: taking the estimated value of the pedestrian's centroid as the center, the height and width of the predicted pedestrian rectangle are enlarged by 1.5 times, and the region of interest for pedestrian search is established. The cascade classifier is used to detect the target in the new region of interest, and then the detected pedestrian centroid and rectangle width and height are used as the initial values ​​for the next prediction;

(4) Handling of tracking failure: During the tracking process, if detection fails, the pedestrian information of the previous frame is kept and detection is continued in subsequent frames to prevent tracking failure due to accidental missed detection.

If the detection fails for more than a certain number of frames (set as 3 frames in this paper), the pedestrian target is considered to have disappeared and will no longer be tracked.

To verify the effectiveness of the proposed method, a test was conducted on the asphalt road in the school under normal lighting conditions during the day. The test platform was the unmanned driving test platform DLUTIV-I developed by the developed test platform, as shown in Figure 3. The CCD uses the F-033 black and white digital camera produced by AVT Company of the United States, with an image resolution of 320 × 240 pixels, a lens field of view of 25°, a focal length of 12 mm, and an industrial computer with Core 2 Duo 2.2 GHz and 2 GB memory. The experimental results show that the proposed algorithm can realize pedestrian detection within a range of 10 ~ 30 m, the recognition algorithm takes about 80 ms per frame, and the correct detection rate can reach about 88%; after using Kalman tracking prediction, the time for scanning pedestrians in the image is effectively reduced due to the establishment of the region of interest for pedestrian detection. The average time consumption of the algorithm is about 55 ms/frame, and the real-time performance is good. Table 2 statistics the pedestrian detection and tracking results of the vehicle in both static and moving states. In the case of vehicle movement, the false alarm rate is high due to the large changes in the positions of the background and pedestrians in the image.

Table 2 Pedestrian detection and tracking test results statistics

Figure 4 shows the tracking result of a pedestrian in front of the vehicle when it changes from a stationary state to a moving state.

When the vehicle is stationary, the pedestrian walks forward with his back to the vehicle. After walking to a certain position, he waits for the vehicle to start. After the vehicle moves forward for a certain distance, the pedestrian continues to move forward, accompanied by actions such as crossing the road. From the tracking curve in Figure 5, it can be seen that the width of the enclosed rectangle of the identified pedestrian changes significantly, which is in line with the actual situation. The width of the enclosed rectangle changes significantly due to the leg span of the pedestrian when walking; because the distance of the pedestrian relative to the vehicle changes continuously, the height of the pedestrian's enclosed rectangle changes continuously.

Experiments show that different lighting conditions will have a certain impact on the recognition and tracking results of pedestrians, especially when the lighting is strong, some parts of the pedestrians are integrated with the ground or background, and the grayscale difference is very small, resulting in inaccurate pedestrian positioning or recognition failure, as shown in Figure 6.

4 Conclusion

This paper proposes a pedestrian detection method based on the Adaboost algorithm. According to the characteristics of slow pedestrian movement speed, it is proposed to use Kalman filtering to predict the center, width and height of the pedestrian's circumscribed rectangle, establish a dynamically variable pedestrian search region of interest, and reduce the average processing time of each frame of image from about 80 ms to 55 ms, thus meeting the real-time requirements. Experiments show that this method can identify stationary and moving pedestrians in front of the vehicle, with a detection rate of about 88%.

Pedestrian detection technology is an important part of the research field of automobile safety assisted driving. The next step is to analyze the movement state of pedestrians and the degree of danger posed by the vehicle to them based on pedestrian detection. When a collision between a vehicle and a pedestrian may occur, the driver will be warned in time. If the driver does not take effective collision avoidance measures, the system will start deceleration or emergency braking to effectively ensure the safety of pedestrians.

The experiment shows that there are still some deficiencies in this study. The factors considered are not comprehensive and are greatly affected by lighting conditions. The next step is to use a CCD aperture automatic adjustment device to improve the imaging quality under strong or dark lighting conditions during the day, and study how to extract features that effectively represent and distinguish pedestrian targets, increase training samples or improve training methods. At the same time, combine infrared sensors to study pedestrian detection under conditions such as dusk or night.