Implementation of lane departure warning system based on machine vision

Abstract: Traffic accidents caused by lane departure on highways have caused huge losses, making the research on lane departure warning systems a hot topic in society. This paper constructs a vision-based lane departure warning system based on DSP computing and FPGA computing. This paper gives the framework structure and hardware implementation of the system, and studies the transmission, storage and processing of large-capacity data in the vision system in detail based on the proposed system. Finally, based on the constructed system, the feasibility of the system is verified through a series of typical visual processing algorithms and software algorithms applied to lane deviation decision-making.

1. Introduction

Driving a car is an action involving three factors: the driver, the car, and the road. The driver perceives road information through a series of physiological organs such as the eyes, and makes a series of driving decisions by understanding and judging the road information and tracking the driving conditions of the car. However, due to a series of factors such as long-term driving and carelessness of the driver, as well as certain "blind spots" in the driver's perception of road information and understanding of vehicle conditions, traffic accidents are increasingly becoming a major safety hazard in society. Starting from the three aspects of the driver, the car, and the road, this paper studies a set of active safety systems that simulate human driving behavior to improve vehicle driving safety, and realizes active safety technologies such as lane departure warning.

In machine vision-based systems, real-time performance has always been a major issue, especially for systems such as lane departure warning. The algorithms to be implemented cover the entire visual computing process from image preprocessing to high-level visual computing, including low-level regularization algorithms such as convolution and grayscale transformation that can be implemented in high parallelism and large amounts of data, as well as mid-level pattern recognition algorithms such as feature extraction and some high-level visual processing methods to achieve road understanding. For a long time, the versatility and efficiency of embedded systems have always been a pair of contradictions that are difficult to reconcile [1]. How to construct a more general visual processing system that has high implementation efficiency for parallel and serial algorithms at all levels of vision is a problem to be solved in this paper. The emergence of high-speed DSP processing chips and large-scale reconfigurable chips provides us with a good solution.

Large-scale reconfigurable chips, such as FPGAs, integrate large-capacity gate circuit systems based on lookup table structures, which are very suitable for constructing fine-grained SIMD processing arrays or pipeline structures. This is naturally adaptable to the typical underlying image processing algorithm shown in Figure 2 [1]. TI's C6000 series DSP computing chips use Harvard architecture, have 8 processing units inside, and also integrate dedicated hardware multipliers. They are suitable for completing some serial visual processing algorithms and have good implementation effects for high-level road understanding with less data volume but more complex calculation processes.

Based on the above analysis, this paper combines high-speed DSP computing chips and reconfigurable FPGA chips to construct a high-speed embedded visual processing system. FPGA, as an auxiliary logic control and front-end processing module, mainly completes the early image acquisition and some low-level high-density image processing tasks such as histogram statistics and convolution. It can be flexibly configured in the hardware structure, and completes the underlying algorithms by implementing a series of parallel structures or pipeline structures to avoid the waste of resources and time caused by DSP completing these fine-grained algorithms; DSP, as the main computing center, completes high-level serial visual processing tasks, such as road understanding and interpretation, lane deviation decision-making, etc. The high-speed processing capability of DSP can better complete these highly serial algorithms. The basic block diagram of the system is shown in Figure 2.

2. System design and implementation

To realize the vision-based active safety system of Figure 2, data storage and transmission is a major problem, especially for the application background of active safety of automobiles, which has very high real-time requirements. How to obtain the final understanding and judgment of the road from the original collected image data through a series of memory data transfer and signal processing processes in the shortest time, and make the correct decision on whether the vehicle is running off the road, is the main goal of this system. This article will introduce this system in detail based on the "flow" order of data in the system.

[page]

2.1 Data Acquisition Module

In this system, image data is the main sensor signal, and obtaining high-quality image data is of great significance for later processing and function realization. However, for the actual application environment, due to the influence of factors such as weather and light, and even the emergence of a series of special situations, it is difficult for general image sensors to have good robustness under various conditions. In order to better adapt to various road and weather conditions and ensure driving safety, we have independently developed a visual acquisition module using a CMOS digital camera chip suitable for road use. The camera chip can be fully programmable and controlled through a 159-bit SPI control word, and can better adjust the exposure time, gain and noise. At the same time, the camera chip uses a multi-slope exposure mode output. In different exposure modes, the exposure time is also different, ensuring that better images can be obtained when the light is too strong or the light conditions are very weak.

Regarding the SPI control of the camera chip, in order to adaptively adjust the working mode under different backgrounds, this article uses the DSP to give the FPGA certain communication signals based on the final understanding and judgment of the image to complete the top-level system-level control of the camera chip, while the modification and adjustment of the actual SPI control word at the bottom layer is completed by the FPGA. This can avoid the waste of DSP working efficiency caused by the inconsistency between the SPI working frequency and the DSP external bus frequency, and it is easier for FPGA to complete such work.

2.2 Data transmission and storage module

In this system, from the front-end image acquisition module to the final alarm signal output, data is constantly transmitted and moved between FPGA, SDRAM and DSP. The following mainly discusses how to move and process data between these three at high speed and effectively, which is also the main module of this article. The data flow in the system mainly includes the following aspects:

Ⅰ. The image data pre-processed in FPGA must first be stored in the main memory chip SDRAM, which is the basis for back-end high-level processing.

Ⅱ. DSP reads the latest image data from SDRAM for processing and stores the intermediate results in SDRAM. Such a reading and writing process is likely to be cyclical, and the block storage characteristics of SDRAM are just suitable for such performance requirements.

III. For the convenience of system debugging, it is likely to require the display of intermediate processing results. Therefore, we have mounted a display debugging module on the FPGA. During debugging, the FPGA can read the intermediate processing results from the SDRAM for display.

IV. There is another situation. In some cases, we may require direct transmission of image data between DSP and FPGA. This is also a scope we must consider.

Considering all the above possible situations and the fact that the DSP chip we use supports seamless connection between FIFO and SDRAM chips, we designed the data transmission scheme shown in Figure 3:

[page]

First, the image data output by the front-end acquisition module is 8-bit, while the data bus of the SDRAM memory chip and DSP processing chip we use is 64-bit. In order not to waste resources, during the FPGA preprocessing of the image, we simultaneously splice the 8-bit image data into 64-bit data output, so that the system resources can be fully utilized.

Secondly, the image data that has undergone a series of parallel preprocessing in the FPGA is stored in the FIFO opened inside the FPGA. The FPGA gives a handshake signal to notify the DSP that it can start transferring data from the FIFO to the SDRAM. The data transmission process is completely controlled by the DSP through the EDMA transmission method. When the core processor DSP completes the current operation, it reads the latest data from the SDRAM for processing, which can make full use of the rich peripheral resources of the DSP chip.

At the same time, from the data flow diagram in Figure 3, we find that in the process of data transmission from FPGA to SDRAM, compared with DSP, the FIFO module used to store data in FPGA and the external main memory SDRAM module are mounted on the same address bus and data bus on DSP. In this transmission form where the source device and the destination device are mounted on the same external bus, the DSP chip we use supports an optimized EDMA transmission mode - PDT transmission, which optimizes the data transmission process, so that the EDMA transmission mode of DSP can directly transmit the data from the source device FIFO to the destination device SDRAM without passing through the buffer area in DSP, so that the data transmission can be completed within one cycle. Compared with the traditional transmission mode, which is first transmitted from FPGA to the internal buffer area of ​​DSP and then written to SDRAM by DSP, the PDT transmission mode can basically shorten the data transmission time by half, which is of great significance for large-capacity data transmission such as images.

After completing the data transfer process from the front-end preprocessing module to the main memory SDRAM, the next step is the high-speed data transmission process at the back end. The data transmission between DSP and SDRAM can reach a high-speed rate of more than 100MHz, especially the EDMA transmission method supported by DSP, which allows the transmission between external data and the cache inside DSP without CPU intervention, which can improve the efficiency of CPU work. In addition, there are two levels of cache inside the DSP chip, and the second-level cache L2 can reach 1MB. Therefore, we consider using the popular ping-pong transmission method. When the CPU is processing the data inside the ping or pong in the cache, under the premise of predicting the remaining processing time, we can pre-transfer the next data to be processed to the corresponding pong or ping storage area through EDMA, so that the data transmission time is contained in the processing process, saving the corresponding transmission time and improving efficiency.

Combining the above analysis of the entire data transmission process, we can derive the data movement process shown in Figure 4. As can be seen from the figure, the data from the sensor to the pre-processing in the FPGA and the data processing inside the DSP can be transmitted and processed in a pipeline manner under the cache processing of SDRAM. FPGA, as a front-end pre-processing module, completes the processing of large-capacity data, which plays a very important role in improving system efficiency.

3. System software algorithm

[page]

There are many introductory articles on front-end image preprocessing methods and lane departure decision algorithms [2][3][4], so we will not describe them in detail in this article. The main implementation process is shown in Figure 5. Through some typical algorithms in [2], we have verified that the automotive active safety system designed in this article fully meets the real-time requirements of automotive driving applications, and the system has strong scalability.

4. Conclusion

This paper introduces in detail a solution for the lane departure warning system based on DSP computing and FPGA logic computing, which mainly solves the high-speed transmission, storage, and processing of large-capacity image data in the system. In the design process, this paper fully utilizes the adaptability of FPGA logic resources to the underlying visual algorithm and the high-speed processing capability of DSP chips to meet the high-serialization visual algorithm of the high level. Through a series of experiments, it is verified that the system meets the high real-time requirements in the application context of automobile safety.

Innovations of this article:

(1) Combining the relationship between the driver, the car and the road environment, simulating human driving behavior to design the active safety system of the car is more realistic in terms of the principle of system design;

(2) In view of the adaptability of computer vision system design to multiple levels of algorithms, a combination of DSP computing and FPGA logic computing is used to meet the requirements of each level of the system algorithm;

(3) The data transmission, storage and processing processes in the computer vision system were analyzed in detail, and the optimized EDMA transmission method - PDT was used to meet the real-time requirements of the system;

References

[1] Li Jian. Research on reconfigurable parallel low- and medium-level visual computing architecture [D]. Hunan: Master of Engineering Thesis, National University of Defense Technology, 2006

[2] Wu Mo. Research on vehicle deviation warning system based on computer vision[D]. Hunan: Master of Engineering Thesis, National University of Defense Technology, 2005

[3] Zhang Hongtao, Research on structured light road recognition method for ground robot[J], Microcomputer Information, 2005, 21-4, 15-17

[4] Qiu Wei. Research on highway lane departure warning system[D]. Hunan: Master of Engineering Thesis of National University of Defense Technology, 2006