Design and implementation of real-time image processing system based on FPGA+DSP

Abstract: Aiming at the requirements of large computational load, high real-time performance and small size of image processing system, a high-performance real-time image processing system with DSP as main processor and FPGA as auxiliary processor is developed. By taking advantage of the respective characteristics of these two chips, the algorithm is divided into two parts and processed by FPGA and DSP respectively, which greatly improves the efficiency of the algorithm. The system has the characteristics of simple structure, easy implementation and convenient and flexible use. After loading the corresponding program, it can realize the processing methods such as tracking, recognition and matching of the acquired image. The design ideas and hardware structure of the system are described in detail, and the algorithm simulation and experimental verification are carried out on the hardware system. The experimental results show that the system has high real-time performance and good adaptability, which can meet the design requirements.

1 Introduction

The processing algorithm of the image processing system is complex, the computational load is large, the processing real-time performance is high, and the volume of the system is also strictly limited.

Therefore, these characteristics must be comprehensively considered when designing the system, and the chip must be reasonably selected and a certain margin must be reserved. The usual method is to use FPGA and DSP as the processors of the system, that is, FPGA undertakes the image preprocessing function, and DSP implements more complex image processing algorithms. How to expand the application of the system on such a structure, increase its flexibility and reduce the risks caused by improper early design are the key issues that need to be considered before design.

2 System structure design

In addition to the two core parts of FPGA and DSP, the real-time image processing system should also consist of an image acquisition module, an image output module, and corresponding storage parts and communication control circuits. The functions of each main module are as follows:

Image acquisition module: separates and amplifies the image input by the imager, completes A/D sampling under the control of FPGA, converts the analog signal into a digital signal, and inputs the digital signal into the FPGA module.

FPGA module: The main functions of this module are: 1. Generate the logic control signal required by the system, 2. Provide clock signals for each chip, and 3. Preprocess the input image.

DSP module: Perform complex image processing operations such as target detection, recognition and tracking on the preprocessed image. The DSP module is the core module of the entire system.

Image output module: Perform D/A conversion on the processed digital image, and form a standard video after superimposing the synchronization signal to display the processing results.

The structure of the real-time image processing system is shown in Figure 1.


Figure 1 Block diagram of real-time image processing system

3 Specific implementation scheme of the system

After clarifying the basic components of the system, the work to be done is: how to combine these parts under the condition of meeting various requirements of real-time image processing algorithms, so that the system is easy to implement and has high reliability. At the same time, each module must be carefully analyzed and improved to expand the application scope of the system. The following is the specific composition of each module.

3. 1 Image acquisition module

In order to expand the application of the system, this module has two video input channels, and one or two video inputs can be determined according to actual needs. When the system is used in a binocular imaging system, since binocular vision requires the synchronization of two imagers, the composite synchronization signal separated by EL4501 in the first video channel is connected to the synchronization input port of the second imager to achieve synchronization of the two imagers. Because the imager outputs an analog video signal, the separated video signal must be converted by A/D before entering the FPGA. The A/D sampling process is controlled by FPGA. The field synchronization, line synchronization and odd-even frame indication signals separated from the second video channel directly enter the FPGA. The FPGA uses the synchronization signal and the odd-even frame indication signal as interrupts to control the start and stop of sampling. At the same time, in order to prevent other circuits in the system from interfering with the image acquisition module, an isolation chip should be added to isolate the signal before the signal enters the FPGA.

3.2 FPGA module

FPGA is the chip in the system that realizes image acquisition, image preprocessing, and generates various control logics and clock signals. The XC4VSX35 chip selected by the system is an FPGA in the V4 series that is optimized for digital signal processing. The chip supports four loading modes: master serial, slave serial, master SelecMt AP, slave SelecMt AP and JTAG boundary scan. This system uses the master serial mode and JTAG boundary scan mode. FPGA uses a 50MHz crystal oscillator as its global clock input, and relies on the V4's on-chip digital clock manager (DCM) to generate a clock frequency of 24MHz~500MH z.

At the video signal input end, since the PAL video format uses interlaced scanning, each frame of the image is composed of odd and even frames. Therefore, a dual-port RAM needs to be configured inside the FPGA to store the image of the odd frame, and wait for the even frame to come in before combining the two frames into a complete frame of the image. The size and depth ratio of the dual-port RAM should be adjusted according to the actual needs of the algorithm. As a coprocessor of the system, the image preprocessing is also completed by the FPGA. There are many ways to exchange data between FPGA and DSP. For the image processing system, the amount of data exchanged is large and the pixel rate of the video is high. Therefore, a dual-port RAM is used as a bridge for data exchange between FPGA and DSP. The two chips each control the reading and writing of one end to complete the data exchange, which not only meets the requirements of large exchange data volume, but also facilitates implementation. The dual-port RAM uses IDT70V631S, with an access cycle of 12ns. The two ports have their own address bus, data bus and read-write control. DSP and FPGA can access and control it at their respective connected ports.

The real-time image processing system is generally a part of a control platform. The final processing result of the image processing system is transferred from the FPGA to the host computer through the multi-channel buffered serial port, and the host computer control command is also transmitted to the DSP through the serial port. Serial communication control can be implemented by FPGA programming or by using a dedicated chip. In order to save FPGA on-chip resources, serial communication control is implemented using the serial communication control chip TL16CTL16C554.

3.3 DSP module

DSP is the core processor in the system because of its high calculation accuracy and ability to complete complex image processing algorithms. The DSP chip selected in this system is TMS320C6416 (referred to as C6416) produced by TI, with a maximum operating frequency of 850MH z and a peak processing speed of 6800MSPS, which has powerful processing capabilities. The DSP core clock can be changed by setting the potential of the two pins CLKMODE0 and 1. There are four modes in total: BYPASS (x1), x6, x12 and x20. The data exchange between DSP and FPGA is through dual-port RAM (DPRAM). Although C6416 integrates up to 8M bits of on-chip high-speed cache, considering that the image processing algorithm must involve processing several frames of images before and after, in order to ensure that the storage capacity will not become the performance bottleneck (choke point) of the entire system when the system is running, additional storage space is expanded in the DSP module. Since the data width of the EM IFA port is larger, the system expands two synchronous memories with a total of 128M bits in the CE1 space of the EM IFA. There are three boot modes for C6416, namely: no loading, the CPU directly starts to execute the instructions in the memory at address 0; ROM loading, the program in the ROM located in the CE1 space of the EM IFB is first moved to address 0 through EDMA, and ROM loading only supports 8-bit ROM loading; host loading, the external host initializes the CPU storage space through the host interface, including the on-chip configuration register. This system adopts the ROM loading method. There are three multi-channel buffered serial ports in the C6416 chip. The final result processed by DSP will be transmitted to FPGA through the multi-channel buffered serial port of DSP.

3. 4 Image output module

The function of this module is to convert the image data processed by DSP into digital-to-analog and synthesize it with the character signal to form a video signal in VGA format. The digital-to-analog conversion chip selected here is ADV7125.

This is a three-channel (8-bit per channel) video digital-to-analog converter produced by ADI. Its maximum data throughput is 330MSPS, and its output signal is compatible with RS-343A/RS-170. The digital video signal generated by FPGA enters the three data channels of ADV7125 respectively. After digital-to-analog conversion, the analog video signal is output and superimposed with the original synchronization signal and blanking signal to display the processing result on the display. Like the image acquisition part, in order to avoid interference from other circuits, the image output part also needs to add a data isolation chip.

4 System simulation verification

Image processing algorithms can be divided into preprocessing algorithms and later complex algorithms. Preprocessing algorithms mainly include image correction, enhancement and simple segmentation, etc. The main function is to enhance the characteristics of the image, eliminate interference and extract useful information. The characteristics of this type of algorithm are simple algorithms but large amount of calculation, which is suitable for FPGA to complete. The preprocessed image is generally a binary image, so the redundancy of the image is greatly reduced, so the amount of data for the later complex algorithm to match, identify, track and other operations on the image is greatly reduced, and the feasibility of the algorithm is increased. In order to verify the effectiveness of this hardware system, a large number of simulation tests were carried out. Figures 2 to 5 are the test results of bridge recognition.

Among them, Figures 3 and 4 are the test results of preprocessing such as image enhancement and segmentation on the FPGA module. Figure 5 is the target recognition result based on DSP.

Figure 2 Original image Figure 3 FPGA image enhancement result Figure 4 FPGA image segmentation result Figure 5 DSP-based target recognition result

5 Conclusion

The real-time image processing system is based on DSP and FPGA as the basic structure, and is optimized on the basis of this structure, and a video input channel is added. At the same time, all data exchanges pass through FPGA. The later debugging process proves that this makes debugging very convenient, which can monitor data exchange and facilitate the correction of errors in the early design. The entire system structure is simple, and the functions of each module are clear. After a large number of system simulations in the later period, it is verified that the system has high stability and fast processing speed, which can meet the design requirements.