Real-time Video Processing and Display Design Based on SoPC

Currently, image systems based on soft-core processors have become a hot topic of research. Using FPGA to build image processing systems based on programmable systems on chip (SoPC) has become a trend [1]. Therefore, this paper adopts SoPC technology and uses IP resource reuse technology to integrate NiosII soft-core processors and various input and output interfaces on a single-chip Cyclone series FPGA of Altera, completing the functions of video image acquisition, preprocessing, storage and display. This system performs grayscale equalization on the image, making the system more widely used. Since histogram equalization can directly extract information from known images without the need for additional parameter description, it has practical significance in military, aviation, commercial and other fields, especially in the field of infrared image enhancement [2]. SoPC is a flexible and efficient design proposed by Altera. It can easily expand and modify embedded video acquisition functions without modifying hardware [3]. At the same time, due to the integration of numerous IP cores, the design is efficient and fast.

1 System overall structure and working principle

The system block diagram is shown in Figure 1. After the system is powered on, the I2C control bus in the Nios II soft core configures the video acquisition module SAA7113H. The FPGA relies on the pixel clock and the line and field signals to synchronously acquire the grayscale video data in the video signal output by the SAA7113H chip, and performs grayscale histogram statistics and grayscale distribution equalization on the acquired grayscale data. A delay module is added to the outside of the Nios II soft core. After synchronizing with the signal output by the grayscale conversion module, the video input module starts working, and the video signal is transmitted through the storage control and DMA controller. The LCD display control and other modules display the acquired signal. All processing uses pipeline operation, which greatly saves the processing time of the system.

2 System Hardware Design

2.1 Video decoding chip SAA7113H initialization

The initialization configuration of the video decoder in this system is completed by the Nios II soft-core processor through the I2C bus, which mainly includes the setting of the working mode of the video decoder, the timing relationship of the output line and field synchronization reference signals, and the format of the output digital signal. The I2C bus is a two-wire bus developed by Philips. It consists of a serial data line SDA and a serial clock line SCL, which can realize data communication and complete chip configuration. First, the I2C bus controller issues a start condition for data transmission: when the SCL signal remains high and the SDA signal changes from high to low, the address data stream begins to be transmitted. After the start condition is met, an 8-bit device slave address is issued, and all peripheral devices begin to respond to the start condition and convert the next 8-bit register address (7-bit address + 1-bit read/write bit), and transmit from high to low in sequence. After the peripheral device recognizes the transmission address, it changes the data line to a low level at the 9th clock pulse (confirmation bit), and then starts to write or read 8-bit data into or out of the register (the read/write bit determines the data transmission direction). When the clock line SCLK is at a high level and the data line SDA changes from a low level to a high level, it means that one data transmission is completed, the I2C bus is stopped, and the next transmission is waiting to start. Figure 2 is a data transmission timing diagram of the I2C bus.

The video image signals input from the four analog input terminals AI11, AI12, AI21, and AI22 of SAA7113H are converted into digital chrominance signals and brightness signals after A/D conversion, and brightness signal processing and chrominance signal processing are performed respectively. The result of brightness signal processing is sent to the color signal processor for comprehensive processing to generate YUV signals, which are output from VPO (8 bits) after format conversion. All these functions are completed under the control of the I2C bus. The register configuration of SAA7113 is carried out through the I2C bus and complies with the I2C bus protocol. Table 1 is the "write" operation format of the SAA7113H register.

Among them, S is the start bit, the condition is that SCL is high level, SDA has a falling edge, ACK-s is the slave device response bit, and P is the stop bit. During the initialization process, please note that when the node address (Slave Address) of SAA7113H is powered on and RTS0 is high level, its I2C write address is 48H and read address is 49H; when RTS0 is low level, its I2C write address is 4AH and read address is 4BH.

2.2 Grayscale histogram statistics and equalization

The image information obtained through the input system contains various noises and distortions. For example, uneven illumination will cause the image grayscale to be too concentrated. The image obtained by the CCD will produce noise pollution after A/D conversion and line transmission, which will inevitably affect the clarity of the system image and reduce the image quality. However, image quality can be improved through image enhancement. The histogram equalization algorithm is an important algorithm in spatial domain image enhancement technology. It is the basis of subsequent image processing such as image compression, image segmentation and image recognition [2, 4] and is widely used in image preprocessing technology.

2.2.1 Histogram Equalization Principle

Histogram equalization is based on probability theory and uses grayscale point operations to achieve histogram transformation. The histogram of the original image contains rich image information, describes the grayscale content of the image, and reflects the grayscale distribution of the image. The basic idea of ​​histogram statistics and equalization is to widen the grayscale with a large number of pixels in the image and reduce the grayscale with a small number of pixels, so as to achieve the purpose of clear image[5]. Through point operations, the input image is converted to have the same number of pixels at each grayscale. The following formula is used in image enhancement processing:

(3) Use the cumulative distribution function as the transformation function to transform the image grayscale.

2.2.2 Algorithm optimization and implementation based on FPGA

The key to the algorithm is to implement the histogram statistics and the mapping table after equalization in FPGA. This design uses the Block RAM provided by the FPGA device to store the histogram and the mapping table after equalization. With the array structure of the dual-port RAM as the core, the entire algorithm is divided into two parts for execution: (1) Counting the grayscale values ​​of each level of the original image and storing them in the grayscale counting table RAM; (2) Performing grayscale mapping on the obtained grayscale values ​​and storing the mapped results in the mapping table RAM, that is, generating a balanced mapping table. According to the mapping table, the grayscale level of a certain grayscale level of the original image after transformation can be known. After such processing, the densely distributed grayscale values ​​in the original image can be mapped to the new grayscale level after histogram equalization. Figure 3 is the logic block diagram of the algorithm.

Considering the hardware characteristics of FPGA, when receiving image data and calculating the histogram of this field, it is relatively complex to implement floating-point data operations in FPGA and consume more logic resources. Therefore, when performing histogram statistics on the image, the number of pixels at each gray level is used as the histogram data: pu(uk)=nk to avoid using floating-point data operations. The histogram is equalized to obtain a mapping relationship table from the original grayscale to the equalized grayscale. The specific state transition diagram of the histogram equalization algorithm in this design is shown in Figure 4. The specific transfer conditions are as follows: when the start signal is valid, it switches from st1 to st2; when counter 1 counts to 255, it jumps to st3; when counter 1 does not count to the image height minus 1 or counter 2 does not count to the image width minus 1, it jumps to st3; when counter 1 counts to the image height minus 1 and counter 2 counts to the image width minus 1, it jumps to st5; when counter 1 does not count to 255, it jumps to st6; when counter counts to 255, it jumps to st7; when counter 1 counts to the image height minus 1 and counter 2 counts to the image width minus 1, it jumps to st1 state; when counter 1 does not count to the image height minus 1 or counter 2 does not count to the image width minus 1, it jumps to st6. When counting statistics, after the counter detects the grayscale of a pixel, it not only adds 1 to the counter accordingly, but also adds 1 to all counters whose corresponding grayscale values ​​are greater than the current grayscale value. In this way, the statistics and accumulation of the grayscale values ​​of each pixel of the original image can be completed at the same time, reducing the statistical time.

Because the grayscale equalization process is performed in units of fields, after the grayscale equalization process is completed, the fields need to be merged into frames. The SDRAM controller in the Nios II soft core completes the merging of odd and even fields. The simulation result of grayscale equalization is shown in Figure 5.

3 Video Input Module

The structural diagram of the video input module is shown in Figure 6. After grayscale conversion, the YcrCb4:2:2 format video signal is input into the FIFO buffer under the control of the pixel clock. The color bar test signal module gives a very simple test signal during system testing, which can be simulated as a signal source to facilitate system debugging. The chromaticity conversion module converts the YCrCb format into RGB format and writes its value to the FIFO buffer. Avalon DMA writes the image data to the system memory (SDRAM), and when a frame of image needs to be written, an interrupt signal is given to the Nios II processor.

In the color space conversion module, the resources in the FPGA chip are used to construct a multiplication and addition device using MegaCore to complete the operation. According to the proportional relationship of YUV in the conversion matrix, the signal is amplified by a certain multiple to make it close to an integer value. The maximum value of the YUV signal is 255, but 10-bit DATA can receive 1,023 brightness level adjustments, so this ratio can be amplified by about 4 times (if it exceeds 1,023, it is calculated according to the level of 1,023). After repeated experiments, it was finally concluded that the following operation rules are closest to the conversion matrix:

4 Nios II System Generation

SoPC Builder can be used to design system module hardware and generate underlying software. When designing hardware modules, SoPC Builder provides a graphical configuration interface and some commonly used peripheral IP modules, such as SDRAM, Flash RAM, UART, Interval timer, Parallel I/O, etc. The components contained in the Nios II soft core are shown in Figure 7.

There are no IP cores for I2C configuration module, video input module and LCD controller module in the IP core library of SoPC Builder. These modules are designed based on register characteristics and functional requirements, and connected to the SoPC system through the Avalon bus in the form of IP cores. After building a SoPC system based on the Nios II processor, some system settings are required to generate the final Nios II system. Therefore, in addition to peripheral settings, system configuration also includes settings for the startup program, interrupt vector table, system startup address, etc.

This article introduces a video acquisition solution based on SoPC technology and the implementation of grayscale histogram statistics and grayscale equalization of video signals. This solution combines the advantages of SoPC technology in software and hardware tailorability, upgradeability, and expandability, greatly shortening the overall system design cycle and has good application prospects and scientific research value.