Using FPGA to replace DSP for real-time image and video processing

With the further development of digital convergence, the design and implementation of the system requires greater flexibility to solve the many problems caused by integrating completely different standards and requirements into similar products. This article introduces the application of FPGA in video processing. Compared with ASSP and chipset solutions, FPGA can provide different levels of flexibility according to the actual needs of current design engineers and maintain significantly better performance than traditional DSP.

Real-time video processing places extremely high demands on system performance, so almost all general-purpose DSPs with only the simplest functions do not have this function. Programmable logic components allow designers to implement video signal processing algorithms using parallel processing technology and achieve the desired performance with only a single component. DSP-based solutions usually require embedding many DSPs on a single board to obtain the necessary processing power, which will undoubtedly increase program resource overhead and data memory resource overhead.

By performing video processing in a real-time environment, system engineers can minimize frame memory requirements and data buffering requirements, which are not required in non-real-time applications. As the industry actively promotes the development of high-quality video and the continuous improvement of compression formats, system processing speeds are also increasing. Programmable logic components also adopt the FPGA component architecture widely used in dedicated paging equipment. Because the development of FPGA processes follows Moore's Law, new products can achieve the same functions and performance at a more attractive cost advantage than earlier developed products.

A major driver of this trend is the convergence of networking, broadcasting, processing and display technologies, known in the industry as 'digital convergence.' Because it is extremely difficult to send high-bandwidth video data over very narrow transmission channels (such as wireless channels) and maintain appropriate quality of service (QoS), designers have conducted extensive research to improve error correction, compression and image processing techniques, all of which are based on FPGA implementation.

Image compression/decompression - DCT/IDCT The main video compression format currently used in digital video systems is MPEG2, which has been widely used in digital televisions, video converter boxes, digital satellite systems, high-definition television (HDTV) decoders, DVD players, video conferencing equipment and *. The original digital video information always needs to be compressed so that it can be transmitted through appropriate transmission channels or stored in appropriate media (such as disks). In addition, there are many new standards that are about to be or are being launched, including the most eye-catching MPEG4, but most products based on this technology are still under development.

The core of the MPEG2 and MPEG4 algorithms is an operation called discrete cosine transform (DCT). The basic principle of DCT is to take the square of the pixel block and remove the redundant information that is not noticeable to the observer. In order to decompress the data, an inverse discrete cosine (IDCT) operation is also required.

Although the DCT part of the MPEG algorithm has been standardized and can be effectively implemented in FPGA, there are still many parts of MPEG encoding that have not been clearly defined. It is these unclear parts that allow a company's products to be differentiated from competitors and develop proprietary algorithms. Many dedicated MPEG decoders use FPGAs in these parts (such as motion estimation modules). Because FPGAs are reconfigurable, components can be easily refreshed and new algorithms can be integrated throughout the development phase (including after configuration), while companies that rely entirely on standard ASSP solutions cannot develop similar products due to their own limitations, so the market risk is greater (Figure 1).

[page]

Color space conversion

Another important part of the video system is color space conversion, which is the process that specifies how an image is represented, such as converting from one color format to another different color format.

The human eye sensors can only detect visible light with wavelengths between 400nm and 700nm. These sensors are called cones and there are three different types: red cones, green cones and blue cones. If a single wavelength of light is visible, the relative reactions of these three sensors allow us to identify the color of the light. This phenomenon is of great practical value because it means that we can produce a variety of colors of light simply by mixing the above three lights in different proportions. This is the famous three-color principle, which has been widely used in color TV systems.

We can represent various colors by drawing a ratio diagram of the three primary colors (red, green, and blue, abbreviated as RGB) in a 3D cube, where black is at the origin and white is at the diagonal corner of the origin. The resulting cube is the famous RGB color space.

Regardless of whether the final display medium is paper, LED, CRT or plasma display, the image can always be subdivided into a large number of pixels (for example, HDTV can have 1920×1080 pixels). At the same time, there are some differences between each medium, but the basic principle is that each pixel is composed of a certain proportion of red, green or blue, and the proportion of composition depends on the voltage signal driving the display.

When processing images in RGB format, each pixel is determined by three 8-bit or 10-bit words corresponding to the three primary colors, which is not the most efficient processing method. In this format, pixels must be operated on all red, green and blue channels, and the storage space and data bandwidth required will undoubtedly be larger than other color formats available. To solve this problem, many broadcast standards (such as PAL in Europe and NTSC television systems in North America) use luminance and chrominance video signals. Therefore, a mechanism for converting between different color formats is required, namely color space conversion (Figure 2).

Implementing these circuits in hardware is relatively simple, as you only need to know the coefficients to map from one format to another. The most common conversion is from RGB to YCbCr (and vice versa). Studies have shown that 60% to 70% of the brightness information (Y) perceived by the human eye comes from green light. The red and blue channels are actually just copies of the brightness information, so this duplicate information can be completely removed. The end result is that the image can be represented by signals that represent chrominance and brightness. In this format, the 8-bit system specifies that the brightness value range is between 16 and 235, while the Cb and Cr signals range from 16 to 240, and 128 is specified to represent 0 brightness.

The following equation can be used to convert the color in the YCbCr color space to the color in the RGB color space:

R\'=1.164*(Y-16)+1.596*(Cr-128)

G\'=1.164*(Y-16)-0.813*(Cr-128)-0.392*(Cb-128)

B\'=1.164*(Y-16)+1.596*(Cr-128)

R\'G\'B\' represents the RGB value of image grayscale correction. For example, there is nonlinearity between the signal amplitude and output density of CRT monitors. If image grayscale correction is performed before displaying the signal, the relationship between signal amplitude and output density can be made linear. The output gain must also be limited to a certain critical value to reduce transmission noise in the dark area of ​​the image (Figure 3).

There are many possible ways to implement the required multiplication function, such as using memory, logic components, or embedded multipliers. Obviously, the 74.25MHz data rate required for HDTV systems can be easily achieved, and different design trade-offs can be tried, such as the trade-off between system accuracy and design range. For example, in order to ensure a 3% conversion error, the design size of the YCbCr to RGB color space converter can be reduced by at least half. This may be beyond the range that most display products can afford, but it can still be accepted by other application systems, such as machine vision or security systems. The system architecture using FPGA can adjust the algorithm of the application system to achieve the best performance and efficiency. [page]

Real-time image and video processing capabilities

The inherent performance limitations of standard DSPs have led to the development of specialized chips designed to solve this problem, such as media processors. However, the extremely low flexibility of these components often limits them to very limited applications and easily leads to performance bottlenecks. The limitations of processor-based approaches are particularly evident in high-resolution systems, such as HDTV and medical imaging systems. Fundamentally, the resolution of the processor is limited by the number of clock cycles allocated to each tap of the filter or each conversion stage. Once the limit is reached, there is usually no choice but to add additional DSP components. FPGAs can provide high-performance and high-efficiency products with the most practical value through custom adjustments. Designers can compromise between application range and rate, thereby implementing specified functions at a rate much lower than the DSP clock.

For example, Visicom found that in a median filter application [3], a DSP processor requires 67 clock cycles to execute the algorithm, while an FPGA only needs to operate at 25 MHz because the FPGA can implement the function in parallel. The DSP that implements the above function must operate at 1.5 GHz, so in this specific application, the processing power of the FPGA solution can reach 17 times that of a 100 MHz DSP processor.

Many real-time image and video processing functions are suitable for implementation with FPGA components, including: image rotation, image scaling, color correction and chroma correction, shadow enhancement, edge detection, histogram functions, sharpening, median filters, and speckle analysis.

Many functions are targeted at specific applications and systems and are built on core architectures (such as 2D-FIR filters). We can quickly implement these functions using DSP blocks in HDL design languages ​​or advanced core design tools (such as Xilinx CoreGen software). In addition, we can further reduce design and simulation time by using Matlabs Simulink and Xilinx SystemGenerator tools through system-level design methods.