Implementation of Bayer Transform of Color Image Based on FPGA

0 Introduction

Image engineering has been widely used in many fields such as national defense, education, finance, medical treatment, printing, intelligent transportation, industrial automation, consumer electronics, etc., and has developed rapidly. As we all know, image sensors, as important components of image systems, are basically divided into two categories: CCD sensor or CMOS sensor. Both use photodiodes for photoelectric conversion to convert images into digital data, and the main difference is the different ways of data transmission.

The sensor3 color digital camera needs three monochrome sensors to obtain the R, G, and B components of the color image, which is relatively expensive. The method of obtaining a color image with a single CCD is to cover the surface of the CCD with a mosaic filter containing only red, green, and blue colors, and to implement its output signal through a certain processing algorithm. This design concept was originally proposed by Bayer, so this filter is also called the Bayer pattern.

LatticeECP FPGAs combine efficient FPGA architecture with high-speed application-specific functions. LatticeECP-DSP (EConomy plusDSP) is the first product family, which integrates dedicated high-performance DSP blocks on the chip. LatticeECP-DSP devices are best suited for cost-effective DSP functional application systems, such as software-defined radio, wireless communications, military, image and video processing systems, etc.

1 Bayer interpolation method

The arrangement format of the Bayer Pattern is shown in Figure 1.

Although the original image captured by a single sensor camera with a Bayer filter contains the three primary color components of R, G, and B, it is not possible to simply separate the three components without any algorithm processing. This not only results in poor image resolution, but also serious distortion of the three primary color components of each pixel compared to the object being photographed. In Figure 2, Figure 2(a) is the original color image; Figure 2(b) is an image that only takes the red component and is displayed in grayscale mode; Figure 2(c) is an image that only takes the green component and is displayed in grayscale mode; Figure 2(d) is an image that only takes the blue component and is displayed in grayscale mode. After simply superimposing the images of Figures 2(b) to (d), the original image Figure 2(a) can be obtained. However, the original Bayer image captured by the CCD or CMOS sensor does not conform to this separation principle and must be implemented through a certain image algorithm.

There are many Bayer interpolation methods widely used in the field of image processing, including M. C. Poilpre's processing of JPEG images; H. S. Malvar, etc.'s linear interpolation method; Remi Jean's pixel double interpolation method and T. Guseo's low-resolution image processing. There are three representative methods: bilinear interpolation method, Ron Kimmel method and OptimalRecovery method. Each of these three methods has its own advantages and disadvantages. [page]

1.1 Bilinear interpolation

As shown in Figure 3, each pixel position originally has only one color component, and the missing two color components are obtained by the average value of pixels with the same color components in the 3×3 neighborhood. The R, G, and B components of the pixels at B7 and G3 in Figure 3 are calculated by the following formula:

Bilinear interpolation has the advantages of simple operation and easy implementation. Its essence is a low-pass filter, and its disadvantage is that it ignores the relevant information between different color components and the edges of the image. In this way, erroneous data will cause blurring or even color aliasing at the edges of the restored image.

1.2 Ron Kimmel Method

Ron Kimmel's method calculates the gradient of the green pixel in all directions, and then weights and averages the gradient values; for red and blue pixel components, the ratio of the red and blue components to the green component is taken as the weighted average. This method can significantly improve the color aliasing of the edges of objects in the image.

1.3 Optimal Recovery Method

The Optimal Recovery method has a high computational complexity, but it is also the algorithm with the best image quality that has been published publicly. It is difficult for general embedded systems to complete it in real time. The image processing system in this paper uses Lattice's FPGA chip LFECPIIM50, making full use of the natural parallel structure of FPGA to implement the Bayer conversion algorithm in real time (1208×1024 images, 12 frames/s), and has achieved good results. The Optimal Recovery algorithm is as follows:

(1) As shown in FIG4 , the green pixel interpolation at P5 shown in the figure is completed by using the fine scale mode as much as possible.

(2) Calculating the blue component requires two steps:

① Retrieve the blue component lost in the red space:

②Recover the blue component lost in the green space:

(3) The red component is processed in the same way as the blue component.

(4) Repeat the following calibration steps 3 times:

① Correct the green component to obtain a suitable G/B ratio:

② Correct the green component to obtain a suitable G/R ratio:

③ Average blue and red interpolation results:

④Correct the blue and red component values:

⑤The algorithm ends, where R5, G5, and B5 are the three-color component values ​​at pixel P5.

1.4 Peak Signal-to-Noise Ratio (PSNR)

The calculation formulas for peak signal-to-noise ratio PSNR and normalized color difference NCD are:

Where: MSE is the mean square error between the original image and the processed image; I represents the nth pixel value of the original image; P represents the nth pixel value of the processed image; Framesize represents the image size, such as 1 280×1 024; the larger the PSNR value, the less distortion. [page]

2 Results Analysis

The images acquired by the real-time image acquisition system are shown in Figure 5, where Figure 5 (a1), (b1), and (c1) are interpolated images with bright colors and good subjective feeling. Figure 5 (a2), (b2), and (c2) are the acquired original Bayer images. After partial magnification, the original mosaic effect can be seen, as shown in Figure 5 (a3), (b3), and (c3). The peak signal-to-noise ratio calculation results are shown in Table 1, where PSNR1 is the bilinear interpolation method and PSNR2 is the OptimalRecovery method.

3 Conclusion

At present, FPGA technology is developing rapidly, internal resources are becoming more and more abundant, and the cost performance is constantly improving. Fully exploring the potential of FPGA resources and completing the image preprocessing algorithm at the front end of the system as much as possible will greatly improve the real-time characteristics of the system and expand the application field of embedded systems. Although the focus is on the Bayer interpolation transform of color images, it also has positive reference significance for other embedded system design methods using FPGA devices.