CMOS Image Sensor Interface Technology in Embedded Systems

Background

At present, digital camera technology mainly adopts two methods: one is to use CCD (capacitive coupled device) image sensor, and the other is to use CMOS (complementary metal oxide semiconductor) image sensor.

CCD image sensor has the advantages of low read noise, large dynamic range, and high response sensitivity. However, CCD technology is difficult to integrate with mainstream CMOS technology in the same chip. Therefore, CCD image sensor has the disadvantages of large size and high power consumption.

CMOS image sensor is a new type of image sensor that has developed rapidly in recent years. Due to the use of CMOS technology, the pixel array and peripheral support circuits (such as image sensor core, single clock, all timing logic, programmable functions and analog-to-digital converters) can be integrated on the same chip. Therefore, compared with CCD, CMOS image sensor integrates the entire image system on a chip, which has the advantages of small size, light weight, low power consumption, convenient programming, and easy control. For handheld devices, size and power consumption are the key considerations when designing software and hardware, so CMOS image sensor will have broad prospects in handheld devices.

This paper will study the hardware interface technology and software driver of image acquisition equipment designed for CMOS image sensor in embedded system.

System hardware design

Embedded system hardware platform selection

The Motorola MC9328MX1 processor is based on the ARM920T embedded processor core and can operate at a main frequency of up to 200MHz. It integrates many modules, supports interface modules, GPIO (General Purpose I/O) interface modules, clock generation modules (CGM, Clock Generation Module), etc., and provides flexible interface control functions for various peripherals. The built-in CSI (CMOS Sensor Interface) module of the Motorola MC9328MX1 processor provides a timing control module that can adapt to the requirements of different CMOS image sensors.

COMS image sensor selection

ICM105C is a single-chip digital color image device produced by IC Media, using a 1/4-inch optical system. It has a built-in 640×480 (650×490 physical pixels) sensor array, digital-to-analog converter and correlated double sampling circuit. Its control part uses the I2C bus, and only needs an I2C bus slave device address to set and read its internal control and status registers. ICM105C outputs image data in Bayer format, in which the RGB color components can be adjusted by digital gain and can be fed back to the embedded processor for color processing or compression.

Design of interface circuit

The interface circuit design should mainly consider the interface of the image sensor power drive control circuit and the interface of the data output circuit. ICM105C uses a 3V power drive, a 24MHz clock frequency, and outputs 8-bit data. The control part mainly uses the bus. These conditions must be provided to ensure the normal operation of ICM105C.

(1) Design of the power drive circuit. Since the CMOS image sensor is a power-sensitive component, if the power supply is unstable, it will have a great impact on the imaging effect. The power supply voltage requirement of ICM105C is 3V, with a minimum of 2.8V and a maximum of 3.1V, and the digital power supply and analog power supply are separated. However, the main processing board only provides a single 3V power supply, and this power supply is also provided to the main processing board chip. Therefore, a corresponding mechanism must be adopted to ensure the stability of the power supply and the isolation of the digital power supply and the analog power supply. At the same time, another design goal of this system is miniaturization and simplification. In order to take into account both requirements, the power supply design abandons the solution of using complex voltage regulator chips and only uses capacitors and inductors to stabilize the voltage and eliminate the interference between the digital power supply and the analog power supply. Its circuit diagram is shown in Figure 1.

Figure 1 Power drive schematic

VDD 3V is the power supply provided by the motherboard, VCCD and VCCA are the digital power supply and analog power supply provided to ICM105C respectively, among which L1, L4, C1, C3 play the role of isolating digital power supply and analog power supply and filtering. Inductors are also used to eliminate interference between digital ground and analog ground. [page]

(2) Data output interface circuit design. The CSI module built into the Motorola MC9328MX1 processor provides a timing control module, which can simplify the circuit design. It only needs to connect the 8-bit data line and output clock as well as the field frequency, line frequency and pixel clock to the CSI module to ensure the correct data collection of the processor. The specific logical connection relationship is shown in Figure 2.

Figure 2 Logical connection diagram of ICM105C and main processing board

The clock line is the 24MHz clock signal input from MC9328MX1, PCLK is the pixel clock output by CMOS, VSYNC is the field frequency, HSYNC is the line frequency, and DOUT[0-7] is the output data. The timing relationship is shown in Figure 3. DOUT[7:0] is valid at the rising edge of the PCLK clock, and is valid when HSYNC and VSYNC are at a low level. In order to achieve this timing effect, some pins of the chip need to be initialized correctly. Pin 37 of ICM105C controls the synchronization mode of the data, and a pull-up resistor is connected to a high level, so that the sensor can output HSYNC and VSYNC synchronization signals.

Figure 3 ICM105C data output timing diagram

The polarity of HSYNC and VSYNC can also be configured. Grounding pins 46 and 47 will make HSYNC and VSYNC low when valid. Pin 14 is the clock selection signal. Grounding it indicates the use of an external clock, so that the internal crystal input pins 12 and 13 can be left floating.

(3) Control circuit design. For the sensor to work properly, the registers inside the chip must be initialized. The initialization must be performed through the sensor's I2C interface. ICM105C provides a hardware initialization method. If pin 33 is high when the chip is started, the sensor's I2C interface will first work in master mode and attempt to read initialization data from an external serial EEPROM. Then, the sensor returns to normal slave mode. In order to simplify the interface circuit, the sensor is directly controlled by the I2C interface of the main processing board. Grounding this pin makes it work in slave mode, so that the EEPROM circuit can be discarded.

Software-driven

CMOS image sensors require software drivers from embedded systems to work properly and output correct image data. Since the embedded system at the processing end uses the embedded Linux operating system, the driver of the I2C interface has been integrated into the operating system. Interrupt resources can be applied as resources. The operating system also provides interface functions for calling other resources, which greatly facilitates the writing of the driver. In this design, the writing method of the character device driver in the form of a module under the Linux system is adopted. The entire software driver needs to complete two functional modules: the initialization module of the interface and the receiving and outputting data.

Initialization

(1) Initialization of the CSI module. According to the data sheet of the ICM105C chip, the reset signal of the CSI needs to remain valid until the input voltage remains constant for more than two clock cycles. Figure 4 is a timing diagram.

Figure 4 Soft reset signal timing diagram [page]

Since the main processing board starts working as soon as it is powered on, the driver of ICM105C must start working after the operating system on the processing board is started. In particular, the clock signal is only provided to ICM105C after the driver is loaded. To create the timing effect of the reset signal, a software-controlled reset signal must be used. It should remain valid for at least two clock cycles after the clock is output to the sensor, and then be pulled high to make it invalid. In the design, a GPIO port of the Motorola MC9328MX1 processor is used to implement the soft reset signal.

(2) Set the clock and initialize the I2C interface. ICM105C requires a 24MHz clock. The external frequency of Motorola's MC9328MX1 processing board is 96MHz, so a quarter of the main frequency is required. Only the value of the corresponding clock generation module register needs to be set to ensure the output of 24MHz. The two signal lines SDA and SCL of the I2C bus need to set the two pins of the corresponding GPIO to use them for I2C transmission. It is also necessary to mount the I2C driver of the Linux operating system and write the I2C read and write functions. It should be noted that the I2C address value of ICM105C is 21H.

(3) Initialize the CMOS sensor. The internal control and status registers of ICM105C are initialized through the I2C bus. The initialization sequence (register address and value sequence) is provided by IC Media. After the initialization is completed, data and timing signals can be received.

Receive data

Receiving data is the most important part of the driver program. It needs to coordinate interrupts and DMA transfers to ensure that data is received correctly and can be recovered correctly when errors occur. The software flow of this part is shown in Figure 5.

Figure 5 Flowchart of receiving data software

At the beginning, interrupts and DMA resources are requested and memory space is requested to store the received data. The main task of the interrupt is to start DMA transmission at the beginning of each frame. DMA transmission mainly reads data from FIFO and saves it in memory, and handles possible errors. In this example, if a DMA transmission error occurs, the frame is discarded.

Data processing

The following is the data processing module. Since the received data is still raw data, it needs to be processed to form the final image data.

The specific processing process is as follows:

(1) Linear interpolation. Due to the problem of manufacturing process, the photosensing point in the CMOS image sensor can only place one color filter, that is, each of its physical pixels can only sense one color, R, G or B. This is the data in Bayer format (as shown in Figure 6). It must undergo interpolation operation to obtain the RGB value of each pixel.

Figure 6 Bayer pattern

As can be seen from the figure above, each pixel has 8 adjacent pixels, and the color components of these 8 pixels are different from this pixel. The interpolation algorithm is based on the principle of spatial correlation of the color values ​​of adjacent pixels. The processing method is as follows:

a. For pixels with only R color component, its G color component is calculated by the average value of the surrounding 4 G colors. The B color component is calculated by the average value of the surrounding 4 B colors.

b. For pixels with only B color component, its R color component is calculated by the average value of the surrounding 4 R colors, and its G color component is calculated by the average value of the surrounding 4 G colors.

c. For pixels with only G color component, its R color component is calculated by the average value of the upper and lower 2 R colors, and its B color component is calculated by the average value of the left and right 2 B colors. After interpolation, the RGB of each pixel is obtained, thus forming a complete image data.

(2) White balance. Any object has different color temperature under different light. In short, the so-called color temperature is a quantitative representation of color in Kelvin temperature. The higher the color temperature, the more blue the object has; the lower the color temperature, the more red the object has. Since the human eye is self-adjusting, it can correctly identify the color even if the color temperature of the object is different. However, the CMOS image sensor is not self-adjusting, so when shooting an object in outdoor daylight, the color of the object will be blue due to the high color temperature. When shooting an object under indoor fluorescent light, the color of the object will be red due to the low color temperature. To get the correct color, white balance must be performed. The basic principle of white balance is to adjust the color temperature of the color so that it remains within a specific range. A relatively simple white balance method is used in the application of this interface, and the processing process is as follows:

a. First, calculate the average value of each color component of the data of an image:

b. Calculate the maximum average value:

c. Calculate the correction value of each color data after white balance:

After such calculation, the white balanced data is obtained. At present, there is no good algorithm for white balance to handle all situations. This is just a simple algorithm.

Conclusion

The interface technology of the ICM105C image sensor proposed in this paper has been successfully applied in a two-dimensional barcode reader. For the needs of practical applications, the circuit module of ICM105C is designed to be only 35mm×35mm in size and connected to the main processing board through a 20-core cable. The power consumption is less than 50mW during normal operation, the collected image data is good, and the working mode of the image sensor can be controlled by software, which is very suitable for the application of handheld devices.