Application of digital CMOS cameras in smart cars

Freescale's Smart Car Competition has been successfully held for 4 times. Most of the participating teams that use cameras as their main sensors have chosen analog CCD or analog CMOS cameras. This article introduces the performance characteristics and working mode of the digital CMOS camera MT9M011, gives the application scheme of MT9M011 in the smart car control system based on HCSl2 microcontroller, and analyzes the advantages and disadvantages of digital cameras.

1 Features of CMOS image sensor

CMOS image sensors can integrate pixel arrays and peripheral support circuits (such as image sensor core, single clock , all timing logic, programmable functions and A/D converters) on the same chip through CMOS technology. Compared with CCD (capacitive coupled device) image sensors, CMOS image sensors integrate the entire image system on a chip, with the advantages of small size, light weight, low power consumption, convenient programming, and easy control. At the same time, it can be configured through I2C, SPI and other interfaces , and its exposure time, gain control and other functions are highly controllable. Therefore, the application of CMOS image sensors has become more and more extensive.

2 Performance characteristics and working mode of MT9M011

2.1 Basic parameters of MT9M011

MT9M011 is a 1.3 megapixel digital CMOS camera with RGB primary colors launched by Mieron, which has programmable control and digital signal output functions. Its output is Bayer color format, the image size is 4.6 mm×3.7 mm, the pixel size is 3.6 μm×3.6 μm, the maximum resolution is 1 280×1 024, and the maximum supported clock frequency is 25MHz; the A/D conversion accuracy is 10 bits, the maximum signal-to-noise ratio is 44 dB, and the power consumption is 129 mW in the highest resolution mode.

MT9M011 outputs a digital signal that is scanned line by line. It samples and quantizes the analog video signal through the built-in 10-bit ADC and then outputs a 10-bit data stream synchronously. At the same time, MT9M011 also provides pixel clock (PIXCLK), line valid signal (LVAL), frame valid signal (FVAL), and I2C protocol pins SCLK and SDAT required to configure MT9M011.

2.2 MT9M011 programming functions

MT9M011 has 25 registers, involving various aspects of the camera, which are read and written through the I2C compatible serial bus . The following registers are introduced in conjunction with the smart car competition application:

① Row start register (Ox01) and column start register (Ox02). These two registers determine the starting point coordinates of the output image. These two registers can be used to adjust the overall position of the image captured by the camera.

② Row width register (0x03) and column width register (0x04). One of the major features of MT9M011 is that the image size of the output data can be adjusted arbitrarily. Through these two registers, the appropriate image range can be set for the current application environment (i.e. the track).

③Exposure time register (Ox09). The value of this register determines the exposure time of the photosensitive component when the camera collects a frame of image. By adjusting the value of this register, the imaging quality of the image can be directly adjusted.

④ Data read mode register (Ox20): This register can reduce the output row data and column data to 1/2 or 1/4, and can also make the output image mirror symmetrical.

⑤Gain setting register. It contains five registers: Green1 component gain setting register (Ox2B), Blue component gain setting register (Ox2C), Red component gain setting register (Ox2D), Green2 component gain setting register (Ox2E) and global gain setting register (Ox2F). Since the RGB components show different values ​​for the same light source, the values ​​of the first four registers need to be adjusted according to different light sources, which is also the biggest disadvantage of this camera.

2.3 Digital Image Output of MT9M011

2.3.1 Bayer color format output

The output of MT9M011 is Bayer color pattern. This output format directly converts the analog level value of the reserved primary color corresponding to each pixel on the filter array into A/D and outputs it in sequence. Figure 1 shows the local pixel matrix corresponding to this output format. The black pixel is the first point to output valid data.

2.3.2 MT9M011 output signal timing

The pixel data output timing and the frame and line valid signal timing are shown in Figure 2 and Figure 3 respectively. The pixel clock is in the same frequency as the main clock. When a frame image starts to be output, the frame valid signal changes from low level to high level, and changes from high level to low level when a frame output ends; while the line valid signal changes from low level to high level when a line of data output is valid, and changes from high level to low level after a line of data output is completed. According to the output signal timing of MT9M0ll, the entire frame image can be correctly collected. In Figure 3, P is the frame blanking area, A is the valid data area, and Q is the line blanking area.

3 Application of MT9M011 in Intelligent Vehicle Control System

3.1 Hardware Design

The level of MT9M011 is 3.3 V, while the I/O level of HCSl2 microcontroller system is 5 V, so the level of camera output data needs to be raised to 5V through a level conversion chip. Combining the timing relationship of MT9M011 output signal and the characteristics of HCSl2 microcontroller, this paper uses the ECT module of HCSl2 to extract the rising edge of MT9M011 frame valid signal, and uses the common I/O port function of the corresponding pin of ECT module to collect the rising edge of line valid signal. The data output signal of MT9M011 has 10 bits. In view of the relatively simple track environment, only the upper 8 bits are intercepted as the camera signal output, which can not only ensure the image resolution, but also save the storage space of HCSl2. HCSl2 microcontroller itself has a standard I2C module, which is directly used to connect with SCLK and SDAT of MT9M011.

3.2 Software Design

3.2.1 HCS12 MCU I2C module configuration

The HCSl2 microcontroller provides a standard I2C module. The SCLK pin can generate a variety of clock frequencies, supports master-slave mode, and has a variety of flags for query. The following tasks need to be completed when initializing the I2C module:

① Set the IBFD register so that the generated SCLK signal can configure the MT9M011 normally;

②Set the IBEN register bit to enable the I2C module;

③ Set IBCR to determine the master/slave mode, receive/transmit mode, and whether to enable the interrupt function of the I2C module.

3.2.2 MT9M011 Image Data Acquisition

Due to the constraints of the microcontroller clock, the general I/O port of HCSl2 cannot detect the camera pixel clock signal jump. This paper adopts the method of collecting a row of data in a fixed period. Therefore, the key to collecting pixel data is to accurately collect the rising edge of the frame valid signal and the rising edge of the row valid signal. The selected solution is: use the ECT module of the HCSl2 microcontroller to collect the rising edge of the frame valid signal, and use the ECT module common port mode to collect the rising edge of the row valid signal.

The acquisition object requires that the vertical and horizontal resolutions must not be too low, but the internal storage space of HCSl2 is limited and cannot provide favorable support for high resolution.

The correlation with the values ​​of the surrounding points and the camera's interlaced row and column output functions are used for skipping acquisition. The 1 280x 480 resolution and the mode of skipping 4 rows and columns are determined.

At the same time, the software performs another interlaced acquisition, and finally obtains 80×60=4 800 pixel data. The data acquisition process is shown in Figure 4.

3.2.3 Problems encountered when acquiring images and their solutions

(1) Selection of camera crystal

The choice of MT9M011 crystal is a very important issue. If the time of one frame of image exceeds 20 ms, its main advantage cannot be realized. At the same time, the limitation of HCSl2 microcontroller should also be considered. The bus frequency of HCSl2 can reach up to 32 MHz. Under this condition, if the crystal of MT9M011 is too large, the number of points collected in each line will be too small.

The time required for MT9M011 to output one frame of image can be calculated by formula (1):

Wherein: ColNum+HBlanking and RowNum+VBlanking represent the total number of columns and rows containing blank data respectively; fcamera represents the camera crystal oscillator frequency. When the size of a frame image is determined, the larger the value, the shorter the frame time.

The number of pixels collected by a single-chip microcomputer row can be calculated by formula (2):

In the formula: RowPixNum represents the data that can be collected in one row; ColNum represents the number of pixel data output in each row; Tcamera is the camera crystal, the reciprocal of fcamera; Cycle represents the number of machine cycles used in one acquisition cycle, which is fixed in the entire acquisition process; fbus represents the bus cycle of the microcontroller. From formula (2), it can be seen that when fbus is constant, the smaller the camera, the less data collected in one row, which is not conducive to data analysis.

After comprehensive consideration, it was finally decided to use an 8 MHz crystal oscillator as the clock of MT9M011, and a 16 MHz crystal oscillator as the clock of HCSl2. Collecting 80 points in a single line can meet the needs of path identification.

(2) The setting of camera exposure time and gain MT9M011 is a color camera with RGB primary colors, and the collected data is in Bayer color format. Since the extracted information is the black line on the track, the image environment is relatively simple, so the value of each component is directly processed as the grayscale value of the pixel at that point.

As shown in Figure 5 (a), the image acquired using the default exposure time and gain has obvious interlaced noise effects, and there is also obvious interference noise between lines. This is because the same light source has different values ​​for the three RGB components. Directly treating them as grayscale values ​​will result in an uneven and noisy restored image. The solution is to use multiple

In practice, the exposure time of the camera and the gain value of each RGB component are adjusted to make the values ​​of the three components under the same light source basically consistent. The image obtained after adjustment is shown in Figure 5 (b).

4 Comparison between digital CMOS cameras and analog cameras

The biggest advantage of the digital CMOS camera MT9M011 is that it saves time. Choosing the right crystal oscillator and image size can limit the entire car control cycle to about 20ms. The author has done an experiment, using a 20MHz crystal oscillator as the camera clock and multiplying the microcontroller to 32 MHz. Under this condition, the response time of a control cycle can even be shortened to about 4ms, which is significantly better than the 40 ms frame image of the analog camera. Another advantage of MT9M011 is that the image size can be set arbitrarily, so the author can adjust the image size and the height of the field of view by software, and the interface between MT9M011 and the microcontroller is simple, which greatly reduces the hardware burden.

MT9M011 also has some disadvantages in the application of smart car competition. It cannot automatically adapt to various light sources and needs manual adjustment; at the same time, its dynamic characteristics are not as good as those of CCD cameras. This can be solved by increasing the camera crystal clock to shorten the acquisition cycle.

Conclusion

This paper introduces the application of digital CMOS camera MT9MO11 in intelligent car based on HCS12 single chip microcomputer, and compares the characteristics of digital camera and analog camera. Practice shows that it is feasible to use digital CMOS camera as path recognition sensor for intelligent car.