Design of real-time image acquisition and transmission system based on ARM and FPGA

The rapid development of image processing technology has made image acquisition and processing systems more and more widely used in improving the degree of automation in agricultural production. At present, some image acquisition systems are based on CCD cameras, image acquisition cards and computers, some are based on CCD cameras, decoders, FPGAs and DSPs, and some are based on CMOS image sensor chips, FPGAs and DSPs. They have their own advantages and disadvantages in terms of real-time performance, flexibility and maintainability. In agricultural production, most systems are based on CCD cameras, image acquisition cards and computers. This paper combines the needs of front-end image processing and image data transmission in actual systems. Taking full advantage of the flexibility of ARM and the parallel characteristics of FPGA, a fast image acquisition and transmission system based on ARM+FPGA is designed. The selected ARM (Advanced RISC Machines) architecture is a 32-bit embedded RISC microprocessor structure, which has a rich instruction set and flexible programming. FPGA (Field Programmable Gate Array) has great advantages in speed and parallel computing, which is suitable for the real-time requirements of image processing. This paper selects Intel's Xcale PXA255 and Xilinx's Spartan-3XC3S1000 to implement the design of this system.

1 System structure design

The structural block diagram of this system is shown in Figure 1. In the figure, the image sensor module is responsible for image acquisition, FPGA is used to control the CMOS image sensor chip, ARM is responsible for image data exchange, Ethernet chip control and UDP/IP protocol implementation, Ethernet module mainly realizes Ethernet data transmission, SDRAM is used to store image data, and FLASH is program memory. When the system is working, FPGA first stores the data collected by the CMOS sensor into the dual-port SRAM, and then ARM reads the data from the dual-port SRAM of FPGA and stores it in SDRAM. After storing a frame of image data, ARM sends the data to the host computer through the Ethernet chip.

2 System Hardware Design

2.1 CMOS Image Sensor Module

This system uses OV9650 as the image acquisition sensor. OV9650 is a color CMOS image sensor from Omni Vision, which supports SXVGA, VGA, QVGA, QQVGA, CIF, QCIF, QQCIF modes and SCCB interface, and has functions such as automatic exposure control, automatic gain control, automatic white balance, automatic bandpass filtering, and automatic black level calibration. The maximum frame rate of OV9650 is 30 fps in VGA format and 15 fps in SXVGA format. This system uses VGA format.

2.2 FPGA Module

FPGA is mainly used for image sensor control, image data cache and peripheral chip timing generation. Considering the implementation needs of image processing algorithms, this system uses Xilinx's XC3S1000, which belongs to the SPARTAN-3 series and has a capacity of 1 million gates. In this design, FPGA usually runs at 80 MHz. The interface circuit diagram of XC3S1000 and OV9650 is shown in Figure 2. The internal structure of FPGA includes CMOS control unit, SC-CB interface unit, dual-port RAM unit, FIFO unit, etc. The CMOS control unit also includes frame synchronization module, field synchronization module, pixel clock module, etc.

2.3 ARM Processor Module

The ARM processor of this system uses PXA255, which is used to realize functions such as image data exchange and Ethernet data transmission. PXA255 is a microprocessor based on the fifth-generation ARM RISC architecture ARMV5TE launched by Intel. The CPU has a maximum operating frequency of 400 MHz, with 32 KB instruction cache, 32 KB data cache, 2 KB micro-number cache, and rich serial peripheral interfaces, and can support various memory chips. This system uses two SDRAMs and two FLASH memories, of which SDRAM is used for temporary storage of image data. This system uses K4S561632C-TC75 from SAMSUNG, with a capacity of 32 MB. The FLASH program memory uses Intel's E28F128J3A-150 as NOR FLASH, with a capacity of 16 MB.

2.4 Ethernet transmission module

The Ethernet chip of this system is LAN91C113. This is a fast Ethernet 10/100 Mbps controller launched by SMSC, which supports a variety of embedded processor host interfaces. LAN91C1B has an 8 KB FIFO inside, which can be used to store received and sent data. In addition, the LAN91C113 chip also integrates the MAC (media access control layer) and PHY (physical layer) of the CSMA/CD protocol, and the IEEE802.3/802.3u-100BASE-TX/10Base-T specification. In this system, the serial EEPROM interface of LAN91C113 is connected to AT93C46. PXA255 accesses LAN91C113 using asynchronous static memory. The address enable signal AEN of LAN91C113 is connected to the static chip select four of PXA255. The schematic diagram of the interface between PXA255 and LAN91C113 is shown in Figure 3.

3 Working principles and implementation of main modules of the system

3.1 Implementation of FPGA Image Acquisition

The data output of OV9650 adopts Bayer raw data output format, and each pixel outputs only one color at a time. The odd-numbered scan lines output RGRG..., and the even-numbered scan lines output GBGB... The FPGA is responsible for collecting the image sensor data.

After power-on, the system first initializes the CMOS image acquisition chip to determine its working mode. These parameters are controlled by the corresponding register values ​​inside the OV9650. The FPGA completes the parameter configuration by controlling the SCCB bus.

After the system configuration is completed, image data can be collected. Figures 4 and 5 are timing diagrams for data collection and output. VSYNC is the field synchronization signal. HREF is the horizontal synchronization signal. PCLK is the pixel data output synchronization signal. When HREF is high, effective data collection can begin, and the arrival of the falling edge of PCLK indicates the generation of data. For each falling edge of PCLK, the system transmits one bit of data. During the period when HREF is high, the system can transmit a total of 640 bits of data. In one frame of image, that is, during the period when VSYNC is low, HREF will appear high 480 times. The arrival of the next rising edge of the VSYNC signal indicates the end of the image acquisition process with a resolution of 640×480.

3.2 Data Exchange between FPGA and ARM

An 8 KByte dual-port SRAM is used as the image storage area inside the FPGA. In order to realize the real-time acquisition and processing of image data, the acquisition of image data and the reading of external image data should be carried out simultaneously. Therefore, this system adopts a double buffer structure. The implementation method is to divide the 8 KB SRAM into two 4 KB SRAMs (set as SRAM1 and SRAM2), and each SRAM stores six lines of image data at a time. In this way, at the same time, one piece can be used to store image data, and the other piece can be used for external ARM to read image data. The two SRAM storage areas are switched in a ping-pong manner. When the image data fills up SRAM1, the FPGA sends an interrupt signal to the ARM. After that, the ARM responds to the interrupt and reads the image data in SRAM1 and writes it to SDRAM. After that, the image sensor data will be written to SRAM2. When the image data fills up SRAM2, the FPGA also sends an interrupt signal to the ARM. The ARM responds to the interrupt and reads the image data in SRAM2 and writes it to SDRAM. After that, the image sensor data will be written to SRAM1 again.

3.3 Implementation of Ethernet Data Transmission

In order to realize the network transmission of image data, this system has developed a UDP/IP protocol implementation program based on PXA255 to store image data in SDRAM. A complete data frame format includes an Ethernet header, an IP header, a UDP header and a line of image data. The system workflow diagram is shown in Figure 6. After power-on, the system will wait for the FPGA interrupt. If an interrupt occurs, PXA255 will read the data in the FPGA and write it to SDRAM. Then it will determine whether a frame of image data has been read. If it has been read, a UDP packet will be sent and the image data will be sent to the host computer via Ethernet. Otherwise, it will continue to wait for the FPGA interrupt.

4 Conclusion

This paper introduces the design of a real-time image acquisition and transmission system implemented using ARM and FPGA. This system design uses FPGA technology to provide a platform for the real-time implementation of image processing algorithms such as crop recognition and weed recognition.