Using FPGA to implement camera sensor interface

The image sensor is arguably the most important part of the video or still image processing pipeline in a digital video or still camera. Without the sensor, there is no image signal to process. Sensors are notoriously non-standardized. In the scheme they employ, they differ in the following ways:

The method of converting visible or infrared light into an electrical signal; especially the method by which the signal is encoded and (sometimes) compressed before it leaves the chip.

A way to program registers inside the sensor to adjust gain, exposure time, sensor mode (such as linear, HDR), sensor image coordinates, etc.

Ways to achieve special features, such as high (or wide) dynamic range (HDR/WDR); for example, through multiple sensors in the same package, multiple exposures of the same image frame, etc.

These sensor vendors use interfaces to get these electronic image signals off the sensor and into downstream processing logic.

FPGAs provide a cost-effective, very small programmable logic platform that can easily convert signals from different image sensor interfaces into digital signals for downstream logic processing. FPGAs provide cost-effective programmable mechanisms to accommodate a variety of signal encoding schemes, register management schemes, and sensor interfaces, thereby providing programmable support for different types of sensors.

Image Sensor Technology

Image sensors can be divided into two major categories based on the basic technology used to convert visible light into electrical signals. They are CCD (charge coupled device) sensors and CMOS (complementary metal oxide semiconductor) sensors. By far, the image sensor with the largest shipment volume is the CMOS sensor. This article focuses only on the CMOS sensor interface.

A typical application of an image sensor in a video processing chain is shown in Figure 1.

There are several well-known image sensor manufacturers today, namely Aptina, OmniVision Technologies, Sony, Samsung, Panasonic, Toshiba and Altasens.

As mentioned earlier, sensor manufacturers configure a range of interfaces for passing image signals leaving their chips to downstream logic for processing. It is very common for the same sensor manufacturer to use different interfaces depending on the amount of data that needs to be extracted from the chip. For example, a modern sensor with megapixel resolution needs to pass out much more data in a given cycle time than a sensor with only VGA-level resolution. Requirements like high dynamic range (HDR) also increase the amount of data that needs to be read from the image sensor for each image frame, and the number of frames that need to be extracted from the sensor chip in a given time to support smooth, low-latency high-quality video also affects the choice of sensor interface.

The Evolution of Image Sensor Interfaces

Until now, all sensors have been connected to a parallel LVCMOS interface, as shown in Figure 2. Sensor resolution and frame rates have increased to a level where the previously mainstream CMOS parallel interface can no longer handle the required bandwidth.

The need for higher speeds has surged due to the advent of megapixel sensors, HDR and the need for new, higher speed sensors that support higher frame rates are using different interfaces to overcome the limitations of parallel LVCMOS. For example, Sony and Panasonic use parallel sub-LVDS interfaces, and OmniVision uses MIPI or serial LVDS. As another example, to support the need for higher bandwidth, Aptina Imaging has introduced a high-speed serial interface called HiSPi (High Speed ​​Serial Pixel Interface). The HiSPi interface can operate with 1-4 serial data channels, plus 1 clock channel. Each signal is a sub-LVDS differential signal centered on a common mode voltage of 0.9V. Each channel can run at up to 700Mbps.

The need for a HiSPi to Parallel Sensor Interface bridge

Multiple sensor interfaces present a problem for manufacturers who are standardizing downstream video processing logic, as it is difficult to support many different sensor interfaces with one ASSP.

Most ISP (image signal processing) devices support traditional CMOS parallel sensor interfaces, but generally lack support for high-speed serial interfaces. Many ISP parallel interfaces run much faster than the sensor's parallel interface. However, as sensors have migrated to different serial interfaces, ISP devices require logic to convert to parallel interfaces. Therefore, FPGA bridge devices are needed to convert high-speed serial data to parallel format. For manufacturers of video signal processing ASSPs (who have off-the-shelf products that support faster parallel CMOS sensor interfaces), FPGAs solve the problem of connecting to high-speed serial sensors. FPGAs provide a simple, cost-effective, programmable bridge between high-speed sensors and traditional image signal processing ASSPs. This concept is shown in Figure 3.

FPGA-based Serial Sensor Bridge Reference Design Example

As a practical example, the LatticeXP2-5 non-volatile FPGA provides an efficient and cost-effective solution for bridging the HiSPi serial interface of Aptina Imaging to the parallel interface of TI DSP, as shown in Figure 4.

The reference design uses the HiSPi serial interface on the input side and the TI TMS320DM3X5 on the output side to connect to the Aptina sensor. The evaluation hardware has been tested with Aptina's A-1000 sensors MT9M034/MTM024 and MT9J003. The reference design supports both Packetized and Streaming SP HiSPi formats: 1-4 channels running at up to 700Mbps per channel. It also emulates parallel sensor outputs with output bus widths of 8, 10, 12, 14, or 16 bits. The parallel interface can be configured for 1.8V, 2.5V, or 3.3V LVCMOS levels. The block diagram of the reference design is shown in Figure 5.

Challenges of FPGA in Sensor Interface Bridging

Programmable logic as a bridge between image sensors and ASSPs faces three challenges. First, the FPGA must provide electrical signal support for the interface signals. Second, the FPGA's I/O must have enough gearing logic to support the fast serial sensor interface. Third, the FPGA must provide a very small form factor that is cost-effective to fit the compact form factor requirements of modern cameras.

The LatticeXP2 non-volatile FPGA family with full sub-LVDS file support has been proven to address the electrical requirements of image sensor bridges. Integrated PLLs, dedicated clock edges, and I/O gearing logic address high-speed serial sensor interfaces. Finally, Lattice Semiconductor's XP2 offers a cost-effective 8×8mm area. In addition, due to its non-volatile nature, the LatticeXP2 family of devices does not require an external boot PROM, further saving board space, making them an attractive programmable logic platform for sensor interfaces. The availability of image signal processing (ISP) IP also enables larger LatticeXP2 devices to provide a variety of functions such as sensor data linearization, sensor register programming, de-Bayering, defective pixel correction, gamma correction, and simple HDR up to 24 bits per color channel.