FPGA implementation of depth perception for stereo cameras

For depth perception of stereo cameras, FPGA solutions can relieve processor time and reduce or eliminate the cost of components such as MPUs, DSPs, lasers, and expensive lenses. By providing the robot with differential mapping of its environment, FPGAs allow the CPU in the robot to focus on important high-level tasks such as mapping and positioning.

Difference Mapping

A common technique for adding depth perception to a robot is to use two independent cameras placed horizontally, parallel to each other and spaced apart. The two cameras are compared using a difference mapping algorithm, see Figure 1.

Figure 1 Difference mapping calculation

In simple terms, disparity is the difference between the right and left images. The closer the object is to the camera, the greater the difference between the two images.

You can do an experiment yourself by holding a pen horizontally upwards against your face and blinking your left and right eyes repeatedly. The closer the moving pen is to your face, the more movement you perceive when you blink. The more movement you make, the closer the object is to your eyes, and the greater the difference in the image.

As shown in Figure 1, the difference between the position of the object in the two images is calculated: d = x1-x2. In order to calculate the actual coordinates of point M, the calculation can be performed using the left and right cameras:

X=Bx1/d

Y=By1/d

Z=fB/d

Related questions

Computing the difference map relies on correlating the images, and matching properties between the left and right images. Computing the correlation pixel by pixel is computationally expensive, so other algorithms are used to simplify the problem. One method, called image point correlation, looks through blocks of pixels, called points, in the left image, and then looks for the same points in the right image. Other methods include edge detection and matching. Once the corresponding images are located, the difference can be calculated.

Distortion and exposure issues

If you use a high-quality camera, you can ignore lens distortion. To save money, use a cheap CMOS or CCD camera, or a low-cost wide-angle lens. These lower-quality products will introduce distortion or exposure issues.

Lenses introduce scale and positioning distortions, such as a white spot effect where objects near the outside of the image appear curved, which can be compensated for using well-known image processing algorithms. One way to correct for white spots, for motion or other geometric algorithms, is to remeasure the pixels in the image using a lookup table of coordinates.

In both very bright and very dark environments, additional algorithms are needed to reliably capture detailed features in the image. For example, determining the best exposure settings for different regions, and using composite images of regions can compensate for the lack of background brightness.

Solution using FPGA with embedded DSP

The depth perception algorithms described in this article are computationally complex to implement at real-time video rates. System designers must evaluate CPU/software, ASIC, ASSP (e.g., DSP processor), and FPGA solutions to determine which is best, as well as FPGA price.

Figure 2 shows the proposed FPGA co-processor solution that handles distortion, exposure, communication, and disparity mapping algorithms for depth perception. This approach frees up the CPU, allowing the DSP processor to handle more serial tasks.

Figure 2 FPGA supports computationally intensive difference mapping algorithms

[page]

When selecting an FPGA, you usually need to evaluate three circuit requirements:

● Aiming at co-processing functions that utilize parallel characteristics, DSP functions and performance.

DDR (Double Data Rate) and LVDS (Low Voltage Differential Signaling) support interface to off-chip SDRAM buffer memory, directly connected to image data from the camera.

●Security solutions to protect intellectual property.

For image processing algorithms, traditional DSP processors have limited parallel capabilities, with only 1 to 4 multipliers per DSP chip, as shown on the left in Figure 3. Therefore, DSPs use very high clock rates to compensate, with clock rates as high as 1 GHz or higher to achieve high throughput. DSP algorithms that require complex serial tasks to be completed quickly should be implemented using DSP processors.

However, many depth-sensing image processing functions require a lot of parallel processing, such as linear interpolation techniques, media filters, and geometry estimation. Compared with DSP chips, FPGAs can perform serial functions in parallel to accelerate system performance, as shown on the right side of Figure 3. For example, image correspondence is a very simple algorithm that compares a large number of pixel values, or pixel points between two images. In order to obtain an effective sum of squared differences (SSD), the image correspondence algorithm must perform millions of calculations on the pixels in the image, which is a challenging task for DSP chips that perform functions serially.

FPGAs with DSP functions contain several multiplier blocks, but some FPGAs also have embedded adders, subtractors, and accumulators that greatly increase processing performance. Although low-cost FPGAs operate at system clock frequencies below 300MHz, high DSP throughput (3,000 MMACs) can be achieved by implementing multiple DSP functions in parallel using multiple DSP blocks on-chip.

Memory and DDR

With multiple multipliers, the flexibility of FPGA parallelism is beneficial for memory access. Various memories can be easily accessed in parallel. Distributed memory on chip can be used to build small high-performance scratch pads. Lookup tables using coordinates for removing white spots are useful for remapping pixels in an image. Large embedded memory blocks on chip can build large high-performance memories for FIFOs and line buffers.

The off-chip memory DDR II SDRAM can provide large high-performance memory. For DDR SDRAM, which is commonly used for image processing and stores the entire frame, FPGA can provide memory control and interface for DDR SDRAM.

At lower clock speeds (below 100MHz), the DDR memory controller interface is easy to do and can be implemented using general-purpose I/O and logic in the FPGA. However, at higher frequencies, FPGAs with specialized circuits are required to implement a robust DDR memory interface. Not all FPGA devices have these specialized circuits, and the cost and complexity of implementing a high-speed DDR memory interface varies greatly, depending on the specific FPGA.

LVDS

Depth sensing for cameras and video chipsets via LVDS interface, such as camera link, FPGA interface image processing engine. LVDS is a low noise, low power, small swing differential signal used to transmit high speed (gigabits per second) data over copper wire.

As an example for a video interface is a 7:1 LVDS interface, with multiple LVDS signal pairs for data transmission and one signal pair for clock. For some FPGAs, this is the native I/O interface. The video data enters the FPGA through three LVDS pairs, and the I/O structure deserializes the packet and then sends the parallel data in the FPGA to the logic array.

Security

Because autonomous robots are sometimes used in military applications, the FPGA should not introduce additional vulnerabilities to the system during configuration. For SRAM-based FPGAs, the configuration data comes from external nonvolatile memory. Some FPGAs have a built-in 128-bit AES decryption engine to prevent hackers from gaining access to the functions in the FPGA through reverse engineering.

AES configuration bitstream encryption provides a measure of protection as long as the key is unknown. However, non-volatile FPGAs completely remove this security risk. Some non-volatile FPGAs combine Flash and SRAM on the same chip. After power-up, the SRAM obtains the working configuration. This technology provides high configuration security while taking advantage of the unlimited reconfiguration of SRAM.

Conclusion

FPGAs that support DSP, differential signal LVDS, and DDR II memory interfaces make real-time perception of robots possible.

For robotic stereo vision applications, a combination of CPU, DSP processor, and FPGA coprocessor can provide the best balance of performance and price for real-time image processing implementation. FPGA can handle highly parallel disparity mapping and depth perception algorithms. DSP performs more complex functions with less parallel processing, high speed, and simple code. CPU does those parts that require flexibility and high-level software implementation.

Based on Flash, non-volatile LatticeXP2 FPGA supports many requirements, including LVDS, built-in DDRII memory and full-featured DSP blocks, as well as a large number of programmable logic and memory blocks to support interfaces and other custom functions. Many DSP intellectual property cores and MATLAB SIMULINK make the design more convenient. Lattice also has reference designs and platforms based on 7:1 LVDS video interfaces. Some of the functions provided by the free reference design IP are color depth conversion, brightness, contrast and video overlay.