FPGA-based video surveillance/security system implementation solution

The video surveillance and security industry is undergoing a dramatic transformation, moving from traditional analog CCTV cameras to logic-based digital cameras. The trend toward higher video resolution, image signal processing, advanced video analytics, multi-camera systems, and digital video compression is driving this shift. As a result, new challenges have emerged in video surveillance camera and digital video recorder (DVR) designs, including the increased use of CMOS sensors, network IP camera technology, increased system integration, smaller form factors, the use of advanced codecs, and increased DSP functionality.

Due to the diverse and competitive nature of the video surveillance/security market, it is often difficult for manufacturers to add differentiated features to their own products that distinguish them from those of competitors. As a result, similar camera and DVR products are very common. However, the trends mentioned above have created new opportunities for video security/surveillance OEMs to achieve product differentiation.

Meeting the requirements of these applications requires additional processing power. Current digital cameras utilize DSP chips, specialized ASSPs, or ASICs to provide processing capabilities, but these solutions present some challenges. Performance bottlenecks often occur with DSP chips because these chips typically handle image signal processing tasks in a serial manner. ASSPs may offer more performance, but often at the expense of design flexibility. ASICs have more performance optimizations, but the production volumes are not large enough to justify the expense and time required for ASIC development. Designers need a flexible approach to provide the computing power required for different market segments, from low-end cameras/DVRs to high-end applications.

Implementation options

Video surveillance systems can be divided into two categories: camera systems and DVR systems. Camera systems can integrate a single camera or multiple cameras. Some systems integrate DVR with a single camera or multiple cameras.

Figure 1: Block diagram of a multi-camera DVR video surveillance system.

Figure 1 is a general multi-camera DVR video surveillance system. The video source comes from a CMOS video sensor, a CCD video sensor, or an analog video source. A proprietary video interface converts the input video stream into a common format, then multiplexes the different video streams (and audio) and pre-processes them in the image signal processing unit. The purpose of video pre-processing is to reduce noise and eliminate pixel defects.

Video analytics is used to detect motion in predefined frames. This motion detection output reduces the required storage capacity. Video analytics is required in some video surveillance applications, such as people/car counting, license plate number recognition of cars and/or face recognition.

A typical video surveillance system with multiple camera video sources always generates a lot of data, so it is very important to reduce the required storage capacity. MPEG-4, H.264, and MJPEG are used to reduce the required storage capacity. H.264 is a popular compression algorithm for video surveillance applications because it can provide good video quality at a fairly low bit rate (half or less than MPEG-2 or MPEG-4 Part 2).

The compressed data is stored in a video storage server via a hard drive interface or sent over an Ethernet network. Video data is usually taken from a hard drive and decoded before being sent to the display, as well as some image post-processing such as scaling, color space conversion, or overlay applications. Typically, a memory interface (such as DDR2) is used to store video frames. In addition, some systems require real-time encryption of video content for security and source/user authentication. Finally, a processor is needed to control and coordinate/schedule the various tasks.

Figure 2: Steps in making selections based on throughput, flexibility, batch size, and form factor requirements.

Figure 2 illustrates how system requirements such as throughput, flexibility, expected volume, and form factor drive different implementation technologies and devices. For low data throughput requirements, DSP processors are the most cost-effective devices. However, expected volume during the product life cycle and hardware flexibility can greatly influence the designer's choice between ASIC/ASSP and FPGA devices. For system requirements with low hardware flexibility and expected large volume, designers favor ASIC/ASSP solutions; while for system requirements with high hardware flexibility and smaller expected volume, designers favor FPGA solutions. For the small size requirements of common small video surveillance cameras, designers favor non-volatile FPGAs because they do not require additional external non-volatile configuration storage devices. [page]

System integration combines the advantages of different devices: DSP processors sometimes integrate ASSP IP modules, ASSP/ASIC sometimes integrates processors, and FPGA sometimes integrates hard processors and hard IP cores.

For video surveillance systems (Figure 1), video analytics functions and processor blocks can be effectively implemented using DSP processors due to the sequential nature of these tasks. However, in some cases, the acceleration capabilities of FPGAs can help perform some video analytics functions such as motion detection, facial recognition, etc.

The H.264 encoder shown in FIG. 1 is a large and complex functional block requiring high throughput, low hardware flexibility, and is usually implemented using an ASSP or ASIC to obtain a small-size solution.

However, ASICs and ASSPs bring high non-recurring engineering costs unless the batch is very large. If only a small number of users use special functional modules, then FPGA solutions can provide better returns. For ASICs and ASSPs, the limited functions and inherent inflexibility of these devices do not allow improvements or additions of new functions according to market requirements after the design is completed. FPGAs can provide a wide range of functions and high flexibility.

Video surveillance system modules with the following requirements will benefit from the flexibility of FPGAs: (1) The video and audio interfaces and multiplexers require hardware flexibility to support different cameras and different numbers of cameras. (2) The DDR interface must support different memory bus widths and different DDR standards. (3) Support a variety of image signal processing algorithms (2D FIR filter, 2D median filter, scaling, edge detection, gamma correction, alpha blending, white balance, lens shading correction, defective pixel correction, demosaicing, line scanning, color space correction, etc.), and a subset of various algorithms may be required in a specific video surveillance camera implementation. The effective implementation of these algorithm structures depends on the throughput requirements, which in turn depends on the number of cameras in the system and the video standard sampling rate. (4) Hard drive interfaces and Ethernet interfaces are not always required in video surveillance camera systems, but are available as options. (5) Display interfaces are not always required, and when they are required, there are many different display interface implementations to choose from. (6) Encryption and authentication are only required in certain systems. Depending on the different throughput requirements of different video formats and the different number of cameras in a specific surveillance system, the optimal encryption and authentication system may require different size-optimized implementation architectures.

Many network IP cameras are constrained by very compact form factors, power consumption, and cost, especially in embedded camera/DVR systems. Using non-volatile FPGAs as a solution can address these issues while still gaining the traditional advantages of FPGAs.

Product Differentiation Using FPGAs

There are many advantages to implementing video surveillance systems using FPGAs, including the ability to accelerate DSP processing. Non-volatile FPGAs can make FPGAs more attractive in video surveillance camera system implementations, especially for embedded camera/DVR systems used in mobile/transportation systems. These systems have strict requirements on portability, power consumption, and board area.

These applications also have high requirements for flexibility and scalability, but the premise is that the requirements for power consumption, circuit board area and portability cannot be compromised. For example, some network IP camera systems have multiple cameras and sensors, and even if the circuit board area is very limited, it must provide channels for multiple video streams. Other network IP cameras often use very small housings with circuit boards close to the size of a MicroSD card. The challenge faced by OEMs of such systems is to implement multiple systems with as few chips as possible while meeting the requirements of function, form factor and low power consumption.

Non-volatile FPGA solutions, such as the LatticeXP2 device family, combine the performance and flexibility of SRAM-based FPGAs with the inherent advantages of non-volatile Flash technology. The LatticeXP2 family has many features for security and surveillance applications, including support for DDR/DDR2, 7:1 LVDS interfaces for low-cost displays, and 18×18-bit multiplication DSP modules, which are particularly useful for image signal processing functions often required in video surveillance applications. When the LatticeXP2-5 and LatticeXP2-8 devices are packaged together with low-cost chip-scale packages, the LatticeXP2 non-volatile FPGA device family provides a lot of value to OEMs, especially when designing compact network IP cameras.

Figure 3: The LatticeXP2 nonvolatile device family is available in low-cost packages.

Conclusion

For a large variety of video surveillance camera and DVR applications, FPGAs offer the advantages of hardware flexibility, parallel processing capabilities, and zero non-recurring engineering costs. Using non-volatile FPGAs eliminates the need for an external boot chip PROM, allowing a single-chip solution to be built that provides instant startup capabilities. Features with on-chip key storage and built-in AES 128-bit encryption technology provide security for protecting the FPGA programming bit stream and preventing external tampering. Many security camera system solutions, including embedded camera/DVR systems, can benefit from traditional SRAM-based FPGAs, but when power consumption, board area, cost, and integration characteristics are important, non-volatile FPGAs are a more attractive alternative.