Design strategy for intelligent video surveillance application system

With the improvement of broadband wired and wireless network infrastructure and the growth of global security market demand, the application of video surveillance is showing explosive growth. The development trend of video surveillance systems is very obvious. After digitalization and networking, the next important trend is intelligence, that is, the application of intelligent monitoring and video analysis technology.

Traditional video surveillance is performed manually to detect safety hazards or abnormal conditions, or for post-analysis. This application has its inherent shortcomings and is difficult to achieve real-time safety monitoring and detection management. Monitoring systems with intelligent analysis functions can actively collect and analyze data by distinguishing the appearance, movement and other characteristics of the monitored object, and perform actions such as alarm, recording and analysis according to preset conditions. Intelligent monitoring systems can run on servers or on DSP-based embedded systems, and the latter has gradually become mainstream.

The application of intelligent video can be roughly divided into three aspects: security, human behavior detection and intelligent transportation. Among them, security application is widely regarded as the market with the most potential. It includes the following application categories: intrusion detection, which can automatically detect the motion behavior characteristics in the video screen; object removal detection, which can automatically detect the object removal event - when the object in a specific position in the defense zone is taken or moved away, an alarm is issued; abandoned object detection, which can automatically detect abandoned objects - when the object is placed or abandoned in a certain defense zone, an alarm is automatically issued; intelligent tracking, which can enable the camera to autonomously drive its own pan/tilt and zoom lens. Human behavior detection applications include off-post detection (which can realize automatic detection of the position of sentry personnel) and wandering detection (detection of human wandering in important areas). Intelligent transportation applications include: detection of illegally parked vehicles, and alarm is issued when the vehicle illegally stays in the defense zone; vehicle reverse detection, timely identification of reverse vehicles.

As accuracy and reliability gradually improve and product costs decrease, smart video is being used in more and more occasions. It can replace some security equipment, reduce the workload of security personnel, improve work efficiency, and reduce management costs. In fact, the application of smart video has huge potential. As the technology matures, the application areas of smart video technology are rapidly expanding. These applications mainly include the above-mentioned security, transportation, retail, service and other industries, such as people counting, face recognition, crowd control, attention control and traffic flow control.

The demand for real-time video monitoring is growing rapidly, especially with the increasing demand for real-time security monitoring applications. The function of real-time detection of security risks or abnormal target behavior has become increasingly important in reality. Driven by this growing demand, intelligent video monitoring system products are becoming a new hot spot in video monitoring applications. In particular, with the advancement of semiconductor technology, such as the introduction of advanced embedded solution platforms represented by Blackfin convergent processors, intelligent video analysis devices with extremely high cost performance and high practicality are constantly being introduced, and play an extremely important role in some key applications.

Figure 1: Schematic diagram of traditional video monitoring applications.

Design strategy for intelligent video applications
The choice of hardware platform solutions often determines the overall solution cost, performance, availability of development tools and methods, and feasibility of future continuous upgrades of the solution. Therefore, the selection of solution platform is crucial. The uniqueness of intelligent video applications requires a comprehensive trade-off in the selection of hardware platforms. The networking and intelligent analysis requirements of video surveillance systems, as well as the restrictions on cost, volume and power consumption of large-scale engineering installations, non-standardized intelligent video analysis methods and almost customized solution optimization methods have made the convergent processor solution platform that combines the advantages of MCU and DSP, with software design flexibility and powerful processing capabilities, more obvious. This article will analyze the main design technology points in intelligent video design based on the characteristics of ADI's unique Blackfin convergent DSP processor.

1. Hardware platform selection
Customization is very important. There are many factors that restrict the pace of intelligent application of video surveillance systems: first, the video algorithm of intelligent surveillance is relatively complex and difficult to standardize. The video analysis software of each system provider has its own unique algorithm, resulting in no unified standard for products on the market; second, the application scenarios of video surveillance systems are relatively complex, and the requirements of users are diverse, so there are many customization requirements. Therefore, video analysis solutions usually need to be optimized according to the application characteristics and needs of customers, and the algorithms used vary greatly. In addition, due to the high complexity of intelligent video applications, higher requirements are placed on the processing capabilities of the solutions. The traditional video surveillance solution of MCU+ASIC is difficult to achieve various personalized designs and high computing power requirements. Even if the dual-chip solution of ordinary DSP+MCU is selected, it is usually difficult to meet the complex computing requirements of intelligent video surveillance applications. A coprocessor needs to be added. This complex solution is not worth it in terms of BOM cost, power consumption or development difficulty. The Blackfin processor fully utilizes the advantages of the MCU+DSP converged architecture to meet the system control and high-intensity computing requirements of intelligent video applications. In particular, the high-performance dual-core architecture represented by BF561 has become the preferred solution platform for intelligent video applications.

The scalability of the solution is also a factor that needs to be considered. In addition to optimizing the solution for the application environment and application purpose, different customers may have different needs in other aspects. For example, some leading digital video surveillance solutions currently support H.264 basic class @ Level 3.0 and MPEG4 D1 + CIF dual streams, and may be expanded to support H.264 D1 + CIF dual streams in the future. With the wider application of intelligent video analysis, such as IP cameras, wireless video surveillance, intelligent transportation systems, etc., different applications may have greatly differentiated requirements for various interface functions, communication standards, user interfaces, etc., and the flexible scalability of hardware platform solutions for various requirements is very important. At the same time, as mentioned above, the development of intelligent video analysis technology has only been a few years. With the continuous maturity of technology and the introduction and improvement of some related standards, the upgradeability of the product is crucial. It is not only an issue that developers must pay attention to, but also an important feature that end customers are concerned about. Blackfin DSP has unique advantages in algorithm parallel processing, especially ADSP-BF561 uses dual DSP cores, which can implement very complex intelligent video processing algorithms.

Video application optimization features. Although some solutions have strong processing capabilities and scalability, it is also worth paying attention to whether the solutions have been optimized for video applications, because this is directly related to the software and hardware design resources available to design engineers, as well as the difficulty of system design and achievable performance. Take the Blackfin processor as an example. Blackfin is specially optimized for high-intensity, high-data-rate digital and media processing: Blackfin's dozens of DMA channels and flexibly configurable caches well meet the requirements of video surveillance systems for large computing volume and high data throughput; ADI has specially developed a fully optimized audio and video codec and provides it to major customers free of charge; Blackfin integrates many hardware drivers for video applications, including WiFi drivers and audio/video codec drivers; Blackfin's four video arithmetic operation units and video pixel instruction sets greatly accelerate video computing speed; in some basic operators of intelligent video analysis, such as histogram statistics, median operations, Sobel operations, and dilation operations in morphology, Blackfin's MIN and MAX instructions can be used to eliminate conditional jumps and save processor cycles. In addition, Blackfin also supports 13 types of vector operations for non-video data. By properly designing the data structure, Blackfin's special instructions can be used in multiple links such as foreground and background separation, threshold calculation and update to make the intelligent video analysis algorithm faster. Most of these instructions, which are already very effective, can be executed in parallel, doubling Blackfin's processing power.

Low power consumption and stability are important. Considering that intelligent video surveillance equipment usually runs 24 hours a day, 7 days a week, stability and power consumption are also important. In terms of low power consumption, the Blackfin processor uses a variety of energy-saving technologies: based on a gated clock core design, the power supply of functional units can be selectively cut off according to instructions; it supports a variety of power-off modes for periods when the required CPU action is minimal; the Blackfin processor supports a self-contained dynamic power management circuit, with the help of which the operating frequency and voltage can be independently controlled to meet the performance requirements of the algorithm being executed; most Blackfin processors provide on-chip core voltage regulator circuits and can operate at voltages as low as 0.8V. Blackfin's unique convergent processing architecture and 90nm process lay the foundation for its leading low-power processing. Due to its high processing power, the system solution based on the Blackfin platform can reduce the number of main chips, and the rich functions and interfaces can meet various peripheral and function expansion requirements, reduce the number of components, and thus ensure higher stability and reliability. Currently, the low power consumption characteristics and stability of the Blackfin DSP are the best among DSPs of the same price.

Which embedded operating systems are supported. Intelligent video analysis is usually a network-based application that must be supported by an operating system. Therefore, it is very important to choose a solution that has a wide range of embedded system support capabilities. This will ensure that future products do not have to replace the hardware platform when the operating system is replaced, and ensure the continued availability of research and development results. There are many embedded operating systems available, each with its own advantages. The ability of the hardware platform solution to support these operating systems is one of the key points to consider when selecting a solution. For example, the Blackfin processor can support the current mainstream operating systems, including more than ten embedded operating systems such as uCLinux, ThreadX, Nucleus, and uCOS-II. Customers can completely choose a familiar or more cost-effective software architecture based on their own requirements.

Figure 2: Block diagram of an intelligent monitoring terminal based on BF561.

2. Development Tools and Available Resources
Intelligent video surveillance equipment is a complex system, involving complex software and hardware design, human-machine interface, communication connection, etc., and has a high degree of system design difficulty. Therefore, whether the selected hardware platform solution can provide a complete development tool kit, necessary software modules, mature reference design, system design support, and whether there is a complete design ecosystem, etc., is very critical to whether the system design can be completed on schedule and with high quality. In fact, not all platform solution providers can provide this support.

Taking the Blackfin series processors as an example, the hardware platforms using Blackfin processors have been widely adopted by a large number of equipment companies around the world, from general DVRs, IP cameras, digital video surveillance to intelligent video surveillance. Blackfin processors have been favored by many companies, and one of the important reasons for their wide popularity is that they have complete development tools and reference design support. ADI provides industry-leading tools, beginner kits and support, including the well-known ADI CROSSCORE® software and hardware tools that can support other Blackfin processors. These tools include the award-winning VisualDSP++® integrated development and debugging environment (IDDE), emulators, and EZ-KIT Lite® evaluation hardware.

In order to improve development efficiency and reduce development difficulty, development should be carried out on existing resources as much as possible, such as open routines. ADI provides a very rich set of routines and materials for this purpose. For example, ADI provides a free "Image Tool Box" image processing function library software package, which is specially optimized for mathematical functions commonly used in image processing applications for customers to call when developing applications. ADI also provides complete reference designs, as well as evaluation boards, development tools, algorithm IP, application modules developed by local partners, and a full set of turnkey solutions including software and hardware provided by third-party partners. At present, there are many third-party partners with rich engineering experience in China for video surveillance applications of Blackfin processors, and a complete ecosystem has been established.

Take ADI's announcement in March this year that it would provide a complete IP surveillance and machine vision camera reference design based on the company's Blackfin BF526C as an example. This reference design provides powerful video and audio processing capabilities on a single converged processor, providing engineers with a unified software development environment for faster system debugging and deployment, as well as lower system costs. The processor provides integrated audio codecs, streaming video and IP protocols, on-chip DRAM memory, and interfaces for 10/100 Ethernet, USB and SD storage, and local RS-232 ports. This fully programmable solution can meet a variety of video compression standards, such as H.264 and MPEG4, and supports audio G.729 standard encoding. It supports two-way voice communication from the control center to the camera, as well as lens panning, tilting, and stretching actions using the Pelo-P or Pelo-D protocol. The reference design also provides a daughter card with a dual-core BF561 processor, enabling the system to achieve higher video resolutions and provide advanced video analysis functions such as motion detection and tracking.

Application Solution Revealed - Emvideo Smart Network Camera
Beijing Emvideo Technology Co., Ltd. (Emvideo) is a professional smart security product solution provider and a third-party partner authorized by ADI. Emvideo currently has a number of products based on the ADI Blackfin processor as the core hardware platform. Among them, the "software + hardware" turnkey WiFi wireless video surveillance overall solution is based on the dual processor architecture of BF536 + BF561. The hardware structure diagram of the solution is shown in Figure 3.

The BF536 processor is the main processor. In addition to being responsible for some basic management and control such as audio encoding, remote control and user interaction control, it is also responsible for the operation of the embedded operating system uClinux and advanced intelligent video analysis functions. It can complete a variety of intelligent video analysis such as security, human behavior, and intelligent transportation. The dual-core BF561 is responsible for the video encoding algorithm as a coprocessor. Its powerful video processing capabilities enable the solution to support H.264 basic class @Level3.0 and MPEG4 D1+CIF dual streams. In the future, it may be expanded to support H.264 D1+CIF dual streams. The two processors can communicate through a high-speed synchronous serial interface. The video signal first enters the BF561 processor, and the collected and encoded code stream is sent to the BF536 processor, and then sent to the client through the network for decoding and display.

Figure 3: Hardware block diagram of the solution using Blackfin BF536 and BF561.

The solution uses advanced background modeling methods to effectively overcome the interference of backgrounds such as light changes, leaf swings, and water ripples on foreground target analysis, achieving accurate foreground detection. At the same time, it uses unique optimization algorithms in target tracking, achieving an accuracy rate of over 90% in intrusion detection (including regional alerts and tripwire detection). All of this is achieved based on the powerful processing capabilities of the hardware architecture of the BF536 + BF561 dual processor.

The intelligent video analysis function of this solution is independently developed by Yiwei Dongfang Company. The unique algorithm and rich experience in the development of intelligent video analysis technology ensure the realization of customers' intelligent recognition application needs and provide customers with complete services including software upgrades. Since there are currently no demand standards and test standards for intelligent video recognition applications, video analysis solutions usually need to be optimized according to the application characteristics and needs of customers. For example, some users are in the subway system, and they need functions such as detecting whether they cross the yellow line of the waiting area, whether the crowd density is too large, and whether there are suspicious objects left behind; some users are in the bank system, and they need intelligent monitoring of ATM machines, such as analyzing whether fake keyboards and card swallowers are installed, whether there are violent behaviors at ATM machines, and whether the faces of criminals appear. With this solution, customers can easily adjust the algorithm according to their needs. Intelligent video processing requires the chip to have strong processing capabilities. Many algorithms must be implemented in parallel processing. Blackfin DSP has unique advantages in algorithm parallel processing, especially the dual DSP cores of ADSP-BF561 can implement very complex intelligent video processing algorithms. This is difficult to achieve with traditional MCU+ASIC or general DSP solutions.

The solution's software and hardware have been verified by applications and are currently being produced by a number of customers. The target applications will mainly be users in industries such as government administrative efficiency monitoring and education systems.