Image processing difficulty analysis and processor selection

Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by sampling and digitizing with a digital camera, scanner, or other device. The elements of the array are called pixels, and their values ​​are integers, called grayscale values. The main contents of image processing technology include image compression, enhancement and restoration, and matching, description, and recognition.

With the improvement of people's requirements for quality of life and the trend of global anti-terrorism, as well as the continuous progress of digital technology itself, biometric solutions and video surveillance solutions based on technologies such as fingerprint recognition, iris recognition, and face recognition are gradually becoming important means to improve the security of individuals, families, enterprises, and society. The biometric solution mainly includes four steps: image acquisition, image preprocessing, feature sampling, and matching analysis; while the video surveillance solution mainly includes image acquisition, image preprocessing, image processing and transmission, image display, and image management. It is not difficult to see that image preprocessing is necessary for both biometrics and video surveillance. In fact, the flexibility, complexity, occupancy of image processing chip resources, and processing time of the image preprocessing algorithm will directly have a significant impact on the operation of the entire system. Therefore, image preprocessing is a difficult and critical task for the entire security solution, which directly determines the accuracy and convenience of subsequent image processing and analysis.

Image analysis

Extract some useful metrics, data or information from an image. The purpose is to obtain a numerical result rather than to produce another image. The content of image analysis overlaps with the research fields of pattern recognition and artificial intelligence, but image analysis is different from typical pattern recognition. Image analysis is not limited to classifying specific areas in an image into a fixed number of categories. It mainly provides a description of the analyzed image. To this end, it is necessary to use both pattern recognition technology and a knowledge base about image content, that is, the content about knowledge expression in artificial intelligence. Image analysis requires the use of image segmentation methods to extract the features of the image and then describe the image symbolically. This description can not only answer whether a specific object exists in the image, but also provide a detailed description of the image content.

The various contents of image processing are interrelated. A practical image processing system often combines and applies several image processing techniques to obtain the desired results. Image digitization is the first step to transform an image into a form suitable for computer processing. Image coding technology can be used to transmit and store images. Image enhancement and restoration can be the final purpose of image processing or a preparation for further processing. The image features obtained through image segmentation can be used as the final result or as the basis for the next step of image analysis.

Image matching, description and recognition compare and register images, extract the features and relationships of images through a scoring system, obtain a symbolic description of the image, and then compare it with the model to determine its classification. Image matching attempts to establish a geometric correspondence between two images and measure their similarity or difference. Matching is used for registration between images or between images and maps, such as detecting changes in scenery between images taken at different times and finding the trajectory of moving objects.

Extracting some useful measurements, data or information from an image is called image analysis. The basic steps of image analysis are to divide the image into some non-overlapping regions, each of which is a continuous set of pixels, measure their properties and relationships, and finally compare the obtained image relationship structure with the model describing the classification of scenes to determine its type. The basis of recognition or classification is the similarity of images. A simple similarity can be defined by the distance in the regional feature space. Another similarity measure based on pixel values ​​is the correlation of image functions. The last similarity defined on the relationship structure is called structural similarity.

Segmentation, description and recognition for the purpose of image analysis and understanding will be used in various automated systems, such as character and graphic recognition, product assembly and inspection by robots, automatic military target recognition and tracking, fingerprint recognition, automatic processing of X-rays and blood samples, etc. In such applications, it is often necessary to comprehensively apply technologies such as pattern recognition and computer vision, and image processing appears more as a pre-processing. The rise of multimedia applications has greatly promoted the application of image compression technology. Images, including dynamic images such as videotapes, will be converted into digital images, and stored in computers together with text, sound, and graphics, and displayed on computer screens. Its application will expand to new areas such as education, training, and entertainment.

Image preprocessing analysis

According to different purposes, image preprocessing can be divided into clearing the captured image, preprocessing the image before recognition, and preprocessing the image before compression. Among them, clearing the captured image mainly includes subsequent correction of the inconsistency of the photosensitive unit of the CMOS or CCD image sensor, compensation for the difference between the actual environment and the image captured by the sensor (such as backlight), and denoising the captured original image. Although this preprocessing algorithm itself is not difficult, with the popularization of real-time requirements, especially when the pixels are large, this algorithm still places high demands on the processing capabilities of the DSP.

The preprocessing of images before recognition is very purposeful and may require the destruction of the original pixels and distribution for subsequent feature extraction. The difficulty of this preprocessing algorithm varies depending on the recognition occasion. It is necessary to comprehensively consider the recognition algorithm part later and select the appropriate DSP. The preprocessing before image compression mainly refers to converting YUV422 to YUV420, RGB to YUV, etc. This type of processing often has real-time requirements. If it is implemented in software, it will have higher requirements for processing performance; if it is implemented in hardware, although the processing performance is guaranteed, the hardware cost will increase.

At the same time, according to different applications, image preprocessing can be divided into image preprocessing in biometric applications and image preprocessing in video surveillance applications. For biometric applications, taking fingerprint recognition as an example, its preprocessing mainly includes fingerprint image enhancement, fingerprint image binarization, fingerprint image refinement, and fingerprint image refinement post-processing. Image preprocessing in video surveillance applications mainly refers to analyzing the continuous images output by the image sensor to obtain sufficient information and improve the actual effect of the image through automatic white balance, gamma correction, automatic focus, automatic exposure, backlight compensation, etc.

Challenges of Image Preprocessing

Whether it is biometrics or video surveillance, image preprocessing is facing the following challenges:

⑴ Users have higher and higher requirements for image quality, and the image preprocessing algorithms are becoming more and more complex, which puts more stringent requirements on the processing power and storage space of the image preprocessing main chip.

⑵ Users have increasingly higher requirements for real-time processing and transmission of images. On the one hand, they require image preprocessing algorithms to be optimized and streamlined as much as possible. On the other hand, they also put forward higher requirements on the core processing capabilities, internal bus architecture, data transmission capabilities, peripheral interfaces, and overall hardware architecture and instruction set support for preprocessing algorithms of the image preprocessing main chip.

⑶ Unlike image and video encoding and decoding algorithms, which have unified algorithm standards and clear evolution roadmaps in the industry, image preprocessing algorithms not only have no unified standards and clear development direction, but to a large extent, solution providers use these "secret" personalized algorithms as a magic weapon for market competition. In addition, with the differences in application fields, the improvement of requirements and the evolution of technology itself, the original algorithms will be continuously upgraded and new algorithms will continue to emerge, all of which require image preprocessing chips to have higher flexibility and adaptability.

⑷ For solution providers, not only do their core algorithms that reflect their competitiveness need to be protected from illegal reading or copying, but also their image data, whether it is biometrics or video surveillance, often involves privacy, so they also need to provide trustworthy security guarantees. Both of the above aspects require that the image processing chip must provide a reliable and complete processing platform.

In addition to the above challenges, from the perspective of system design, there are also the following requirements:

⑴ Although image preprocessing and image processing are huge tasks, engineers do not want to use multiple chips to handle this. Because the traditional DSP architecture in which signal processing and control systems run on different processors has already given engineers a headache, if image preprocessing and image processing are separated, it will be even more difficult for engineers to develop, debug and maintain the system. Therefore, for the main chip DSP in system design, there is also a requirement for integration - is it possible to implement image preprocessing, image processing, and even system control functions on a single chip?

⑵ With the increasing complexity of the entire image processing algorithm, including pre-processing, the DSP, as the main processor, not only needs to provide sufficient hardware processing capabilities, but also should provide an instruction set specifically optimized for the processor in software, so as to help engineers reduce the difficulty of familiarizing themselves with the physical architecture of the processor, maximize the control and utilization of the processor's characteristics, and develop a streamlined and optimized image processing algorithm as soon as possible.

⑶ In addition to the specially optimized instruction sets mentioned above, faced with increasingly complex image processing and product time-to-market pressures, engineers also expect processor vendors to share some of their work - for example, providing low-level image processing software modules that are optimized specifically for the processor and take up only a very small number of clock cycles, to help them shorten the image processing algorithm development process and accelerate software porting.

In addition, powerful, user-friendly, and easy-to-learn development tools are also key requirements for engineers in system development. Moreover, with the increase in system complexity and module reusability requirements, higher requirements are also placed on the compatibility of development tools.

Hardware Features

The hardware architecture of the Blackfin series processors is optimized for image processing. Multiple DMA channels and flexibly configurable caches can effectively address the requirements of image processing applications with large computational load and high data throughput. In image processing applications, although the transmission of image data can also be implemented by software, this will consume a large number of CPU clock cycles, making it difficult for the DSP to fully utilize its high-speed data processing capabilities. If DMA is responsible for data transmission independently, then after the system kernel initially sets up and starts DMA, the DMA controller can directly transfer image data from the PPI interface to the SDRAM memory for storage without the involvement of the kernel. In addition, two-dimensional DMA can also simplify the transfer of macroblocks to and from external memory, allowing data control to become part of the actual data transmission, which is very convenient and important for the crossover and decrossover of color space elements.

For image processing applications, the Blackfin series DSPs are constantly strengthening support for hardware functional modules. For example, the latest version of the ADSP-BF54x series Blackfin processors has added a hardware accelerator for processing overlaid images and an extended video interface, which enables tasks such as color space transformation, scaling, and image overlay to be completed without the processor being involved in the calculation, thereby reducing the processing pressure on the core and providing more space for higher-performance, higher-speed image processing.

Software Features

In terms of instruction set, the Blackfin series DSP provides a wealth of vector instructions and video instructions for image processing. Among them, vector instructions can implement operations on 16-bit numbers (most instructions can complete two 16-bit operations in parallel). Since most image processing operations are for 16-bit operations, it is very important to optimize image operations by using these vector instructions properly. Not only do most arithmetic instructions and shift instructions in the Blackfin instruction set have corresponding vector instructions, but also there are special instructions such as addition based on signs and conversion of 32-bit numbers to 16-bit numbers in the vector instructions. The reasonable application of these vector instructions in the assembly optimization of image preprocessing can improve the parallelism of the algorithm and greatly speed up the operation.

Video pixel instructions mainly include BYTEOP16P, BYTEOP3P, BYTEOPIP, SAA, BYTEAPCK, and BYTEUNAPCK. A video pixel operation instruction can complete 11 video pixel operations such as addition, subtraction, addition and subtraction mixing, averaging, or subtraction and absolute value of 4 pairs of video data components in one cycle. Since video pixel values ​​are generally stored in 8 bits, the use of video pixel instructions can greatly improve the speed of various video image operations including SAD, pixel interpolation, and direct conversion between 8-bit and 16-bit numbers.

Safety Features

In terms of security, ADI's Blackfin Lockbox Secure Technology combines software and hardware protection by providing one-time programmable (OTP) memory and secure processing mode (Blackfin secure mode), providing developers with the means to implement the above security measures. Among them, developers can use the public, non-secure, user-programmable area of ​​the OTP memory to store public keys, so that the system can be identified in a controllable and configurable manner. In the private, secure, user-programmable area of ​​the OTP memory, developers can set private device assets such as private keys and maintain the confidentiality and integrity of these device assets.

Software module library support

ADI provides a variety of software modules for image processing, including H.264 Baseline Profile Decoder module, enhanced video post-processing (eVPP) module that can scale images with different input and output sizes, JPEG encoder module, MPEG-2 Decoder Simple & Main Profile Decoder library, MPEG-4 Simple Profile & Advanced Simple Profile Decoder library, and MPEG-4 Simple Profile & Advanced Simple Profile Video Encoder module, all of which are optimized for Blackfin processors and have been strictly verified by the industry.

ADI has also launched the "Image Tool Box" software package specifically for image processing applications. The software package consists of a series of dedicated modules and is specially optimized for some common and basic functions of image processing algorithms. It can perform image processing operations such as image transformation, image analysis and image enhancement, binary image operations, and morphological processing. This software package helps to reduce the development difficulty of engineers and accelerate the implementation and optimization of upper-level algorithms.

Development environment support

The VisualDSP++ development environment used for Blackfin series DSP processor development applications and project management mainly includes the integrated compilation and debugging environment (DIDE) with the ViusalDSP++ kernel; CC/++ optimization compiler with real-time runtime library; assembler and linker, as well as simulation software and program routines. Among them, the compiler allows program developers to write signal processing and control codes in C or C++ language, which facilitates system development and maintenance. The graphical user-friendly information exchange interface enables engineers to manage projects, edit, compile and debug programs in the window, and switch between them quickly and easily. In addition, the VisualDSP++ development tool is also compatible with Green Hills Software's MULTI environment, NI's LabVIEW software, and MathWorks' MATLAB and Simulink software, providing a more convenient and relaxed environment for system development and module reuse.