No more confusion through the fog - A brief analysis of Hikvision's fog-penetrating technology

    The demand for video surveillance can be divided into two categories: daytime demand and nighttime demand. The main way to improve the nighttime effect is low illumination and fill light, which has led to the prevalence of various cameras on the market, such as white light fill light, infrared fill light, and laser fill light; for non-fill light cameras, starlight-level super illumination is currently popular. As for the improvement of daytime effects, a key point is the ability to adapt to complex and changing environmental changes, which is what we usually call "all-weather".

    Surveillance cameras often face the challenge of outdoor all-weather surveillance. In addition to strong light, dark nights, rain and snow, haze is a major "natural enemy" of surveillance. Aerosol particles composed of water vapor, smoke and dust are the main body of haze and are also the root cause of image quality degradation. In haze weather, the image color is dim and the contrast is low. The details of some important targets are submerged in the fog and difficult to be detected. Therefore, removing the fog in the video and improving the image quality is an important technology to enhance the application value of outdoor video surveillance.

    Development of fog-penetrating technology

    The demand for fog-penetrating monitoring first appeared in defense applications such as maritime border defense. At the beginning, related products were only researched on fog-penetrating lenses, and an optical fog-penetrating technology was developed, which is what we usually call the first generation of fog-penetrating technology. Optical fog-penetrating uses a lens specially optimized for infrared band imaging, and intercepts specific near-infrared band light through a filter, thereby using the infrared light in the fog for imaging. Optical fog-penetrating has a very strong ability to penetrate fog and has a good imaging effect, but because it is infrared band imaging, it can only present a black and white image, and this fog-penetrating method has a large equipment cost investment, so general users have the demand but will be discouraged.

    In recent years, with the growth of civilian demand, and in order to achieve better fog penetration effect, relevant manufacturers have begun to study the camera video image enhancement technology, which is the digital algorithm fog penetration camera that is now very common in the security monitoring market. We call it the second generation of fog penetration technology. At present, many security manufacturers mainly stay in the first stage of digital fog penetration technology - the "shallow fog penetration" stage. This method adjusts the distribution of information collected by the sensor, enhances the color and details of the observed target, reduces the loss of information in subsequent processing (such as ISP and encoding compression), and improves the user's observation effect. However, this fog penetration effect is not obvious, also known as false fog penetration. "Shallow fog penetration" is also a fog penetration technology commonly used in the market.

    However, there are always some users who are not satisfied with the improvement of contrast, so the digital fog-penetrating camera has entered the second stage - the "algorithm fog-penetrating" stage. Algorithm fog-penetrating can judge the concentration of haze by the gray-white degree of local areas according to the physical formation model of haze, thereby restoring a clear haze-free image. Algorithm fog-penetrating can retain the original color of the image, and at the same time can greatly improve the image fog-penetrating effect on the basis of "shallow fog-penetrating".

Figure 1: First and second generation fog penetration technology

    At present, some manufacturers are striving for excellence, and a third-generation fog-penetrating technology has appeared on the market. In addition to using photosensitive chips with better low-light performance, they even take a different approach by adding special fog-penetrating filters inside the machine. The most important innovation is to add corresponding algorithm fog-penetrating in conjunction with the optical fog-penetrating lens. Some people call it "optical + algorithm fog-penetrating". Optical + algorithm fog-penetrating focuses on integrating fog-penetrating algorithms on the basis of optical fog-penetrating, and further improving the fog-penetrating effect on the basis of the original optical fog-penetrating. This technology depends on the different fog-penetrating algorithm technologies of major manufacturers, and the effect is also good or bad.

Figure 2 Comparison of optical + algorithm fog penetration and optical fog penetration

    From this point of view, when choosing a fog-penetrating device, in addition to the capabilities of the camera hardware itself, it is also necessary to compare the "soft power" of products from various brands, that is, the pros and cons of the fog-penetrating algorithms.

    Colored fog penetration hotspots: fog penetration algorithm

    Digital fog penetration technology can preserve the original color of the image while penetrating fog, and the core of digital fog penetration is the fog penetration algorithm. As mentioned earlier, the fog penetration algorithm technology mainly has two stages and can be divided into two categories: one is a non-model image enhancement method, which achieves the purpose of clarity by improving the contrast and meeting the requirements of subjective vision; the other is a model-based image restoration method, which examines the cause of image degradation, models the degradation process, and uses inverse processing to ultimately solve the image restoration problem.

    At present, typical methods for fog penetration processing through enhancement (shallow fog penetration) include: histogram equalization, filter transformation method and fuzzy logic-based method. Histogram equalization method, in which the global method has a small amount of calculation but does not enhance the details enough; the local equalization method has a better effect, but may introduce block effects, large amount of calculation, noise amplification and the problem of difficult to control the algorithm effect. The filter transformation fog penetration algorithm can obtain relatively good processing results through local processing, but they have huge amount of calculation, high resource consumption, and are not suitable for equipment with high real-time requirements. The known fuzzy logic-based methods have less than ideal fog penetration effects.

    The method based on image enhancement (algorithm fog penetration) can improve the image contrast to a certain extent and improve the recognizability by enhancing the region of interest. However, this method fails to compensate for the cause of the image degradation process, so it can only improve the visual experience but cannot achieve a good fog penetration effect. Therefore, the real digital fog penetration is achieved based on image restoration, which is what we usually call "algorithm fog penetration".

    There are mainly the following types of methods based on image restoration: filtering methods, maximum entropy methods, and image degradation function estimation methods. Filtering methods such as the Kalman filter method are generally computationally intensive. The maximum entropy method can achieve higher resolution, but it is nonlinear, computationally intensive, and difficult to numerically solve. Image degradation function estimation methods are mostly designed based on certain physical models (such as atmospheric scattering models and polarization characteristics of fog penetration models). It is necessary to collect multiple images at different time points as reference images in order to determine multiple parameters in the physical model, and finally solve the result image in a fog-free state.

Figure 3 Differences in fog penetration effects based on image enhancement and image restoration

    The fog penetration algorithm is also the core competitive point of fog penetration products of powerful manufacturers in the current security industry. Taking Hikvision as an example, after fully analyzing the advantages and disadvantages of fog penetration theory and conducting in-depth research and exploration, Hikvision has developed a real-time video fog penetration technology based on the special requirements of video image fog penetration in the security monitoring field. This technology is based on the principle of atmospheric optics, distinguishes the depth of field and fog concentration in different areas of the image for filtering processing, and obtains accurate and natural fog-penetrating images, which is called "SSD algorithm fog penetration".

    In image processing, the following model is generally used to express the foggy image we see:

    represents the intensity of the image seen, is the intensity of the scene light, is the atmospheric light component, and is used to describe the part of the light that is not scattered when passing through the medium. The goal of fog penetration is to recover,,, from it, which corresponds to the result image after fog penetration. Among them, is called the direct attenuation term, which represents the part of the scene light after attenuation in the medium, which is the atmospheric light component caused by forward scattering.

    The process of real-time video fog penetration is shown in Figure 4. The input is dot matrix format video data, and the output is the processed result, which is the same as the input video format. Since the real-time video fog penetration method can effectively restore and enhance the details, it can also have a considerable effect on videos that have undergone a certain degree of lossy compression. The input video data can be a foggy video that has not been processed by lossy compression encoding, or it can be a decoded image of a compressed foggy video. Relatively speaking, the video that has not been processed by lossy encoding can obtain a more ideal processing result after fog penetration because it contains more information.

    Figure 4 Real-time video fog penetration flow chart

    Based on the input foggy video, this method uses the atmospheric imaging model to analyze the features and estimates two key parameters: atmospheric light value and transmittance. It combines computer vision and image processing algorithms and uses histogram statistics, contrast enhancement and filtering to achieve real-time video fog penetration.

    Since it includes both global and local fog concentration estimates, the real-time video fog penetration technology can automatically adjust to various changing scenes and local areas within the scene according to the fog conditions, avoiding the situation where the near view is too dark and the distant view is blurred; at the same time, it takes into account the efficiency and complexity of implementation, ensuring the real-time and engineering feasibility of the entire fog penetration.

    Black and white fog hotspot: breaking the limit

    We mentioned earlier that the third stage of fog penetration development is "optical + algorithm fog penetration". On the basis of optical fog penetration of black and white imaging, fog penetration algorithms of various companies are added to break through the limit. This technology has very high requirements for security manufacturers. When designing products, a large number of experimental demonstrations are required to select the most suitable filter, lens, sensor and specific fog penetration algorithm combination from a variety of options. While ensuring the fog penetration effect, it is necessary to avoid excessive image noise and reduced clarity, so as to obtain a shocking fog penetration effect.

Figure 5 Schematic diagram of the optical + algorithm fog penetration principle

    Take Hikvision as an example. Hikvision independently developed the third-generation "optical + algorithm fog penetration" technology: Super Fog Penetration, the industry's first integrated optical fog penetration, SSD fog penetration algorithm, and adaptive sensing algorithm. Super Fog Penetration can be used in a variety of outdoor occasions where dense fog needs to be prevented, such as important or accident-prone sections such as highways, railways, shipping, and airport runways; key monitoring areas such as power plants and power transmission equipment, and communication base stations; monitoring applications with long observation distances such as rivers, ports, borders, maritime monitoring, and forest fire prevention; primary and secondary school campuses, city squares, tourist attractions, etc. From the perspective of application industries, it includes transportation, public security, aviation, communications, environmental protection, water conservancy, agriculture, forestry, animal husbandry, and fishery, as well as border defense.

    Summarize

    After years of development, fog penetration technology has gone through multiple stages and has formed two different demand directions: color fog penetration and black and white fog penetration. Color fog penetration is based on the "fog penetration algorithm" that is currently being competed for by major security manufacturers; black and white fog penetration is based on and is a typical representative of "optical fog penetration". The fog penetration algorithm can be added to achieve better fog penetration effects, and strive to obtain better fog penetration effects. The most typical approach is "optical + algorithm fog penetration", which still has clear imaging in dense fog. The development of fog penetration technology has also been accompanied by the progress of fog penetration algorithms. It can be said that whoever masters the core fog penetration algorithm of the security manufacturer will master the core competitiveness in "foggy days".

    The fog-penetrating technology helps you clear the fog and get out of the confusion. Do you choose color? Or the more transparent black and white?