Research and application of starlight-level low-light camera

　　The current security market has higher and higher requirements for the low-light performance of cameras. Not only does it need to achieve 24-hour all-weather monitoring, but the application environment is also becoming more and more complex. Currently, the main imaging application technologies in low-light environments include infrared thermal imaging, active infrared, frame accumulation, etc.

　　1. Infrared thermal imaging

　　The principle of infrared thermal imaging is not mysterious. From the perspective of physics, the human body is a natural biological infrared radiation source that can continuously emit and absorb infrared radiation. Infrared thermal imaging technology uses the principle of detecting infrared radiation imaging to record and display information such as the human body on a plane map. The night effect of infrared thermal imaging is relatively good, but it is only limited to nighttime, and it cannot display the normal imaging effect of the human eye. Not many people can accept the display of the reverse effect. In addition, at this stage, due to the high price of thermal imaging cameras, they can only be applied to some special fields, such as the digital army.

　　Figure 1 Infrared thermal imaging effect diagram

       2. Active infrared

　　At present, in the market, active infrared cameras are basically used in environments with very high requirements for low light. That is, in the night vision state, the infrared lamp will emit infrared light that is invisible to the naked eye to illuminate the object being photographed. After the infrared light is reflected by the object, it enters the image sensor for imaging. At this time, what we see is the image formed by the reflection of infrared light, not the image formed by the reflection of visible light. In this way, objects that are invisible to the naked eye in dark environments can be photographed, but this imaging method has the following problems:

　　(1) Limitations of reflected light applications. If the monitoring environment is relatively open, there is no object for infrared light to reflect, and the image is completely black, with poor display effects. If the monitoring environment is relatively complex, with multiple layers of scenery, the foreground will reflect the infrared light, and the background will not be visible. In addition, infrared light is monochrome, resulting in the final image being in black and white.

　　(2) Limitation of light attenuation distance. Currently, most infrared lights used for monitoring are LEDs, which emit scattered infrared light. With the addition of a lens, the maximum distance can only reach 100-120 meters.

　　(3) Limitations of infrared lamp life. Currently, the nominal life of infrared lamps ranges from 50,000 hours to 100,000 hours, but in actual applications it is even shorter. Excessive heat generation and excessive power are common problems in current applications [2]. If the infrared distance is required to be longer, the power will be greater, and the heat will be greater. If the infrared lamp is too close to the camera sensor, it will also affect the camera's life. Since the electro-optical conversion efficiency of LED infrared lamps is still below 20%, most of the electricity is converted into heat. As the number of lamps continues to increase, the product heat generation is also increasing. The temperature of the infrared machine has reached 80°C, or even above 100°C. At high temperatures, the LED chip, camera, and epoxy resin that encapsulates the LED are prone to accelerated aging, which greatly shortens the service life of the infrared camera. This is also the main problem with active infrared cameras.

Figure 2 Active infrared camera and imaging effect diagram

              3. Frame accumulation

　　Frame accumulation cameras improve the brightness of images by extending the exposure time. When the slow shutter function of such cameras is turned on, the output image is not a real-time image. As long as the shutter of the camera is slowed down by more than 4 times, you can see obvious image smear on the monitor. Slow shutter cameras do not improve image effects through the hardware performance of the machine, but improve image clarity in low-light situations by sacrificing the real-time nature of the image. However, for monitoring, what is important is a clear, coherent, and real-time image. Therefore, frame accumulation slow shutter cameras are often used to monitor static objects, and are not recommended for non-static applications.

　　4. More advanced technology

　　The most reasonable way to improve the effect of low-light cameras is to use hardware to improve product performance. For example, by making efforts in sensors and ISPs, we can maximize the use of ambient light and improve the low-light effect. ULLSTM (Ultra-Low Light SensingTM) independently developed by Beijing Zhongdian Xingfa Technology Co., Ltd. adopts the most advanced image sensor design technology in the world. It is not only small in size, fast in speed, and low in power consumption, but also has the characteristics of high sensitivity, high signal-to-noise ratio, and high dynamic range. In addition, with the unique ISP front-end image processing technology, it can accurately restore and greatly enhance the image, making the visual performance of the camera in low-light environments reach the world's leading level.

　　2. Implementation of ULLS Ultra-Low Light Sensing Technology

　　Different equipment manufacturers have different starlight-level low-light imaging processing technologies. This article focuses on the ultra-low-light visual perception technology independently developed by a well-known Beijing manufacturer. It integrates BSI image sensor design technology and ISP front-end imaging processing technology, and greatly improves the camera's photosensitivity through the most basic light collection and signal processing.

　　1. BSI image sensor design technology

　　Since entering the era of high-definition, as the number of image sensors continues to increase, their photosensitivity has been decreasing in proportion, and a new BSI sensor architecture model has emerged. BSI is designed to flip the upper and lower layers of the image sensor, so that the sensor can collect light through the silicon substrate that was originally on the bottom layer. This method is different from the traditional FSI sensor. On the FSI sensor, the light reaching the photosensitive area is to some extent limited by the metal wires and dielectric layers that perform photoelectric conversion in the sensor. The FSI method will hinder the light from reaching the pixel or cause the light to deviate from the pixel, ultimately reducing the fill factor and causing problems such as crosstalk between pixels. BSI moves the metal wires and dielectric layers to the bottom of the sensor array, providing the most direct channel for light to reach the pixel and optimizing light absorption [3]. Based on the BSI architecture, the ULLS image sensor provides a number of superior performance improvements, including stronger sensitivity per unit area, higher quantum efficiency, and reduced crosstalk and photoresponse non-uniformity. Combined with a special nano-scale design process and a new production process module, the low-light sensitivity of the image sensor is increased by more than 30%, and chromaticity and brightness interference can be effectively controlled, thereby significantly improving color and photoelectric performance.

Figure 3 Photoelectric performance of ULLS image sensor based on BSI architecture

　　
        2. ISP front-end image processing technology - 3D digital noise reduction

　　When the camera is running under low light conditions, fixed or random sensor noise will inevitably appear, and the noise generated by each frame of the image is different. If there is too much noise, the image will be blurred. Traditional 2D noise reduction technology belongs to the intra-frame noise reduction method, which has limited effect in eliminating dynamic noise. 3D digital noise reduction is based on intra-frame noise reduction. By comparing adjacent frames of images, the noise is automatically filtered out, thereby displaying a clearer and more delicate image. Starlight-level low-light imaging technology uses the inter-frame prediction method and the statistical law of image noise distribution to simultaneously estimate, predict and reduce noise in two dimensions of space and one dimension of time, thereby maximally eliminating the impact of noise on the image, greatly enhancing and improving the final imaging effect, and better identifying image details. In addition, when facing scenes with excessive light and dark contrast, 3D digital noise reduction can also effectively improve the display capability of wide dynamic range.

Figure 4 Comparison of intra-frame noise reduction and 3D noise reduction imaging effects

　　3. ISP front-end image processing technology - 3A imaging control

　　3A imaging control refers to automatic exposure control (AE), automatic focus control (AF), and automatic white balance control (AWB). Automatic exposure control can automatically adjust the brightness of the image, automatic focus control obtains the highest image frequency component by adjusting the position of the focusing lens, and automatic white balance compensates for the color difference caused by different light. Basic 3A imaging control is a necessary function of the camera, and an excellent 3A imaging control algorithm has a vital impact on the imaging effect of the camera. Starlight-level low-light imaging technology is based on the 3A imaging control algorithm of differential image analysis, which has the characteristics of accurate statistics, rapid response, and smooth adjustment. Whether in the early morning or dusk, or in a night environment with complex lighting, it can provide accurate color reproduction without being affected by framing and light and shadow, presenting a perfect day and night monitoring effect.

　　3. Application bottleneck of low-light technology

　　1. Price factor

　　At present, the prices of starlight-level low-light surveillance cameras on the market are generally high, which has deterred many industry customers from purchasing them. Under cost control, they have to settle for second best in actual surveillance applications. There are two main reasons for the high prices:

　　First, the quality of low-light camera components is higher than that of ordinary cameras, and some unique technologies are applied to further improve low-light performance. In addition, special low-light circuits are required in product design, so the cost of low-light cameras is relatively high.

　　Secondly, other surveillance cameras also have a certain impact on low-light products, such as infrared all-in-one cameras. In fact, these two products complement each other. Generally speaking, the price of infrared all-in-one cameras is lower than that of low-light cameras. In this way, when users can choose both infrared all-in-one cameras and low-light cameras, and when they do not have special requirements, if the budget is insufficient, users tend to prefer the cheaper infrared all-in-one products.

　　2. Market demand

　　According to statistics, the crime rate at night is as high as nearly 80%. The dim night vision environment catalyzes criminal behavior. With the continuous development and maturity of starlight-level low-light cameras, industry customers have begun to gradually realize the importance of nighttime monitoring effects. However, due to various factors such as price and application areas, people still lack objective cognition of the significance of starlight-level low-light cameras, and there is a great prejudice in terms of cost performance. A considerable proportion of customers still firmly believe that there are not many occasions that require cameras to maintain high-quality monitoring 24 hours a day, and ordinary cameras can meet the monitoring needs of most application areas. As a result, the market demand lags behind, and low-light cameras are still not the mainstream of the market. It can be seen that the application of low-light cameras still needs to overcome the human factors in selection in order to achieve a major breakthrough.

　　4. Application of Ultra-Low Light Sensing Technology (ULLS) in Public Safety

　　The social security management IoT application system based on ULLS (Ultra Low-Light Sensitivity) ultra-low light sensing technology responds to the national strategic development needs of social management innovation and harmonious society construction, and aims to improve the comprehensive management level of cities and the ability to prevent and control social security. The system is innovatively developed in the fields of front-end video perception and back-end information processing based on the three-layer architecture of the IoT system. It is a large-scale, comprehensive and complex application system that integrates multiple high-tech technologies such as software technology, microelectronics technology, computer and network technology, and radio and television technology, realizing the integration of multiple information systems and multiple technologies.

　　The technical innovation and leading advantages of this system are: it integrates a megapixel sensor specially designed for the starlight series products, the latest high-performance ARM processor, and an embedded LINUX operating system, showing a unique and world-leading ULLS ultra-low light perception performance, which can clearly present color images in extremely low-light monitoring environments (up to 0.001Lux), completely surpassing the limits of human vision. At the same time, with 3A imaging control technology and 3D digital noise reduction technology, the monitored objects can be accurately restored to their original colors under any lighting conditions without being affected by framing and light and shadow, greatly enhancing and improving the image effect in low-light and environments with strong light-dark contrast, and completely eliminating dynamic image noise. The company's unique core technology currently occupies an international leading position, effectively solving a series of problems encountered in the current social security management Internet of Things application fields such as safe cities and intelligent transportation, such as severe distortion of image color and low clarity at night or in complex lighting environments.

　　The system has been widely used in many projects of domestic safe city construction, greatly improving the image quality under extremely low illumination conditions, truly realizing the storage of massive video data and increasing the service life of the disk, effectively reducing the customer's investment cost. At the same time, it improves management efficiency through intelligent identification and behavior analysis of objects, and can be customized according to the needs of customers in different industries. It plays a positive role in maintaining social order, combating violent terrorist attacks, promoting social harmony, promoting social management innovation, and achieving long-term stability of the Party and the country.