Cut off 20% of the notch, behind Apple's "toothpaste squeezing": the pain and suffering of under-screen 3D visual technology
Latest update time：2021-09-28 17:40
Reads：
Full screen, why is it so difficult?
Author | Gao Xiusong
Editor | Yu Kuai
At the recent Apple conference, one of the highlights of the new iPhone was that the bangs were cut by 20%.
For those who have suffered from bangs for a long time, 20% makes almost no difference, so it is also ridiculed as a toothpaste-squeezing design.
However, no matter what we say, from a technical point of view, this 20% is actually hard-earned.
Because in this fringe, there are a total of 8 important components, including the front camera, microphone, laser transmitter, etc. Among them, the components that play a decisive role in the Face ID function and the front camera mainly have two parts: the laser transmitter module and the receiver.
"From the basic principle of structured light, there must be a certain distance between the laser transmitter and the receiver to maintain imaging accuracy. The longer the equivalent distance, the higher the accuracy, and vice versa. This is the fundamental reason why Apple retains the bangs." Lv Fanglu, chief scientist of Guangjian Technology, told AI Gold Rush.
Previously, on the iPhone 12, this distance was about 27 mm. This time the bangs have been reduced by 20%. Apple officials said that this was "thanks to the introduction of technologies such as sensor rearrangement and micro-slit earpieces."
"If you want to shorten the equivalent distance and maintain accuracy, the technical difficulty is very high." Lu Fanglu said that Apple should have taken another approach, that is, to optimize its 3D face algorithm model through accumulated face data and relax the accuracy requirements for 3D imaging.
In other words, algorithms are used to make up for the accuracy problems caused by data perception due to sensor rearrangement. This is actually a tricky method with relatively high requirements. It requires a large amount of facial data to support the continuous optimization of the algorithm.
However, this method is not replicable. First, the algorithm itself has defects and requires constant use of data for feedback and adjustment. Second, the supervision of facial data collection is becoming increasingly strict.
In fact, the best solution is to put the original camera module under the screen, which is often referred to as the "under-screen structured light solution" in the industry. This is in line with the development trend of full-screen phones and can solve the problem of low imaging accuracy caused by shortening the distance between the laser transmitter and the receiver.
As early as February this year, Guangjian Technology and ZTE Mobile launched the world's first under-screen structured light technology, which attracted public attention. Lu Fanglu said that the under-screen 3D technology has reached the mass production requirements, but the technology still involves product development and optimization on specific models, so it has not been actually mass-produced. In addition, there are still some problems that need to be solved for the under-screen 3D technology.
1
Transmittance: Under-screen structured light
Achilles' heel
Before talking about the under-screen structured light, let’s briefly talk about the technical principles of iPhone 3D structured light.
The 3D structured light module includes a transmitter (dot projector and floodlight sensor) and a receiver (infrared camera). The principle is as follows:
1) The laser emitter emits near-infrared light of a specific wavelength, forming a parallel beam with a very narrow cross-sectional area. After passing through the beam expander, its cross-sectional area is uniformly enlarged;2) The amplified light beam is parallelized by a collimating lens so that it is evenly incident on the DOE. When the light beam passes through the device, it forms a specific optical pattern and is emitted after passing through the projection lens.3) When these light beams are projected onto the surface of an object, the light signal will change;4) After receiving these changes, the receiving end (camera) will calculate the position and depth of the object and then restore the three-dimensional space through a specific algorithm.
Schematic diagram of dot projector and structured light principle
The flood illuminator consists of a low-power VCSEL laser and a diffuser. Its function is to emit invisible infrared light, allowing the infrared camera to receive the dot pattern reflected from the face even in the dark.
Under-screen structured light actually places the camera module containing these components under a "transparent" screen, which can both maintain the camera function and maximize the use of the screen area.
But the question that arises is: How much does this transparent screen affect the imaging accuracy of the camera module? How to compensate for the energy loss caused by light passing through the screen?
Light is emitted from the laser emitter, reaches the surface of the object, and then is absorbed by the sensor. It has to pass through the screen twice, which will cause energy loss. This is the "transmittance" problem faced by the under-screen structured light.
"For infrared rays, the current screens have been specially processed, and the light transmittance is about 30%, or even lower."
Lu Fanglu said that under-screen structured light can only be used on OLED screens, because OLED is an organic material that emits light and does not require a backlight like LCD.
However, organic light-emitting materials require electric current to emit light, and the electrode layer is generally not light-transmissive. Therefore, when the light emitted by the laser emitter passes through the screen, most of the light will be blocked or scattered by the electrode layer, resulting in only a small portion of the light being usable.
In addition, when light passes through the screen, since the screen itself is a periodic structure and the pixels are constantly repeated, it will cause a diffraction effect. For example, a beam of light will be diffracted into several beams of light, and the direction of light propagation will also be affected.
In actual scenes, light transmittance will also be affected by various noises.
For example, when the infrared rays in the sunlight shine on the face of a person, they will become noise of the infrared energy emitted by the laser transmitter. If the energy of the laser transmitter is not enough, it will be interfered by the noise, which will seriously affect the imaging effect.
"The under-screen structured light must first ensure that the screen display effect is normal and does not affect the user experience. Under this condition, solving the problem of light transmittance is the biggest difficulty." Lu Fanglu said that this is also the fundamental reason hindering the "mass production" of under-screen structured light.
2
Three solutions: screen, components and algorithms
The problem has been raised, and what remains is to find a solution.
The biggest challenge of under-screen structured light is "transmittance", which can be solved from three dimensions.
One is to start from the OLED screen, improve the panel transmittance, or customize the screen.
For example, the foldable screen mobile phone launched by Samsung uses UPC (Under Panel Camera) technology, the core of which is "Eco OLED" to improve the panel transmittance and optimize the pixel aperture.
Although improving light transmittance and customizing the screen are the most direct and effective solutions, this relies on the advancement of panel technology and involves cooperation between panel suppliers, under-screen structured light solution providers and mobile phone manufacturers.
"The cycle from screen design to development to actual use is relatively long." Lu Fanglu said that before panel technology has achieved groundbreaking breakthroughs and large-scale application, it is necessary to find another way to improve light transmittance.
The second is to start from the device module.
Since light is lost when passing through the screen, the energy of the light can be enhanced at the transmitting end to offset this energy loss.
It's like a 10-watt light bulb, the light energy can never be compared to a 100-watt light bulb.
But the crux of this method is that to enhance the energy of light, it is necessary to increase the power, which in turn increases power consumption.
How to enhance light energy without increasing energy consumption is the core of this solution and also the technical difficulty.
The third is to use algorithms to compensate.
After receiving the optical information, the receiver will perform calculations and use algorithms.
Then the algorithm can be optimized to compensate for some of the energy loss caused by "light transmittance".
If the amount of data is large enough and the algorithm is mature enough, the requirements for imaging accuracy can be lowered.
For example, Apple's Face ID uses facial data to optimize the 3D algorithm model. The flaws of this method are also obvious: it requires a large amount of data, and the algorithm will also deviate when facing specific scenarios.
Although these three solutions improve the "light transmittance" from different aspects, they are actually overlapping with each other. For example, panel customization must be adapted to the device module and combined with the algorithm. Although the relevant algorithms can optimize imaging accuracy, they can only serve as an auxiliary.
"To achieve mass production of under-screen structured light, we actually need the cooperation and progress of the entire industry chain," said Lu Fanglu.
3
The pain of domestic under-screen structured light
From notch screen to perforated screen, reducing the bangs to expand the screen's usable area and ultimately achieving a full screen has always been the ultimate goal of mobile phone manufacturers.
Under-screen 3D structured light is considered the best solution. Although there is a "transmittance" problem that needs to be solved urgently, there are always more solutions than problems. The above three solutions can actually solve a large part of the difficulties.
However, domestic manufacturers who want to develop under-screen structured light have to face a major hurdle besides technology: patents.
3D structured light requires a laser emitter to emit tens of thousands of laser speckles, which places high demands on the performance and power consumption of the laser itself. Currently, VCSEL laser emitters are basically used. In addition, the light must be "diffracted" by a derivative optical device (DOE) to "split" the light in order to achieve the function of projecting laser speckles.
Unfortunately, the manufacturing of these two components, VCSEL lasers and diffractive optical elements (DOE), is basically monopolized by manufacturers in the United States, the United Kingdom and Taiwan. Currently, only Sanan Optoelectronics can manufacture VCSEL chips in China.
Apple is undoubtedly the core beneficiary of the "VCSEL+DOE" 3D structured light solution, and also holds the patent for this technology. With patent protection becoming more and more important, this leaves manufacturers other than Apple with a difficult problem: how to make a solution with complete intellectual property rights?
Guangjian Technology's solution to this problem is to use edge emitting laser (EEL) as the light source, and then use the self-developed wavefront modulator (WFP) to modulate the light field on a sub-wavelength scale to achieve the function of projecting laser speckle.
This method avoids the technical patent of "VCSEL+DOE". The process technology of edge-emitting laser (EEL) is relatively mature and the cost is controllable. Domestic manufacturers have the corresponding production capacity.
"The WFP chip produced by Guangjian Technology has a manufacturing process of approximately 250nm, and domestic manufacturers have been able to achieve mass production," said Lu Fanglu.
As for the difficulty of under-screen structured light, Guangjian Technology mainly makes breakthroughs in device modules and algorithms, and optimizes 3D effects and screen display effects through long-term R&D cooperation with OLED screen manufacturers.
In terms of components, Guangjian Technology has improved the overall conversion efficiency of the laser emission module from electricity to light, that is, to enhance the energy of light at the emission end; at the same time, based on the EEL edge emitting laser, it shortens the pulse and increases the light intensity, thereby compensating for the energy loss caused by "transmittance".
It is reported that the overall conversion efficiency of its laser emission module from electricity to light is improved by 80% compared with the VSCEL solution; and the brightness of each pulse of the EEL edge light emitter is about 4 times that of the VSCEL solution.
In terms of algorithms, Guangjian Technology uses self-developed algorithms to construct laser information, reducing computing power while maintaining the same effect, without having to rely on ASIC chips, thereby reducing computing costs.
"Guangjian Technology and Apple have different technological paths, which avoids the risk of patent conflicts with Apple, and has achieved localization of core components." Lu Fanglu said that domestic manufacturers are also currently carrying out technological innovation, such as Obbec, and under-screen 3D technology is receiving more and more attention.
4
Summarize
From Apple's Face ID using 3D structured light to the full-screen displays currently promoted by the industry, the shrinking of the "bangs" is due to the continuous change and evolution of technology.
"In the future, vision will inevitably change from 2D to 3D, because 3D information is richer and more comprehensive." Lu Fanglu said that 3D information can simplify the amount of calculation, and there is no need to do too much data collection and model training. As the core technology of full-screen, 3D structured light under the screen still has difficulties to overcome and the localization capacity needs to be strengthened, but the application prospects are relatively optimistic.
"In the next one or two years, under-screen 3D structured light technology should be able to be used on a large scale."
This article is originally written by Leifeng.com, author: Gao Xiusong. Please reply "reprint" to apply for authorization. Reprinting without authorization is prohibited.
END
Recommended Reading
Featured Posts
京公网安备 11010802033920号