Android flagships are all equipped with rear-mounted 3D cameras to compete with Apple's first release

With the release of the Huawei P30 series, the limits of smartphone photography have been further broken through; especially the Huawei P30 Pro mobile phone, with a DXO score of 112, which also allows people to see the power of the rear quad-camera (40-megapixel wide-angle lens + 16-megapixel ultra-wide-angle lens + 8-megapixel telephoto lens + ToF depth-sensing camera); of course, among them, the ToF depth-sensing camera brings Huawei P30 Pro not only improvements in photography.

In fact, the addition of ToF technology has become a new highlight of Huawei P30 Pro, and has also pushed the application of 3D vision technology in the entire smartphone industry to a new climax. According to rumors, Apple is also actively deploying rear-mounted ToF technology based on the existing front-mounted 3D camera, which makes people have greater expectations for the future of 3D vision on smartphones.

Currently, there are three mainstream solutions for 3D vision technology, namely structured light solution, ToF solution and binocular stereo imaging solution; among them, structured light and ToF have received widespread attention in the smartphone industry. The working principles of structured light and ToF are:

• Structured light technology: Through near-infrared lasers, light with certain structural characteristics is projected onto the object being photographed, and then collected by a special infrared camera. This light with a certain structure will collect different image phase information due to different depth areas of the object being photographed, and then the change in this structure is converted into depth information through the calculation unit to obtain a three-dimensional structure.

• TOF is the abbreviation of Time of Flight technology. Its principle is: the sensor emits modulated pulsed infrared light, which is reflected after encountering an object. The sensor calculates the time difference or phase difference between the emission and reflection of the light to convert the distance of the photographed scene to generate depth information. In addition, combined with traditional camera shooting, the three-dimensional outline of the object can be presented in the form of a topographic map with different colors representing different distances.

Hu Kefeng, general manager of the mobile phone business unit of Orbbec, told Leifeng.com that of the two solutions mentioned above, the current structured light technology is already quite mature in terms of technology and production. The industry has even developed a variety of high-end, low-end and medium-end solutions suitable for various mobile phone needs. In terms of maturity, it is actually not inferior to the current large-area optical fingerprint. He believes that compared with structured light, the implementation of ToF solutions on smartphones still has many problems to be solved:

• The resolution of the ToF solution is not as high as that of structured light, and its own power consumption is relatively large;

• In scanning applications, the current TOF solution actually has some impact on different colors, textures and materials. This is determined by its principle. Structured light is better in this regard. If the two solutions are used to scan the panda at the same time, the structured light solution is less affected by color, so the result will be more realistic; while under the ToF solution, the white part may come out OK, but the black part may be affected.

• The current TOF solution faces a problem: at long distances, its depth integrity is often not as good as that of the structured light solution, which has been demonstrated by some comparative tests conducted by Obbec. In somatosensory games, the principle of somatosensory is to extract relevant skeleton points as feature points; if the depth integrity is good, these feature points will not disappear.

ToF Principle

However, Hu Kefeng said that there are some important requirements for 3D solutions suitable for smartphones - miniaturization, low power consumption, and easy mass production. Judging from these requirements, TOF and structured light solutions have their own advantages and disadvantages; if at the cost of increasing power consumption or taking the cost of enlarging the volume, many problems can be solved. However, when it comes to smartphones, it is necessary to balance various aspects to make the machine small, power consumption low, and mass production very easy. In summary, both technologies have some difficulties to overcome.

However, Hu Kefeng told Leifeng.com:

Relatively speaking, for long distances, the relative accuracy of structured light will be a little worse. Basically, the accuracy curve of ToF is very flat. The accuracy curve of structured light has a higher linear rate, so at close distances, the relative accuracy of structured light will be better. At long distances, from 1.5 meters up to 3 meters, 4 meters, and 5 meters, the accuracy of structured light will actually drop faster. To solve a problem, both accuracy and depth are very important, otherwise there will be no application. So, in summary, ToF needs to solve the problem of how to make depth integrity better and reduce power consumption (in fact, power consumption of ToF is a big problem); and structured light needs to solve the problem of how to improve relative accuracy at long distances.

Regarding the front-end and back-end solutions of structured light and ToF on smartphones, the most common solution is to put ToF at the back and structured light at the front. In this regard, Hu Kefeng said that in fact, both ToF and structured light are possible, which depends on the degree of technological development and the actual choice of mobile phone manufacturers. He said that there were rumors that Apple had considered the rear structured light solution; but in fact, whether it is structured light or ToF, they are all building momentum for the entire 3D array and are very welcome; in fact, in addition to structured light, Obbec also has its own layout in ToF technology.

In general, on current smartphones, structured light mostly adopts front-end solutions, which are mainly used in unlocking, secure payment, 3D face modeling, etc.; while ToF solutions are mostly used in the rear of smartphones; in terms of application scenarios, in addition to being used for rear-end photography, ToF technology also plays a certain role in fields such as AR (including 3D photography, somatosensory games, etc.).

Of course, it is worth emphasizing the promotion of 3D vision technology to the AR field.

SLAM will help solve the unreasonable problems of AR games in the right picture

For AR, 3D vision will help smartphones better implement SLAM (Simultaneous localization and mapping) technology, which is the core of AR. Leifeng.com once reported that one of the great features of Apple ARKit is that it can implement SLAM technology on smartphones only through the monocular camera of the phone and the IMU; and relying on the depth of field information and modeling capabilities that ordinary cameras cannot match, 3D vision technology can help smartphones achieve more advanced visual SLAM (also known as VSLAM), thereby realizing a variety of complex AR applications on mobile phones.

In this regard, Hu Kefeng also said that through the combination of 3D and IMU, the ultimate direction of smartphone applications must be AR, and it must also be combined with 5G to make AR very convenient and eventually integrated into users' daily use.

In September 2017, with the release of iPhone X, 3D vision finally demonstrated its value on smartphones. Thanks to Apple's strong technical strength and its emphasis on user experience, iPhone X used 3D structured light technology to create Face ID, which can be directly used for facial recognition and quickly replaced Touch ID.

Therefore, with consumer-grade smartphones as the focus, major Android manufacturers have begun to focus on 3D vision, based on the combination of technology and application scenarios, mainly focusing on structured light and ToF.

Honor was the first to take this step. On November 28, 2017, Honor V10 was released, which was equipped with a "point cloud depth camera" speckle structured light mobile phone accessory with functions such as 3D face recognition and unlocking, and 3D face modeling.

At the end of May 2018, Xiaomi released the Mi 8 transparent explorer version, which is equipped with 3D structured light technology and is known as the first Android Face ID phone. The Mi 8 transparent explorer version uses coded structured light technology developed by Israel's Mantis Vision. It is understood that in Mantis Vision's coded structured light system, the VCSEL in the dot projector and flood illuminator is provided by ams, and the global shutter image sensor in the near-infrared camera is provided by OmniVision.

In June 2018, OPPO Find X was released. One of the main selling points of this flagship phone is the "FaceKey 3D structured light" technology and the "dual-track periscope structure" design. The supplier behind it is Obbec, and the two parties have an exclusive partnership. From the application scenario point of view, OPPO Find X's 3D structured light is mainly used for face recognition unlocking, 3D beauty, etc. - and Find X's "dual-track periscope structure" means that the 3D structured light camera module must be made thinner than iPhone X.

In addition, the Huawei Mate 20 Pro model released in October 2018 is also equipped with front-facing 3D structured light technology. It also realizes face unlocking and face payment in terms of functions. At the same time, it can also realize real-time 3D modeling through the 3D depth of field sensing function of 3D structured light; for example, take pictures of your own facial expressions and generate 3D emojis. Leifeng.com learned that the transmitting VCSEL of Huawei's models with structured light is supplied by Lumentum; the collimator G+P is supplied by Lianchuang Electronics and Sunny Optical; the DOE is supplied by AMS and DNP; the receiving end photosensitive chip is supplied by OmniVision Technology; the 3D structured light algorithm is self-developed by Huawei; and the entire 3D structured light module is supplied by OFILM and Sunny Optical.

However, while actively exploring 3D structured light, the entire industry is also continuing to explore ToF technology that is more suitable for rear cameras (of course, there are reports that Apple is also actively deploying ToF technology); at the same time, as the entire industry advances step by step in screen fingerprint recognition technology, the unlocking convenience brought by 3D structured light seems to be gradually eliminated - therefore, ToF technology is popular in the Android camp.

The first model to use ToF was the R17 Pro released by OPPO in August 2018. With the support of ToF, it has some new features in 3D photography, stereoscopic games, and AR applications. Leifeng.com learned from the supply chain that the 3D TOF camera on the R17 Pro is supplied by Sunny Optical.

On December 21, 2018, the vivo NEX dual-screen version was released, which is also equipped with ToF technology. Due to the front and back dual-screen solution, this model can also achieve three-camera selfies assisted by ToF technology, and can realize ToF face recognition. This face recognition function can achieve 24-hour face unlocking and even support Alipay payment function. Leifeng.com learned that the camera modules of the vivo NEX dual-screen version are supplied by Truly International, OFILM Technology and Samsung Electro-Mechanics, of which the 3D ToF module is exclusively supplied by Truly International; the other two camera modules are supplied by OFILM Technology and Samsung Electro-Mechanics, and Samsung Electro-Mechanics is the main supplier.

A few days after the release of the vivo NEX dual-screen version, Huawei's brand Honor also equipped ToF technology in its flagship model V20 at the end of 2018. At the press conference, Honor demonstrated functions such as somatosensory games, 3D modeling, and video slimming and body beauty. It is understood that in terms of 3D image sensors, Honor V20 and OPPO R17 Pro both use Sony's IMX316 sensor, and its module processing is completed by OFILM Technology and Sunny Optical Technology - the same supplier as Huawei P30 Pro.

It is worth mentioning that at the Samsung S10 launch conference in February 2019, Samsung also announced an S10 5G version, which is a mobile phone with four cameras, including a 3D camera that can realize ToF technology. Due to Samsung's self-developed advantages in the field of 3D image sensors, ToF technology is very likely to appear on other Samsung mobile phones, such as the Samsung Note 10 in the second half of the year.

Samsung's Korean "sister" LG also did not want to fall behind, and took a different path: using a ToF 3D module on the front camera instead of using structured light like Apple and other Android manufacturers. According to the official introduction, the LG G8 THINQ released in February is equipped with a front camera with ToF technology, and the sensor is Infineon's REAL3.

Of course, Huawei chose ToF technology for the newly released P30 Pro. According to Huawei, the ToF lens on the Huawei P30 Pro can achieve the following functions:

• Accurately capture depth of field information to make the pictures more layered;

• Achieve professional-level portrait blur, even down to the finest detail of a hair;

• Realize 3D somatosensory games, 3D magician and other functions;

• Utilize depth perception to measure the length, height and area of ​​objects.

It can be seen that Huawei P30 Pro's application of ToF technology is mainly reflected in two aspects. One is the enhancement of the picture layering and portrait blur function during mobile phone photography, and the other is other application scenarios opened up based on 3D imaging technology, such as 3D somatosensory games, AR, etc. However, in essence, these two aspects are inseparable from the basic principle of ToF technology, which is the perception and use of picture depth information.

According to information Leifeng.com has learned from the supply chain, the key suppliers of Huawei P30 Pro's ToF module are as follows: VCSEL comes from Lumentum; ceramic box comes from Kyocera; diffuser comes from Viavi Solutions. The lens module is provided by Sunny Optical and O-Film. Despite this, the suppliers behind some other key components of Huawei P30 Pro's ToF technology (such as 3D image sensors) are still unclear.

In addition, one thing that must be paid attention to is Apple's development in ToF technology. In November 2018, the United States Patent and Trademark Office (USPTO) announced Apple's new patent application, which showed that Apple may use optical electronic components such as VCSEL and ToF, single photon avalanche diode (SPAD)-based laser LiDAR to improve depth cameras.

Therefore, when Apple will launch ToF on the iPhone has become another focus of the industry.

In September 2013, Apple launched its latest iPhone 5s with fingerprint recognition technology, which immediately set off a revolution in the fingerprint recognition industry supply chain. In just a few years, fingerprint recognition has spread from flagship models to entry-level models, and finally has been basically popularized on smartphones, becoming a standard feature of smartphones.

So, will 3D vision technology become as popular on smartphones as fingerprint recognition?

Hu Kefeng said that from the perspective of unlocking and paying alone, if the emergence of fingerprint technology improves the unlocking experience by 60 to 80 points, then the improvement in facial unlocking or payment brought by 3D vision may be 85 to 95 or even 99 points. The former is "eating enough" and the latter is "eating well". Obviously, the demand for the former is stronger, so it is popularized faster.

Hu Kefeng said that for smartphones, in addition to unlocking payment (unlocking payment is even just a bonus function), 3D vision solutions have many other application potentials, such as AR games and AR video calls. However, these applications have not been fully developed and must be combined with AI, 5G and other technologies. If developed to a certain stage, 3D vision will gradually become popular.

There is no doubt that 3D vision has become a technological trend that the entire smartphone industry is actively embracing. It also contains numerous application potentials and will find greater vitality with the development of AI and 5G. 3D vision technology will undoubtedly move towards more smartphones, but it is still concentrated on flagship models. If you want to achieve a downward trend in product positioning and price, I am afraid it will take a long time.

In addition, judging from the current development trend, smartphone manufacturers' choice of 3D vision has gradually shifted from front-mounted 3D structured light to rear-mounted ToF technology, which reminds Leifeng.com of the Project Tango that has been declared dead. In fact, Google's mass-produced model Phab 2 Pro of Project Tango that was strongly promoted that year adopted ToF technology. Asus also tried it, but it did not receive a good market response. In the end, Google had to terminate Project Tango in 2017 and use ARCore to promote AR.

So, the final question is, now that ToF technology has been supported by more and more mobile phone manufacturers, should Google consider reviving Project Tango?

Apple's entire product line has reduced prices; 35 universities have added AI undergraduate majors; Yu Chengdong responded to Huawei's decision not to go public

The second batch of acceptance lists for the Science and Technology Innovation Board were announced. Why were all AI unicorns “annihilated”?

Zhou Hongyi launched three new products, targeting middle-aged people who are married and have children and are particularly tired of living.

Shenzhen Airlines App hijacks WeChat; Apple News crashes on its first day; Huawei P30 "telescope phone" officially released