The 3D sensing market is about to change! With Apple's strong push, dToF will become a new trend!

In the early morning of October 14th, Beijing time, Apple officially released four models of the iPhone 12 series. As previously reported, the iPhone 12 Pro and iPhone 12 Pro Max have both added a laser radar scanner based on dToF technology that was first adopted by the iPad Pro. This also means that Apple is further increasing its application of ToF technology, and this move may accelerate the explosion of the entire ToF market.

Although many flagship smartphones have deployed ToF 3D cameras, they are basically based on iToF technology. In comparison, dToF has the advantages of low power consumption and anti-interference, and is suitable for long-distance ranging scenarios with high requirements for ranging accuracy. With the adoption of Apple's iPhone 12 series and the development of AR/VR technology, dToF is expected to become the next outlet for smartphone cameras.

1. dToF opens up a new future of deep information

3D sensing is one of the trends in smartphone innovation and is currently accelerating its penetration into mid- and low-end smartphones. Currently, there are three technologies for implementing 3D sensing, namely binocular stereo imaging, structured light, and ToF. The more mature solutions are structured light and ToF. Among them, the structured light solution is the most mature and has been widely used in industrial 3D vision. ToF has become a more promising solution on the mobile side due to its own advantages.

The multi-scenario application of ToF presents a broader development prospect than structured light. The disadvantage of the working distance limits the application of structured light, while ToF technology makes up for the disadvantage of distance and can be applied to more scenarios including 3D face recognition, 3D modeling and gesture recognition, somatosensory games, AR/VR, etc., which can bring more entertaining and practical experience to smartphones. In addition, compared with structured light technology, ToF has low module complexity and simple stacking, and can be very small and durable. With the trend of increasing screen-to-body ratio, it is more favored by mobile phone manufacturers.

After Apple’s new iPad Pro took the lead in adopting the “radar scanner” based on dToF technology this year, it also attracted the industry’s attention to the two ToF technologies, dToF and iToF.

ToF (Time of Flight) calculates the distance between the light source and the object by measuring the flight time of the emitted light and the reflected light. It is essentially a measurement in the time dimension. Depending on the distance measurement method, there are currently two ToF technology routes: iToF (indirect-ToF) and dToF (direct-ToF). dToF directly measures the flight time. The principle is to directly emit a light pulse to the measured object and measure the time interval between the reflected light pulse and the emitted light pulse to obtain the flight time of the light, thereby directly calculating the depth of the object to be measured. iToF measures the flight time by emitting modulated light of a specific frequency and detecting the phase difference between the reflected modulated light and the emitted modulated light.

iToF indirectly measures the time of flight and has the advantages of low cost and high resolution, making it suitable for short-distance ranging. The principle of iToF is to modulate the emitted light into a periodic signal of a certain frequency, measure the phase difference between the emitted signal and the signal reflected back to the receiving end after reaching the measured object, and indirectly calculate the time of flight. Since the pixel of the iToF sensor is relatively small, a relatively high image resolution can be achieved. However, the problem with iToF is that the realization of ranging accuracy limits the maximum ranging distance. In principle, the higher the modulation frequency, the better the ranging accuracy. A high modulation frequency means that the corresponding ranging distance cannot be too large, and the ambient light will interfere with the circuit. Therefore, iToF is currently mainly used in scenarios with short ranging distances such as mobile phone facial recognition and gesture recognition.

The iToF sensor circuit is relatively simple, and the difficulty lies mainly in the depth algorithm. The Android camp has introduced iToF since 2018 and promoted its mainstreaming. Currently, brands such as Samsung, Huawei, OPPO, and vivo have equipped it in mid-to-high-end models. In addition, iToF can reproduce more detailed information in the scene in application scenarios such as object recognition, 3D reconstruction, and behavior analysis, so it is also widely used in robotics, new retail and other fields.

dToF directly measures the time of flight and has the advantages of low power consumption and anti-interference, making it suitable for long-distance ranging scenarios that require high ranging accuracy. The dToF principle is to emit light pulses to the object to be measured, and directly calculate the depth of the object to be measured by measuring the time interval between the reflected and emitted light pulses. The ranging principle ensures that the dTOF measurement accuracy will not decrease with increasing distance, and it has lower power consumption and stronger anti-interference ability to ambient light.

The dToF depth algorithm is relatively simple, but the difficulty lies in the SPAD used to achieve higher precision. dToF needs to detect light pulse signals (nanosecond or even picosecond level), so the sensitivity to light is very high. Therefore, the receiving end usually chooses sensors such as SPAD (single photon avalanche diode) or APD (avalanche photodiode) to achieve this. The integration level is weaker than that of ordinary CMOS image sensors, and the pixel size is generally larger than 10μm, so the resolution is usually poor and the cost is higher. SPAD is the core of dTOF technology. It is technically difficult and the manufacturing process is complex. Currently, very few manufacturers in the world have the ability to mass produce it. It is difficult to integrate and it is difficult to miniaturize it for use in small consumer electronics such as mobile phones. Therefore, in addition to the traditional popular application field of automotive LiDAR, Apple is currently the only company in the consumer electronics field that has achieved commercial use (iPad Pro is equipped for the first time).

In the future, TOF will develop towards higher integration, smaller sensor size, and higher resolution. At present, the minimum single pixel size of traditional CIS can reach 0.7μm, and 0.6μm is already under development. However, ToF sensors require the ability of single pixels to obtain signals, so a larger single pixel size is required; the circuit design of dToF sensors is relatively complex and requires a larger on-chip size; the pixel size of iTOF needs to temporarily give way to higher light collection efficiency. For various reasons, the miniaturization of ToF image sensors is difficult.

Improvements in semiconductor processes are expected to enable miniaturization of TOF sensors. ToF sensor manufacturers have improved semiconductor process solutions, such as back-illuminated (BSI) and stacked CMOS technologies, by moving the wiring layer originally located above the photodiode to the bottom, and vertically stacking the photoelectric converter and electron multipier, thereby increasing the pixel aperture ratio and reducing the pixel size. According to Panasonic's latest research results, dToF sensors can also be implemented using CMOS technology, and the degree of integration is already close to that of the iToF solution in terms of magnitude.

At present, the inherent defect of low resolution of ToF technology still exists, and it is expected to achieve breakthroughs in the future with technological changes. At present, the measurement accuracy of ToF is still lagging behind that of structured light solutions, but its sensor resolution has been improving in the past two years. In terms of iToF, Infineon's general REAL3™ sensor (iToF) for the consumer market has also reached 38,000 pixels, and the IRS2771C launched in 2019 has reached 150,000 pixels; in terms of dToF, for example, the LiDAR resolution of the iPad Pro 2020 has reached 30,000 pixels; in addition, the progress of TDC circuit design has gradually improved the TDC time resolution accuracy in CMOS circuits, which is expected to bring about an improvement in the resolution of dToF.

2. ToF Future The most promising application is in the AR field

Currently, mobile phones are the main application area of ​​ToF in consumer electronics. As the market's interest in 3D vision and recognition technology grows, leading terminal manufacturers are pushing TOF technology to continue to expand in the direction of 3D perception and imaging. We see that TOF technology is accelerating its penetration in smartphones, and the use of TOF is further enriching the application scenarios of 3D sensing. With the adoption of Apple's iPhone 12 series and the development of AR/VR technology, ToF is expected to become the next outlet for smartphone cameras.

ToF promotes the popularization of consumer-grade AR. The application of ToF technology is also a catalyst for the AR and VR era. Considering the two unique advantages of ToF - long range and high refresh rate, AR/VR with long-distance 3D ranging needs is one of the functions that best reflects the advantages of TOF. The gesture recognition function provided by 3D camera technology will become the core interactive means in the future AR/VR field. At present, most VR devices launched by major manufacturers require controllers. The advantages of game controllers are timely control feedback and multiple combination states.

According to Markets and Markets, the global AR market size reached US$10.7 billion in 2019 and is expected to reach US$72.7 billion by 2024, with a compound growth rate of 46.6%. In the past few years, companies represented by Facebook, Intel, Qualcomm and Samsung have made large investments in the AR field, driving the rapid growth of the global AR market. The size of China's AR market is expected to reach approximately US$5.9 billion in 2024. From the perspective of downstream applications, industrial applications account for the largest proportion, accounting for approximately 42%, followed by automobiles (18%), retail (15%), and aviation and defense (10%).

AR interior design. The 2020 iPad Pro uses dToF LiDAR technology, which can obtain 3D space depth information, build detailed indoor environment space data, and simulate the situation after placing new furniture. IKEA's IKEA Place app uses AR to directly present the appearance of home products and the placement effect in the home to users.

Medical learning. Complete Anatomy is a software that teaches medical students to understand the human body structure such as the heart, real-time muscle movement, and nervous system through virtual technology. This software can be used on the 2020 iPad Pro. It will help professionals more accurately assess the patient's body movement and bring more possibilities for the future development of medicine.

The gesture recognition function provided by 3D camera technology will become the core interactive means in the future AR/VR field. Currently, most VR devices launched by major manufacturers require controllers. The advantages of game controllers are timely control feedback and multiple combination states. Taking HoloLens as an example, it has a set of four environment perception cameras and a depth camera. The environment perception camera is used for human brain tracking, and the depth camera is used to assist gesture recognition and perform three-dimensional reconstruction of the environment.

Photo blur. ToF has a better depth of field information collection function. After being added to the rear camera module of a smartphone, it can quickly and long-distance obtain a higher-precision depth map, thereby completing 3D modeling with a larger range than structured light. Moreover, due to its own infrared light source, the depth of field information obtained in a dark environment is equally accurate. Therefore, the imaging with the participation of the TOF camera will be more realistic and layered in the blur effect. Huawei's flagship P30 Pro released in 2019 added ToF lens assistance to the rear 3D imaging and perception module. The ToF lens obtains more depth of field information to enhance the background blur function, which is more accurate than binocular vision, making the blurred edges of the obtained image clearer and more expressive.

Gesture recognition. Currently, many mobile phones have floating gesture recognition function, which does not require direct contact with the mobile phone screen. It only relies on the 3D perception of gestures by the front ToF, and realizes common operations such as turning pages and scrolling screens through simple operations such as waving hands in front of the phone. Compared with the former, somatosensory games are more interactive. Through TOF technology, the depth information of the body of the person being photographed can be collected, the body movements can be captured and collected, gesture judgment can be performed, the image and movement of the pre-made 3D modeled doll can be controlled, and the real person and 3D virtual image can follow, and the game interaction can be performed with the body, movement and gesture.

The application of ToF technology is a catalyst for the AR and VR era. Considering the two unique advantages of ToF - long range and high refresh rate, AR/VR with long-distance 3D ranging needs is one of the functions that best reflects the advantages of TOF. The gesture recognition function provided by 3D camera technology will become the core interactive means in the future AR/VR field.

The application of dTOF technology is expected to promote the improvement of AR content and accelerate the popularization of consumer-grade AR. In 2017, Apple released the ARKit development tool for developers to develop AR applications on iOS devices. The iPad Pro released in 2020 is the first consumer electronic device equipped with a dToF module, which can be seen as Apple's further layout in the AR field in the 5G era.

The LiDAR (laser radar scanner) previously equipped on the iPad Pro uses Sony's 30,000-pixel 10μm dTOF image sensor, a SPAD array detector, and integrates Lumentum's VCSEL chip and TI's VCSEL driver chip. It can achieve ps-level time resolution, realize 3D perception and imaging within a range of 5 meters, and has faster AR modeling speed, higher measurement accuracy, and less jitter and misalignment.

▲iFixit disassembles the rear camera module of the new iPad Pro

▲The sensor inside the "LiDAR Scanner" of the new iPad Pro disassembled by TechInsights

Sony's dTOF solution uses a 3D stacking process for the first time. The pixel connection is achieved by bonding the detector wafer and the logic circuit wafer together through hybrid bonding interconnection technology. At the same time, deep trench isolation is also used, and the metal-filled trench completely isolates the pixel. This effectively controls the size of the dTOF sensor, allowing it to be successfully used in small consumer electronic devices such as the iPad.

Currently, iPad Pro's LiDAR has three typical application scenarios: AR measurement, AR games, and AR decoration design.

AR measurement: LiDAR can quickly calculate a person's height and display vertical and edge guide lines. Apps developed by developers can achieve more detailed measurements of object sizes and buildings.

AR games: LiDAR has opened up a wider design space for AR games by scanning the surrounding real environment and quickly obtaining depth information. For example, the "Hot Lava" video game displayed on the official website can turn the living room into a virtual lava environment. Players in the game can jump on furniture to avoid the simulated floor lava. After the iPad Pro was launched, developers continued to enrich the AR game content on the iOS platform, and some existing AR games became more vital because of the upgraded gameplay.

AR decoration: Shapr3D app on iOS uses LiDAR to scan the room and create a 3D model. Users can edit the model or add new objects. Using AR, users can view the virtual effect of the actual room after editing, helping users experience the design effect more realistically before the decoration starts. IKEA Place app can also get matching furniture recommendations by scanning a room, and then use AR to view the furniture placement effect.

The application of dTOF on iPad Pro can be regarded as the first step for Apple to build a hardware foundation for the AR ecosystem. In the future, through technological improvements and breakthroughs, Apple is expected to introduce dTOF to mobile phones and more AR devices, promote the development of AR hardware devices, and inspire designers to develop AR content applications for more scenarios such as architecture, education, and medical care based on the characteristics of dTOF, and promote the continuous improvement of the AR application ecosystem.

Huawei Cyberverse defines earth-level XR applications (including VR, AR, MR and other extended reality technologies), and expands AR applications to a wider digital world. Huawei Cyberverse is defined as a digital platform for providing earth-level virtual reality fusion services. Huawei AR Map is its first commercial product. Its main functions include AR real-life navigation, holographic information display, virtual-real fusion photography and other virtual activities. Full-scene spatial computing capability is the core of Cyberverse. The macro map required for this capability can use satellite positioning, while the modeling and positioning of indoor and some micro places depends on the 3D sensing of smartphones. Currently, Huawei P40 series models can already support Huawei AR Maps.

The next wave of mobile terminal innovation will be revolutionary around AR. The establishment and improvement of AR/VR development platforms and the booming augmented reality content market will inevitably promote the development of the TOF industry. TOF is expected to take over from structured light, from biological perception to virtual reality, from face recognition to 3D modeling, bringing industry upgrades and user experience optimization. Front-mounted face recognition + rear-mounted virtual reality functions may become the next form of mobile phones. With the development of AR/VR, ToF is expected to become the next outlet for smartphone cameras in the 5G era.

3. The next wave of innovative revolution, TOF market space is huge

The next wave of mobile terminal innovation will be revolutionary around AR. With the booming market of augmented reality content, content manufacturers continue to promote the development of AR/VR development platforms, which will inevitably promote the development of the TOF industry. TOF is expected to take over from structured light, from biological perception to virtual reality, from face recognition to 3D modeling, bringing industry upgrades and user experience optimization. Front-mounted face recognition + rear-mounted virtual reality functions may become the next form of mobile phones. With the development of AR/VR, ToF is expected to become the next outlet for smartphone cameras.

We can see that in 2019, 3D sensing mobile phones are mostly concentrated in high-end flagship models. Apple is the representative of structured light. All models after iPhoneX have been equipped with structured light function, and Huawei has the largest number of models equipped with TOF. According to Yole's forecast data, the global 3D imaging and sensor market size will have a CAGR of 38% from 2016 to 2022, with a market size of US$1.83 billion in 2017 and more than US$9 billion in 2022. Among them, consumer electronics is the fastest growing application field, with a CAGR of 160% from 2016 to 2022, and the consumer electronics market size will exceed US$6 billion by 2022.

In terms of shipments, we predict that the demand for 3D sensing in smartphones will increase from 40 million units in 2017 to more than 200 million units in 2019. Among them, ToF models in 2019 will mainly be concentrated in several high-end flagship models. Starting from 2020, TOF shipments will further explode, especially after the release of the iPhone 12 series. TOF is expected to account for 40% of the overall 3D sensing.

We forecast that TOF shipments in 2019/2020 will be 77.6 million/160 million units, a significant year-on-year increase of 747%/108%.

4. BOM comparison : T OF may have more cost advantages

We estimate that the BOM cost of ToF and structured light is about US$12~15 and US$20 respectively. In comparison, TOF has a greater cost advantage. Taking iPhone X as an example, the solution of structured light technology includes three sub-modules (dot projector, near-infrared camera and flood illuminator + proximity sensor), while the ToF solution integrates the three into one module, which can reduce the packaging cost.

We estimate that in this TOF module, the cost of the chip will still account for the majority, accounting for approximately 28%~30% of the overall BOM.

5. In-depth analysis of the 3D Sensing camera industry chain

At present, TOF or structured light 3D sensing technologies are all active sensing, so the 3D camera industry chain mainly adds infrared light sources, infrared sensors and optical components compared to the traditional camera industry chain. By disassembling and analyzing the mainstream 3D camera products that have been launched on the market, the 3D camera industry chain can be divided into: 1. Upstream: infrared sensors, infrared light sources, optical components, optical lenses and CMOS image sensors; 2. Midstream: sensor modules, camera modules, light source foundry, light source detection and image algorithms; 3. Downstream: terminal manufacturers and applications .

Although TOF and structured light have different principles, the core components they require are basically the same. The core components in TOF include VCSEL light source and diffuser at the transmitting end, and lens, narrow-band filter, near-infrared CMOS, etc. at the receiving end.

Editor: Core Intelligence-Rurouni Sword

Source: Guosheng Securities, Core Intelligence past articles

Event Preview: On October 30, Xinzhixun will hold the "New Challenges, New Opportunities - 2020 Fourth Biometrics Technology and Application Forum" in Shenzhen. At that time, Xinzhixun will invite leaders of 3D sensing companies such as GigaDevice, ams Semiconductor, and Sunny Optical to discuss the development of TOF technology. Welcome to click "Read the original text" at the end of the article to sign up and participate!