Apple's AR/VR failed to meet its promise: complaints, speculations and truth

Apple's 2021 autumn new product launch conference was disappointing. Various complaints were flooding the circle of friends: "I stayed up late, which was disappointing", "A set of molds every day, changing the color, adding some anti-shake, and upgrading the parameters", "Even Apple is playing the cost-effective card", "After Jobs, Apple has no more surprises"... Of course, there are also praises, about the online press conference, which was held very much like a scenic blockbuster, making people miss the sunshine in California, and the new A15 chip and Apple iPhone 13 have some new features, such as "major image upgrade" and "customized camera function". But no matter how many new features the iPhone 13 has, it is still a mobile phone. Since the launch of the subversive iPhone in 2007, which defined the smartphone, Apple has reaped 14 years of dividends from this product form. Although Apple later launched headphones, watches, TVs, etc., they were not enough to stimulate consumers' nerves.

If Apple still cannot break through itself and bring amazing products on the next generation computing platform, then the era of Apple may really be over. The era of the Metaverse is opening up. Facebook, HTC, Google, Huawei, etc. have all launched various AR/VR products to seize the beachhead and layout the Metaverse, but Apple's AR/VR helmet has not been seen so far. So, if we say what the industry is most looking forward to at Apple's press conference last night, it may not be the iPhone "13 flavors", but Apple's AR/VR headset.

What does Apple's VR/AR headset look like? Which is better, it or Facebook's Oculus Quest? The iPhone once defined the mobile Internet era, will Apple also define the key hardware in the Metaverse era?

Apple chooses AR over VR?

Apple is not an aggressive company. It usually launches products only when the industry is mature. This is true for MP3 players and mobile phones. However, once they are launched, they often subvert imagination and bring surprises. This is why Apple is expected by the world. After Google, Facebook, HTC and others have launched VR devices for a long time, Apple's VR/AR headset has finally surfaced.

Although Apple's annual "Spring Festival Gala" autumn conference last night did not reveal any information about VR helmets, some media recently broke the news about more details of Apple's self-developed AR/VR chips. In addition, there have been reports that Apple's first VR headset will be released at the 2022 Apple Worldwide Developers Conference, and Foxconn and supply chain companies are already preparing stock, and the stock volume is even not less than that of mobile phones. All these information make Apple's VR/AR headset imminent.

At present, Apple has become more and more determined on the path of self-developed chips. Although the A15 chip with 15 billion transistors released last night is considered mediocre by the chip industry, it is already the fastest chip in smartphones, so getting rid of third-party self-developed AR/VR chips is still Apple's choice.

According to the leaked information about Apple's self-developed AR/VR SoC chip, the chip is manufactured by TSMC, uses a 5nm process, and is currently in the trial production stage. It is reported that its chip is not as powerful as the SoC on the iPhone, iPad, and Macbook, and lacks AI and machine learning capabilities, but the chip can communicate wirelessly with the host device, which may be a mobile phone, computer or tablet, similar to the "cloud and end" interaction. In this way, the host can handle the stronger computing power required to display VR, MR and AR images, and the helmet on the end can be lighter. At present, the size and weight of the headset are still a hurdle that affects consumer acceptance.

Weng Dongdong, a researcher at the Institute of Optoelectronic Information Technology and Color Engineering, School of Optoelectronics at Beijing Institute of Technology, said in an interview with China Electronics News that Apple is likely to launch an AR (mixed reality) device rather than a VR (virtual reality) device. AR is very convenient to connect with mobile phones, and a large number of mobile phone software can be presented on AR. This entry point can accelerate the maturity of the software ecosystem.

This is different from the geek route taken by Facebook Oculus Quest. Facebook hopes to enter the market from VR and leverage the geek players. Apple's route has always been extreme and large-scale, and starting with AR can make full use of Apple's advantageous mobile phone resources.

Apple has previously released the ARKit architecture. Many mobile scene applications such as navigation, interaction, and payment can be better presented and interacted with or even extended through AR. For example, with the bionic chip and LiDAR laser radar scanner on the current iPad/iPhone, new spaces and dimensions can be extended. The latest ARKit5 position anchoring supports more regions, face tracking function upgrades, and improved motion tracking. Some people predict that the early Apple AR is just a second screen for mobile phones, so the cost of the education market is not high and can be quickly increased.

However, Zhao Yan, director of the research office of CCID think tank's Institute of Electronic Information, told China Electronics News: "Apple devices may have both VR and AR functions, and it is not entirely certain that Apple will only do AR. In November 2017, Apple acquired Vrvana, a Canadian mixed reality technology provider, which has the technology to integrate augmented reality (AR) and virtual reality (VR) into the same headset - Totem. So there is no problem with the technical reserves. The key is how Apple will choose in terms of business path and implementation options.

How can Apple overcome the three hurdles of VR/AR?

There are three obstacles to VR/AR being accepted by the public. The first obstacle is size and weight. The current device form is still too bulky, and the headset still needs to be further lightweight and miniaturized.

"It can be expected that Apple's AR products will be greatly optimized in appearance and size, and will be closer to general-purpose devices that can be worn daily for a long time, rather than like Microsoft's HoloLens. The outstanding features are simplicity and lightweight," said Weng Dongdong.

The second hurdle for VR/AR is the display dizziness problem. Zhao Yan told reporters: The current display still has dizziness problems. The products on the market are mainly based on the binocular parallax display principle, which theoretically cannot solve the dizziness problem caused by the convergence and focusing contradiction; and light field display and holographic display are still in the research and development stage and have not yet become mature products.

Zhao Yan further stated that Apple devices may have 8K ultra-high-definition display capabilities, using 8K silicon-based OLED. In this way, the resolution of Apple's head display will be the highest in the industry. Currently, Oculus Quest2 and Pico Neo3 use 4K LCD screens, and the brightness, contrast, and response time of silicon-based OLED are all higher than LCD.

According to the leaked Apple chip information, Apple has completed the design of the image sensor and display driver for this chip. This CMOS image sensor version is unusually large, close in size to a head-mounted display lens, allowing users to capture high-resolution image data from their surroundings for use in AR technology. According to industry speculation, TSMC has been having difficulty producing defect-free chips, and the output during trial production was very low.

Apple has been looking for more extreme display technology. In 2018, Apple acquired Akonia Holographics, a US holographic system and material startup. Based on the company's exclusive patented technology "HoloMirror", Akonia Holographics has created an AR display that can achieve ultra-high-definition full-color display on AR headsets and is extremely lightweight.

In addition to being more compact and lightweight, and featuring a higher-definition display, Apple's AR core technology has not yet surfaced. If touch interaction technology is the key to Apple's opening of the smartphone era, then in the metaverse era, what is the key to Apple's human-computer interaction? The answer has not been solved.

The law that one generation of interactive technology determines a product era has been verified. From the PC era to the mobile Internet era, and then to the Metaverse era, what is the core technology of interaction?

This brings us to the third hurdle of AR/VR, the "interaction fidelity problem". "At present, the problem of natural and realistic interaction has not yet broken through the limitation of the physical size of the screen, and it is impossible to achieve natural human-computer interaction with wireless transmission and hand-eye coordination; there are still many problems in the way, mechanism and equipment of realistic force/tactile perception between people and virtual objects, and touch, temperature and humidity, smell/taste, etc. are still in the early stages of research or have not yet been involved." Zhao Yan said.

Apple has been looking for the core logic of AR/VR interactive technology. In 2013, Apple acquired Israeli company PrimeSense. This is a company that focuses on 3D sensing and motion capture technology. Its PrimeSense 3D sensing technology can be applied to AR applications such as interactive games and indoor map navigation. Microsoft's Kinect was the first to use PrimeSense's sensing technology. In addition, this technology can also perform high-precision 360° scanning of the real world. Later, Apple successively acquired facial motion capture technology companies, facial expression AI analysis companies, AR social software, and so on. Apple hopes to find amazing AR/VR interactive technology.

What kind of AR/VR hardware can Apple come up with in the Metaverse era? Can it overcome the three current hurdles? We did not find the answer at this fall conference, but perhaps the truth will be revealed by this time next year.