OPPO launches NPU to usher in the era of computational imaging - what is expected and what is unexpected

Nearly two years have passed since the media broke the news that OPPO was making chips, and today OPPO officially released its first self-developed chip.

Recently at OPPO INNO DAY 2021, OPPO released its first 6nm NPU dedicated to imaging, code-named MariSilicon X, which is a small summary of its two-year chip-making process.

According to official information, MariSilicon X has four major features: extreme power consumption ratio, industry-leading HDR, 20-bit fast calculation for RAW, and support for customized RGBW Sensor optimization.

I wonder how you feel after seeing this news. For me, I initially thought it was reasonable, after all, up to now, several major mobile phone manufacturers have more or less launched their own chips.

But the performance of the parameters introduced shocked me. There are not many companies in the world that have launched EUV 6nm process NPU, and this is OPPO's first mass-produced chip. After OPPO announced a series of detailed chip indicators, I recalled 2017, when HiSilicon's first 10nm SoC Kirin 970 with integrated NPU was launched, opening the curtain of AI on mobile platforms. What is about to come is the next wave of mobile platform computing - AI computing imaging.

It is reasonable for mobile phone manufacturers to develop their own chips

近年来，无论是苹果、谷歌还是华为等手机商，都在自行开发SoC，其中最显著的特性就是增加了NPU，利用AI加速器提升应用性能并降低功耗。以往，计算任务是通用的，大多数任务都依赖于标准 CPU/GPU/VPU或者ISP/DSP等。人工智能（AI）和机器学习 (ML) 的飞跃式发展带来了独特的计算要求，传统的或者说通用的神经网络加速器(CNN)可能无法有效加速个性化的AI 任务，因此业界不仅需要用于处理 AI/ML 工作流的硬件，还需要专门针对特定 ML 模型优化的硬件。这些差异化需求很难依赖英特尔或者高通来快速获取。

For example, Apple and Google may both have voice assistants in their phones, but the underlying ML models and architectures they use may be very different, and general-purpose processors cannot be optimized. These companies can fine-tune their chips for their specific software applications and architectures, thereby achieving perfect tight coupling between algorithms, software, and hardware.

As one of the top two smartphone manufacturers in China, OPPO, which holds a large number of image-related patents, naturally has the motivation and ability to launch chips.

Back in February 2020, OPPO's special assistant to the CEO published an internal article titled "Some Thoughts on Building Core Technologies", exposing the "three major plans", namely the "Mariana Plan" involving the chip business (the MariSilicon X released by OPPO this time is named after this), the "Pantanal Plan" involving software development business, and the "Amazon Plan" involving cloud services.

So from that point on, everything made sense.

Mobile phones enter the era of computational imaging

As imaging becomes an important differentiating feature of mobile phones, the birth of MariSilicon X, which Jiang Bo called "the first step in exploring computational imaging", seems so natural - by combining functions such as ISP and NPU, OPPO has taken the lead in entering the era of computational imaging.

Jiang Bo, Senior Director of OPPO Chip Products

Jiang Bo is the senior director of OPPO chip products and took up the post in 2019. Prior to this, he served as the first smartphone SoC product manager at Qualcomm China and experienced the explosive growth of China's smartphone industry chain. OPPO is the first domestic company that Jiang Bo joined, and it is also a terminal manufacturer, which is completely different from his previous experience in chip companies.

It was during these two years that Jiang Bo gained a deeper understanding of OPPO through leading the chip development team and coordinating various departments: "As a first-tier terminal company, OPPO starts from the perspective of consumers. From the beginning of the project, MariSilicon X has been inferred from the user scenarios. This is completely different from the product development logic of chip companies before."

"OPPO is more focused on vertical integration from the perspective of solving users' pain points. We have SoC algorithm capabilities, hardware design capabilities, and the ability to customize and integrate sensors and camera/optical modules. As the most important link in the mobile phone industry chain, OPPO has the opportunity and ability to achieve the deepest integration and optimization, and it is consumer-oriented." Jiang Bo said.

Talking about the reason for the birth of MariSilicon X, Jiang Bo said that OPPO has accumulated CV (computer vision) algorithms and AI algorithms for general SoC platforms for many years, but general SoCs still face many challenges in achieving the ultimate experience of computational imaging. "For example, if customized sensors are to be used with general SoCs, the development cycles of the two are different and it is difficult to balance them. In addition, if commercial sensors are used, they need to be optimized on general SoCs, and R&D requires a lot of adjustments, which will increase costs and time to market. Only by tightly coupling algorithms with chips, software with hardware, can true computational imaging be achieved."

In fact, it is not just mobile phones. From cloud computing to automobiles, from IoT to industry, with the rise of technologies such as AI, the demand for computing power is becoming more and more personalized. Several software and hardware companies have chosen to develop their own chips. On the one hand, it is to ensure the stability of the supply chain, and more importantly, it is to improve product differentiation by mastering the core underlying technology.

All these changes are just as Nvidia CEO Jensen Huang judged in 2017: software will eat the world, but AI will eat software.

Mariana's Unexpected

Previously, some media said that OPPO executives never compare themselves with "competitors" in interviews, but stick to the corporate culture of "duty". OPPO was indeed not in a hurry to launch Mariana X, staggering the launch dates of several major competitors and SoC suppliers.

But "good things come late", compared with the "secretive" Surge C1 of Xiaomi and the "backed by a big tree" V1 of VIVO, the release of MariSilicon X is more frank. Although the positioning of MariSilicon X is not like the flagship SoC chips newly released by Qualcomm and MediaTek to the C-end, and therefore no detailed sensitive information such as the die area and number of transistors has been announced, Jiang Bo still disclosed as many chip details as possible. Through these unexpected details, the industry needs to take a new look at OPPO's first self-developed chip.

The only solution - self-developed multiple IPs

In this release, Jiang Bo only revealed three IPs in MariSilicon X, namely MariNeuro, MariLumi and MIPI PHY.

Among them, MariNeuro AI computing unit is the NPU developed by OPPO itself. Jiang Bo also admitted that many IP suppliers can now directly provide NPU, but through evaluation, "no NPU can achieve the optimal energy efficiency ratio based on OPPO's scenarios and algorithms."

The birth of MariNeuro can enable OPPO to have the underlying ability of training optimization, so as to achieve the best user photography experience. And at present, CV/AI algorithms are still being iterated, and choosing self-developed NPU can better grasp the initiative.

"Only through the use of a dedicated NPU for imaging can we solve the current problems of insufficient computing power and energy efficiency of ISP and general-purpose SoC. This is the mainstream direction of future imaging development," Jiang Bo concluded.

In addition, in order to solve the memory wall problem, OPPO has also developed an on-chip memory subsystem specifically for MariNeuro, which can achieve a transmission rate of terabit per second (Tb/s). In addition to on-chip storage, MariSilicon X also combines 256Mb of LPDDR4X as an independent DDR bandwidth to improve the overall memory throughput of the system.

MariLumi is OPPO's self-developed ISP Pipeline IP , which works with the NPU to achieve true real-time computing imaging.

In addition to the above two important IPs, OPPO has also developed many other IPs, such as MIPI interface, etc. Jiang Bo explained that the reason for self-development is that OPPO uses RGBW sensors, which require two RAW pipelines, so there are high requirements for the peak throughput of the MIPI bus. OPPO cannot find a matching IP on the market, so it has to choose self-development.

Jiang Bo emphasized that it is obviously unrealistic to develop all IPs by itself. OPPO also has a general principle for selecting commercial IPs, which is to meet the ultimate experience. For example, for the three products mentioned above, third-party IPs cannot meet the requirements, so they will develop solutions by themselves.

Pursuing perfection - choosing TSMC's 6nm EVU process

MariSilicon X uses TSMC's 6nm process. Jiang Bo also admitted that compared with 6nm, 12nm's R&D cycle, IP availability and wafer verification are more friendly. However, OPPO believes through simulation evaluation that only 6nm can meet the requirements of performance and power consumption at the same time.

It was in 2019, before the establishment of MariSilicon X, that TSMC announced the launch of the N6 (6nm) manufacturing process using EUV technology. Compared with the N7 (7nm) using DUV, N6 increased the logic density by 18%, with the same performance and power consumption, and used the same design rules, so that the ecosystem from IP to tools can continue to be used. According to TSMC’s recent announcement, it plans to replace 48-50% of 7nm production capacity with 6nm in the fourth quarter of 2021. Obviously, directly starting the project at 6nm can achieve faster commercialization.

Currently, almost all chip companies using the most advanced processes are SoC or CPU, etc. MariSilicon X chose 6nm technology regardless of cost, making it the chip that uses EUV technology second only to SoC in mobile phones.