Mobile phone manufacturers are so busy making chips, just for this? ?

Yang Jing Xiaoxiao sent from Aofei Temple
Quantum Bit | Public Account QbitAI

Can you believe that in order to make your mobile phone able to directly shoot blockbuster effects, engineers actually did this:

Invest more than 300 people to focus on chip manufacturing.

Previously, they spent two years building the system from scratch to make night scene photography clearer.

This time, in just half a year, they have iterated to the Plus version, upgrading the portrait to a new level...

Since when did the imaging system of mobile phones become so advanced?

It is understandable that the imaging system has always been the protagonist at major mobile phone manufacturers' press conferences. After all, this is what users can feel most intuitively.

But at first, it was just a matter of comparing pixels and the number of cameras. Or more advanced ones were to see what personalized algorithms were available, such as beauty features to differentiate between boys and girls, scene recognition, and night scene noise reduction.

Why are there manufacturers now specializing in developing chips?

Moreover, this trend is still growing and has entered a mode of annual iteration.

Why does the exploration of mobile phone imaging now rely on chips? What room is there for further development?

Mobile phone imaging is gradually entering the involution

To understand the "involution" logic of mobile phone imaging, we need to look at the entire history of mobile phone imaging development.

Since the birth of the world's first mobile phone with a camera, the Sharp J-SH04 (2000) , imaging has become a new competition track for mobile phone manufacturers.

From a technical perspective, image upgrades involve hardware and software:

Hardware includes cameras, sensors, camera modules, chips, etc.; software is algorithms.

But from the perspective of time, mobile phone imaging can be said to be a process of returning to the origin .

On the surface, the past decade or so of mobile phone imaging has been in an era of hardware "involution".

At that time, the idea of ​​each manufacturer was to simulate the camera, surpass and replace the camera .

The first to start involution were image sensors represented by CMOS materials, especially the back-illuminated sensors installed in the iPhone 4 in 2010, which later directly increased the pixels of major mobile phones to tens of millions.

Theoretically, when shooting in low light, noise can be greatly reduced. And as the CMOS area gets larger, the image becomes clearer, which is what the digital industry often says: "A larger sensor is more powerful" and "A larger sensor is king . "

However, the space reserved for CMOS in mobile phones is very limited. It is obviously not feasible to achieve the effect of a camera physically (increasing the sensor area) in the short term.

Even though there were breakthroughs in camera quality and quantity afterwards, they quickly became standard features in mobile phones and did not cause much splash in the industry.

At this time, some manufacturers began to focus on the advantages of mobile phones themselves , creating technologies suitable for mobile phone shooting scenarios, and explored many different paths.

In the end, it turns out that what really differentiates mobile phone images is the camera algorithm :

Automatic HDR, night scene selfie, super portrait, simulated large aperture, stage mode, 4K video...

I believe you are more or less aware of these overwhelming functions.

At this point, the imaging process of the mobile phone is no longer as direct as that of a camera. Instead, before the final picture is presented, it must undergo real-time calculation optimization by multiple processors such as ISP, DSP, and DPU.

This also brings challenges to the computing power of mobile phones, especially when facing multiple cameras, the amount of computing data increases exponentially.

If the computing power cannot support it, users will not be able to perceive the algorithm no matter how good it is.

At this time, there are two paths left for mobile phone manufacturers. One is to rely on upstream SoC platform upgrades to optimize algorithms, and the other is to actively develop compatible imaging chips to meet the implementation of their own algorithms.

One is very passive and may face incompatibility difficulties; the other is high-risk and the cost of failure is unimaginable.

Obviously, the second one is a more difficult but correct path, and it is also the method chosen by mainstream manufacturers today.

If the limitations of sensors force mobile phone manufacturers to think about the possibilities of mobile phones themselves, then the limitations of algorithms will once again force mobile phone manufacturers to return to basic thinking.

Only by understanding what users really need can we provide a more personalized algorithm experience.

Last year, the trend of self-developed imaging chips was very obvious, and vivo, the first-tier mobile imaging company, was the most representative. They took the lead in equipping their flagship X70 series with the self-developed V1 chip, and also proposed the industry benchmark - the "dual-core standard" .

The V1 chip truly solves a series of imaging technology bottlenecks recognized by the industry, including real-time night scene shooting .

With the support of vivo's self-developed chip V1, the mobile phone can run real-time ultra-high-definition night scene shooting denoising and interpolation with low power consumption.

In fact, there are many cases of solving technical bottlenecks by developing self-developed chips.

And with the recent release of the vivo V1+ chip, we have discovered that the possibilities that self-developed imaging chips bring to mobile phones are far more than just imaging itself.

What’s new in this Plus version?

The newly released V1+ chip is positioned as ASIC (application-specific integrated circuit) .

Compared with the V1 chip, which focuses more on image processing, the V1+ has improvements in both performance and functions compared to the V1.

Let’s look at the performance parameters first.

The average data throughput speed of V1+ is 8GB/s, and the theoretical maximum speed can reach 25GB/s; the power consumption is reduced by 72% (V1 is reduced by about 50%) , and the energy efficiency is increased by 3 times.

What is the concept?

Intuitively, this allows mobile phones to run more high-performance algorithms that were previously "unable to handle".

For example, the three algorithms that consume the most performance in mobile phone imaging can all be hardware-encapsulated into the V1+ chip at once:

3D real-time stereo night scene noise reduction, MEMC interpolation and AI super-resolution algorithm.

Among them, MEMC interpolation and real-time night scene noise reduction are inherited from the V1 chip. Not only can the mobile phone run real-time denoising and interpolation for ultra-high-definition night scene shooting with low power consumption, but also through further upgrades, the algorithm is directly used to turn the mobile phone into a professional-grade "night vision device".

This feature allows the phone to shoot real-time video even in extremely dark environments with an illumination of less than 1 lux (the illumination required for regular reading is approximately 300~400 lux) .

As for the AI ​​super-resolution algorithm, it is the first time that vivo has implemented it on a mobile device, making the V1+ break away from everyone's perception of "image processing".

Well, now the pressure is on the mobile game manufacturers (dog head) .

So the question is, how does the V1+ chip achieve the above functions?

On the one hand, V1+ has optimized its own architecture. For example, an SRAM is integrated on the chip to alleviate the storage wall, reduce memory access power consumption, and further improve energy efficiency.

On the other hand, the V1+ chip did not "fight alone" this time, but carried out dual-core joint debugging with the Dimensity 9000 chip.

The debugging took nearly 350 days, with a team size of more than 300 people, and the entire process generated more than 30 patents.

The images are also played with "permutations and combinations"

Looking beyond the chip, vivo has also developed more imaging algorithms this time, and "arranged and combined" them to create more interesting functions.

Looking at portraits alone, vivo said that "there are dozens of complex algorithms hidden behind every portrait photo."

The beautification of a few people is just a minor issue. Vivo even developed a "textured portrait photo" function directly for multi-person scenes, which supports processing up to 30 people at the same time.

In other words, multiple algorithms for AI face ultra-definition, three-dimensional skin color and skin quality optimization can run simultaneously on up to 30 people, and the algorithm performance (execution time, memory usage, etc.) has increased by up to 4 times compared to before optimization.

This way, you can see your face clearly even if you are standing at the edge of the photo.

In addition to portraits, vivo has also come up with features such as "horizon photo", which runs algorithms such as real-time position detection and anti-shake correction compensation at the same time. This way, you don't need to retouch the photo after taking it, and you can directly post it on WeChat Moments (dog head) :

In addition to these algorithms that make photos look "better", vivo has also used Zeiss Natural Color 2.0 to further improve the "accuracy" of photographic photos, with brightness accuracy improved by 16% and white balance improved by up to 12%.

Based on the AI ​​perception engine, vivo has developed its own "intelligent white plus black minus" and "intelligent automatic white balance" technologies. The former aims to obtain photos with proper exposure, while the latter reduces the impact of light sources on the color of objects. These were originally adjusted manually by photography "old masters".

In summary, from the perspectives of performance and functionality, vivo does not intend to benchmark "professional cameras" in terms of parameters like most mobile phone imaging products in the industry.

After all, due to limitations of hardware and other factors, matching the parameters of professional cameras is by no means the "expertise" of mobile phone cameras. In comparison, what mobile phone imaging should think about more is how to take a differentiated route.

In fact, compared with professional cameras, mobile phones, as both competitors (taking pictures) and referees (displaying) of images , should have a greater say in display.

vivo noticed this and, in addition to improving imaging performance, also improved the screen display effect.

The photo-taking and display color gamut is expanded to the Display P3 color gamut, increasing the color range by 25%. Android's first self-developed XDR Photo technology increases the brightness of highlight areas of photos by up to 350%, truly displaying the image effect on the screen.

That’s right, in addition to the lens and algorithm, vivo also actively rolls up the screen, which can be said to have taken advantage of all the possible space for mobile phone imaging and rolled it up.

Looking at the entire development history of domestic flagship phones, such desperate "involution" is not uncommon.

There is not much room left for domestic flagships

Looking back at the entire process of mobile phone technology development, "involution" seems to be a very common phenomenon.

It does not only appear in images, but is spread across every "technical domain" of the mobile phone.

From the hardware performance of mobile phones, the memory has evolved from dozens of MB to 4GB and then to 8GB, and the number of chip cores has increased. Later, the screen display has gone from 1080p to 2k and now some manufacturers have developed 4k, with higher and higher definition. Then there are all kinds of innovative functions...

Almost every time a new field is developed, manufacturers will rush in and compete to develop related technologies for fear of being left behind.

This has also become a tacitly accepted default development direction for mobile phone manufacturers:

As long as new technologies are followed up promptly enough, the parameters are sufficient, and there are enough materials, it will look like a "top-tier" mobile phone in the industry, and users will buy it.

It can be perceived that once each technology reaches the peak of maturity, the speed of "involution" will be accelerated, and the "involution" of images is nothing more than a concrete manifestation of this process.

On the one hand, the cost of developing new technologies is high, and there are few opportunities for trial and error. Once a large amount of investment is made in research and development, the consequence is likely to be no output, just like the image sensor path that stacked small pixels but eventually disappeared.

On the other hand, compared with self-developed mobile phone technology, factors such as market and sales volume are often the focus of mobile phone manufacturers.

Compared with developing more technologies on-premises or holding more patents, continuously launching new models based on mature, market-proven technologies or engaging in price wars is obviously a safer approach.

However, the result of such "involution" is that mobile phones do not have real competitiveness and may be left behind once the market changes.

For example, due to factors such as chip shortages, Apple successfully took the top spot in the Chinese mobile phone market with a 23% market share at the end of last year .

In this case, there is not much room left for domestic flagships to make changes.

Whether to continue to maintain the "involution" of parameters or to find ways to innovate on existing technologies, manufacturers each have their own ideas:

For example, launching a flagship sub-brand, or separating low-end and mid-range models from high-end models, or carving out a business to focus on flagships...

But no matter how the brand changes, as long as new technology is not truly introduced, the essential path remains the same: return to involution.

In this case, being user-demand-oriented and proactively developing new technologies may be the only way out of internal competition.

Looking back at the development history of the vivo X series, this kind of case of breaking out of "involution" through self-research is not uncommon:

Hi-Fi music phone, 2K screen, dual 2.5D curved glass screen, front soft light dual camera, under-screen fingerprint, micro-gimbal anti-shake lens...

What we can see is that they are not rolling in the industry just for the sake of rolling in the industry.

Because compared to the "bloody road" created by stacking hardware and parameters in the mobile phone market, vivo's choice is more like a win-win result for manufacturers and users - manufacturers win and users benefit.

After all, the ultimate demand for any product lies with the user. Users will know whether a product is good or not once they use it. Only by forming a closed loop between users and manufacturers can the industry develop healthily and sustainably.

Now, the V1+ chip, which is about to be used in the X80 series, may also be a new path for vivo to try to break the "involution" again.