Why not use the computational photography camera that greatly upgrades the iPhone's photo-taking capabilities?

      At the launch of the iPhone 11 series, Apple Senior Vice President Philip W. Schiller introduced the concept of computational photography for the first time when introducing the imaging system of the iPhone 11 Pro series. This concept also became known to the public for the first time.

　　In fact, the concept of computational photography is not new. It first appeared in a public paper in 1994, and it was determined that in-camera HDR synthesis, panoramic photos and simulated bokeh all belong to the category of computational photography. But at that time, the mainstream carrier of photos was still film, digital cameras were just starting out, and there were no cameras on mobile phones.

　　Decades later, the medium for recording images changed from film to digital, mobile phones were equipped with cameras, and computational photography emerged from theory and gradually became a major trend.

　　However, this trend has little to do with cameras. Camera manufacturers are still improving pixels, continuous shooting speed and video capabilities step by step, and seem to be turning a deaf ear to computational photography. The photos taken (straight out) are still mediocre and are gradually being "surpassed" by smartphones.

　　On the contrary, the computing power of smartphone chips is getting stronger and stronger, the scope of involvement of AI, algorithms, and machine learning is getting wider and wider, there are more and more ways to interpret images, and ultimately the photos processed by a series of "algorithms" are becoming more and more beautiful.

　　Nowadays, many people prefer to use mobile phones to record and share when they go out, while cameras are becoming less and less common. This is also reflected in the market performance of the two. The smartphone market is growing strongly, while the camera market has been shrinking year after year, and even DC (compact cameras) are gradually disappearing.

　　At this point, someone may ask, since the photos taken with smartphones look so good, why don’t traditional camera manufacturers follow the trend of computational photography and consider improving the look of photos taken directly?

　　Is it because the camera’s computing power is not enough and it can’t calculate it?

　　Let's start with the 'core' of this issue.

　　The core of a mobile phone is the SoC, which integrates the CPU, GPU, ISP, NPU and baseband, etc. It allows you to make calls, take photos, watch videos, play games, surf the Internet, etc., and also directly determines the performance of the phone.

　　The core component of a camera is the image sensor (CMOS), which is similar to a mobile phone except for the size of the component, and is used for imaging and light sensing. In addition, the central processing chip that controls the entire camera system is called the image processor.

　　Take Sony's BIONZ X image processor as an example (exclusively used in the α7 series). It includes SoC and ISP chips, but does not integrate the ISP into the SoC. The advantage is that Sony can increase the number of ISP chips based on the performance requirements of CMOS (the BIONZ X of the α7RIII is equipped with dual ISPs). The disadvantage is that the degree of integration is not as high as that of mobile phones.

　　The role of SoC in BIONZ X is similar to that of mobile phones, controlling the operation interface and camera functions, and the performance requirements are not high. The "data" collected by the image sensor is subjected to Bayer transformation, mosaic demosaicing, noise reduction, sharpening and other operations, mostly relying on ISP, and finally converting the data collected by CMOS into the real-time view of the camera. In this process, the camera's ISP does not involve the calculation process, but only treats the photos as products on the assembly line and processes them uniformly.

　　As the number of pixels, continuous shooting speed and video performance of current cameras continue to increase, the camera's image processor has high requirements for the speed and throughput of image processing. The amount of single data is very large. Without involving "computing", the processing power of the camera's image processor far exceeds that of the current smartphone ISP.

　　But when it comes to computational photography, or AI capabilities, things are a little different. The imaging process of a smartphone is somewhat similar to that of a camera, but before the final image is presented, it still requires ISP and DSP calculations, real-time adjustments, and optimizations, especially after multi-camera systems became mainstream, the amount of computing data on mobile phones has increased exponentially.

　　After the iPhone 11 Pro series launched the multi-camera system, the multi-camera system can switch smoothly and seamlessly because of the huge data processing capabilities of the two new machine learning accelerators in the A13 Bionic, which reaches one trillion times per second. Such high-frequency and efficient data processing capabilities can handle the huge amount of data generated by the three cameras.

　　The camera's image processor mostly pre-processes the raw data and has almost no calculation process, while the mobile phone SoC includes data acquisition pre-processing and subsequent calculation processes. The two focus on different directions.

　　Targeting different groups, the results of market segmentation

　　Mobile phone computational photography is developing rapidly. The root cause is that the size of the mobile phone's image sensor (CMOS) is too small. With current technology, if you want to physically surpass or approach the camera, you can only optimize the algorithm and piece together the look and feel, for example, automatic HDR, super night scene, simulated large aperture, magic sky change and other functions.

　　However, it is still difficult to achieve "personalized" intervention in the interpretation of these algorithms, such as to what extent the filter is added, to what extent the HDR highlight dark part is retained, etc. However, for mobile phones for the general public, it is more in line with the market positioning and population positioning of mobile phones to allow most people to take good photos as much as possible.

　　Since the invention of the camera, it has been an absolute "tool". For the sake of efficiency, the appearance, control, function, etc. will all compromise with efficiency. For a smaller professional group, it will naturally be more in line with their needs. Cameras will record color depth, color, light and other information as much as possible to allow users to make a wider range of post-adjustments. Whether the direct output looks good is not in their needs.

Sandofsky

　　For most people who have no photography background, getting a good-looking photo is far more important than getting a photo rich in information. For camera manufacturers targeting professional fields, improving the color depth of RAW recording is more in line with market positioning than improving the JPG direct output effect.

　　However, things are not so absolute, and cameras are also trying to change. Fuji has been committed to the direct output effect of the camera, and introduced "film simulation", which uses different algorithms to make the photos more flavorful and more beautiful. However, this process does not go through scene calculations, but requires users to choose by themselves. This is somewhat similar to some film simulation apps on mobile phones, and does not involve the so-called "computational photography".

　　Is AI the general direction of cameras in the later stage?

　　In the field of photography, post-processing is an indispensable step. On the one hand, post-processing software can make full use of the rich information recorded in the RAW format. On the other hand, it can also use the high performance and computing power of the PC to quickly process photos.

　　Unlike camera manufacturers, almost all mainstream professional post-production software has begun to focus on AI, emphasizing the processing capabilities of AI.

　　In recent versions of Adobe's Photoshop, automatic recognition functions have been added to operations such as cutout, repair, and skin smoothing, making the operation more and more brainless and the effect more and more precise. As early as 2018, Pixelmator Pro, a photo editing software on the Mac platform, began to use Apple's Core ML machine learning to recognize images, so as to adjust colors, cut out images, select, and even use the ML machine learning engine when compressing output.

▲ Pixelmator Pro 2.0's image editing supports machine learning engines. Image from: Pixelmator

　　As mentioned above, due to the limitation of chip AI computing power and the problem of a niche market, camera manufacturers have hardly put any effort into computational photography. However, the explosion of AI in post-production software can be regarded as making up for the shortcomings of cameras in computational photography.

　　Even with the AI ​​of post-production software, cameras still cannot get rid of the traditional process. Cameras record and software processes. This process is still cumbersome for the general public. For professional photographers, the intervention of AI in post-production software can indeed reduce the workload and make the originally complicated operations such as cutting out images much easier, but it still cannot change the photo processing (creation) process of the traditional photography industry, which is completely different from mobile phones.

▲ Global digital camera shipments in September 2020 are far less than in 2018. Image from: CIPA

　　According to CIPA data, the camera market is gradually shrinking, while the mobile phone market is growing. The "computational photography" that has become a trend on smartphones will not change the direction of cameras becoming more professional, nor will it reverse the gradual shrinking of the camera market.

　　In other words, even if cameras now have "computational photography" capabilities close to those of smartphones, can they save the "declining" camera market? The answer is of course no. To give an extreme example, if direct output is feasible, Fujifilm cameras will have the largest market share. In fact, the top spot in mirrorless cameras is now occupied by Sony, whose direct output is not good.