Google's new patent: Depth of field effect can be achieved with a single camera

Earlier this month, the European Patent Office published a Google patent application that coincides with the technology behind Google's camera. This is achieved by using a single-lens camera, while competitors use dual-lens cameras, such as the iPhone XS Max. The magic behind the Pixel 3's camera is machine learning.   

Google notes that most competitors use stereo cameras (such as the iPhone XS Max) consisting of two or more image capture components (cameras) to simultaneously capture multiple images that can be combined in some way to create or simulate a three-dimensional stereoscopic image.      

Although stereo cameras can determine 3D information of a scene, the use of multiple image capture components (cameras) increases the overall cost and complexity of producing a stereo camera.  

Google's patent covers Pixel smartphone cameras using depth estimation technology, which can be used to estimate the depth of elements in a scene captured by a single camera.    

Google's patent FIG. 5 below depicts a simplified representation of an image capture component (camera) for capturing an image of an object; FIG. 6 depicts determining the distance between an object and a camera; and FIG. I0 is a flow chart showing that a Pixel smartphone uses machine learning to accomplish the task of creating depth, rather than using dual cameras.

      It is in block 1002 of Google's patent FIG. 10 that machine learning comes into focus. Google notes that block 1002 may involve performing machine learning-based foreground-background segmentation.

      In some examples, a neural network is used to perform foreground-background segmentation on the captured images. The neural network can analyze the image to estimate which pixels represent the main focus in the foreground and which pixels do not. 

      In some implementations, the neural network can be a convolutional neural network (CNN) with skip connections. The convolutional neural network can filter the image and then further filter the filtered image. The filtering process using the CNN can be repeated in an iterative manner. 

      Furthermore, the processing stage using a convolutional neural network in conjunction can identify and focus on a desired subject of the image (e.g., focusing on a person), including identifying pixels corresponding to the desired object (e.g., pixels corresponding to the person).

      Block 1004 may also involve estimating a depth map using the image. In particular, the depth of each point in the scene may be determined and used in order to enhance the final rendered image and blur portions of the original image. In some examples, calculating the depth may involve using stereo processing. As described above, the camera may include pixels that may be divided into sub-pixels. 

      Google's patent application is titled "Estimating Depth Using a Single Camera." It was originally filed in November 2017 and published by the European Patent Office on April 11, 2019.