Explain for you the technical advantages of continuous wave CMOS ToF camera system!

Many current products use low-resolution rangefinder type solutions to provide depth information to enhance 2D imaging. However, this approach has many limitations. For applications that can benefit from higher resolution 3D depth information, CW CMOS ToF cameras offer the highest performance solutions on the market. Table 1 provides a more detailed description of some of the system features enabled by high-resolution CW ToF sensor technology. These system features can also be applied to consumer use cases such as video background blur, facial authentication and measurement applications, as well as automotive use cases such as driver status monitoring and automated cabin configuration.

Table 1. CW Time-of-Flight System Characteristics

A depth camera is one that outputs the distance between the camera and the scene for each pixel. One technique for measuring depth is to calculate the time it takes for light to travel from the camera source to the reflecting surface and back to the camera. This travel time is often referred to as Time of Flight (ToF).

Figure 1. Continuous wave time-of-flight sensor technology overview

A ToF camera consists of several components (see Figure 1), including:

• A light source, such as a vertical cavity surface emitting laser (VCSEL) or an edge emitting laser diode, that emits light in the near infrared region. The most commonly used wavelengths are 850 nm and 940 nm. The light source is usually a diffuse source (flood illumination) that emits a beam with a certain divergence (i.e., field of illumination or FOI) to illuminate the scene in front of the camera.

• A laser driver that modulates the intensity of the light emitted by the light source.

• A sensor with an array of pixels that collects return light from a scene and outputs a value for each pixel.

• A lens that focuses the returning light onto the sensor array.

• A bandpass filter, which is co-located with the lens, is used to filter out light outside a narrow bandwidth centered on the wavelength of the light source.

• Processing algorithms that convert the raw frames output by the sensor into depth images or point clouds.

There are many ways to modulate the light in a ToF camera. A simple approach is to use continuous wave modulation, such as a square wave with a 50% duty cycle. In practice, laser waveforms are rarely perfect square waves and may look closer to sine waves. For a given optical power, a square laser waveform produces a better signal-to-noise ratio, but the presence of high-frequency harmonics can also cause deep nonlinear errors.

CW ToF cameras measure the time difference t _d between the transmitted and returned signals by estimating the phase shift ϕ = 2πft _d between the fundamental waves of the two signals . The depth can be estimated using the phase shift (ϕ) and the speed of light (c) as follows:

Where f _mod is the modulation frequency.

The clock generation circuit in the sensor controls the complementary pixel clocks, which in turn control the accumulation of photocharge in the two charge storage elements (Tap A and Tap B), and the laser modulation signal of the laser driver. The phase of the returned modulated light can be measured relative to the phase of the pixel clock (see Figure 1, right). The differential charge between Tap A and Tap B in the pixel is proportional to the intensity of the returned modulated light and the phase of the returned modulated light relative to the pixel clock.

Using the principle of zero-IF detection, multiple relative phase measurements are made between the pixel clock and the laser modulation signal. Combining these measurements determines the fundamental phase in the returning modulated light signal. Knowing this phase allows the calculation of the time it takes for light to travel from the source to the object being observed and back to the sensor pixel.

In practice, non-ideal factors such as photon shot noise, readout circuit noise, multipath interference, etc. can cause phase measurement errors. High modulation frequency can reduce the impact of these errors on depth estimation.

This can be easily understood with a simple example. Assuming the phase error is ϵ _ϕ , the phase measured by the sensor is . The depth error is:

Therefore, the depth error is inversely proportional to the modulation frequency f _mod . Figure 2 shows this graphically.

This simple formula largely explains why ToF cameras with high modulation frequencies have lower depth noise and smaller depth errors than ToF cameras with low modulation frequencies.

Figure 2. Effect of phase error on distance estimation

One disadvantage of using high modulation frequencies is that the phase wraps around faster, which means that shorter distances can be accurately measured. A common approach to addressing this limitation is to use multiple modulation frequencies that wrap around at different rates. The lowest modulation frequency allows longer distances to be accurately measured, but depth errors (noise, multipath interference, etc.) are also larger, and using higher modulation frequencies in tandem can reduce depth errors. Figure 3 shows an example scheme using three different modulation frequencies. The final depth is estimated by weighting the unwrapped phase estimates at different modulation frequencies, assigning larger weights to higher modulation frequencies.

Figure 3. Multi-frequency phase unwrapping

If the optimal value is chosen for the weight of each frequency, the depth noise is inversely proportional to the root mean square (rms) of the modulation frequency chosen in the system. For a constant depth noise budget, increasing the modulation frequency can reduce the integration time or illumination power.

There are many system characteristics to consider when developing a high-performance ToF camera, some of which are briefly described here.

➤ Image sensor

The image sensor is a key component of a ToF camera. Most non-ideal factors related to depth estimation, such as bias, depth noise, and multipath artifacts, are less effective when the average modulation frequency of the system is increased. Therefore, the sensor must have high demodulation contrast (the ability to distinguish photoelectrons between Tap A and Tap B) at high modulation frequencies (hundreds of MHz). The sensor also needs to have high quantum efficiency (QE) at near-infrared wavelengths (such as 850 nm and 940 nm), thereby reducing the optical power required to generate photoelectrons in the pixel. Finally, low read noise enables detection of lower return signals (distant or low-reflectivity objects), thereby helping to increase the dynamic range of the camera.

➤Light intensity

The laser driver modulates the light source (such as a VCSEL) at a high modulation frequency. For a given optical power, to maximize the amount of useful signal at the pixel, the optical waveform needs to have fast rise and fall times and clean edges. The combination of the laser, laser driver, and PCB layout in the illumination subsystem is critical to achieving this. Some calibration work is also required to find the optimal optical power and duty cycle settings to maximize the amplitude of the fundamental in the Fourier transform of the modulated waveform. Finally, the optical power also needs to be transmitted in a safe manner, and some safety mechanisms should be built in at the laser driver and system level to ensure that Class 1 eye safety limits are always met.

➤Optical components

Optics play a key role in ToF cameras. ToF cameras have some unique characteristics, so they have some special requirements in terms of optics. First, the illumination area of ​​the light source should match the field of view of the lens for best efficiency. It is also important that the lens itself has a high aperture (low f/#) for better light collection efficiency. Large apertures can lead to trade-offs with other factors such as vignetting, shallow depth of field, and lens design complexity. Lens designs with low chief ray angles can also help reduce the bandwidth of the bandpass filter, thereby improving ambient light rejection and improving outdoor performance. The optical subsystem should also be optimized for the desired operating wavelength (such as anti-reflection coatings, bandpass filter design, lens design) to maximize throughput efficiency and minimize stray light. There are also many mechanical requirements to ensure that the optical alignment is within the desired tolerance range for the end application.

➤Power Management

Power management is also critical in high-performance 3D ToF camera module designs. Laser modulation and pixel modulation produce short bursts of high peak current, which imposes some constraints on the power management solution. Several features of the sensor integrated circuit (IC) can help reduce the peak power consumption of the imager. Power management techniques can also be applied at the system level to help reduce the requirements for the power source (such as battery or USB). The main analog power supply of the ToF imager usually requires a regulator with good transient response and low noise.

Figure 4. Optical system architecture

➤Deep processing algorithm

Finally, another major part of the system-level design is the depth processing algorithm. The ToF image sensor outputs raw pixel data from which phase information needs to be extracted. This operation requires multiple steps, including noise filtering and phase unwrapping. The output of the phase unwrapping module is a measurement of the distance that the light from the laser travels to the scene and back to the pixel, often referred to as range or radial distance.

The radial distance is generally converted into a point cloud, which represents the actual coordinate (X, Y, Z) information of a specific pixel. Typically, the final application uses only the Z image map (depth map) instead of the entire point cloud. Converting the radial distance to a point cloud requires knowledge of the lens intrinsic characteristics and distortion parameters. These parameters are estimated during the geometric calibration of the camera module. The depth processing algorithm can also output other information, such as an active brightness image (amplitude of the returned laser signal), a passive 2D IR image, and a confidence level, which can all be used in the final application. Depth processing can be performed in the camera module itself, or in a host processor elsewhere in the system.

An overview of the different system-level components covered in this article is shown in Table 2. These topics will be discussed in detail in future articles.

Table 2. System-level components of a 3D time-of-flight camera

Continuous wave time-of-flight cameras are a powerful solution that can provide high depth accuracy for applications that require high-quality 3D information. There are many factors to consider to ensure the best performance level is achieved. Factors such as modulation frequency, demodulation contrast, quantum efficiency, and read noise determine the performance of the image sensor. Other factors are system-level considerations, including illumination subsystems, optical design, power management, and depth processing algorithms. All of these system-level components are critical to achieving the highest accuracy 3D ToF camera system.