Six technical routes for in-vehicle gesture interaction research

Zoss Automotive Research Institute released the "2022-2023 China Automotive Gesture Interaction Development Research Report", which analyzes and studies four aspects: gesture interaction technology, benchmark vehicle gesture interaction solutions, gesture interaction industry chain, and gesture interaction solution providers.

1. The installation volume of in-vehicle gesture recognition functions will increase by 315.6% year-on-year in 2022

With the iteration and upgrade of smart cockpit technology, cockpit services have gradually evolved from passive intelligence to active intelligence, and the human-computer interaction mode has shifted from a single-mode interaction mode to a multi-modal interaction. Under the influence of this trend, the in-vehicle gesture interaction function has developed rapidly. In 2022, the total number of gesture recognition (standard) installed in passenger cars in China was 427,000, a year-on-year increase of 315.6%; the installation rate was 2.1%, an increase of 1.6 percentage points from 2021.

Gesture recognition (standard) installation rate in Chinese passenger cars, 2019-2022

Source: Zoss Automotive Research "2022-2023 China Automotive Gesture Interaction Development Research Report"

In terms of brands, Changan Automobile had the highest installation rate of gesture recognition function in 2022, reaching 33.0%, an increase of 13.1 percentage points from 2021. In terms of specific model performance, in 2022, Changan Automobile had a total of 6 models including UNI-V, CS75, and UNI-K equipped with gesture recognition function as standard, 5 more than in 2021.

Top 10 Chinese passenger cars with gesture recognition (standard) installed in 2022 (by model)

Source: Zoss Automotive Research "2022-2023 China Automotive Gesture Interaction Development Research Report"

The gesture recognition function of Changan UNI-K adopts 3D ToF solution, which can realize functions such as switching songs and starting navigation. Specifically, move your palm left and right to play the previous/next song, "heart" to navigate home, and "like" to navigate to the company.

2. Gesture recognition control is extending from software to hardware and from inside the car to outside the car

With the popularization of gesture interaction technology and the continuous expansion of application scenarios, in-vehicle gesture interaction technology has also developed rapidly. At present, automobile manufacturers are actively promoting the layout of cockpit interaction functions. Gesture interaction technology has evolved from initially controlling the in-vehicle infotainment system, including answering and hanging up calls, adjusting volume, controlling navigation and other functions, to controlling the body hardware and safety systems, such as opening or closing windows/sunroofs/sunshades, closing doors, and driving vehicles. In addition, gesture control outside the car is also a technical field that major manufacturers are actively expanding. For example, WEY Mocha has enabled the driver to control the vehicle's ignition, forward, reverse, pause, and shutdown operations through gestures outside the car. In the future, gesture recognition functions will no longer be limited to drivers and passengers, but will gradually realize the recognition of the actions of passers-by outside the car, such as recognizing the command gestures of traffic police on the road or the gestures made by cyclists around the car.

Evolution of in-vehicle gesture interaction functions

Source: Zoss Automotive Research "2022-2023 China Automotive Gesture Interaction Development Research Report"

3. Six technical routes for gesture recognition

From the perspective of technical routes, gesture recognition technology mainly includes structured light, ToF, binocular imaging based on 3D cameras, millimeter wave and ultrasound based on radar, and electromyography technology based on bioelectricity.

Gesture Recognition Technology Route

Source: Zoss Automotive Research "2022-2023 China Automotive Gesture Interaction Development Research Report"

At present, the gesture sensing technology route based on 3D cameras is the mainstream technology route for in-vehicle gesture recognition. This technology route consists of a 3D camera and a control unit. The 3D camera includes a camera, infrared LED, sensor and other parts, which are used to capture hand movements, and then use the corresponding image processing algorithm to identify the type of gesture and issue relevant instructions. The technology route based on 3D cameras can be subdivided into structured light, ToF and binocular vision.

Structured Light Technology

Structured light technology is a solution that projects light with coded information onto the human body, collects the reflected structure pattern with an infrared sensor, and finally constructs a 3D model with a processing chip. It is suitable for close-range scenes within 10 meters and has the advantages of mature hardware, high recognition accuracy, and high resolution. The gesture recognition solution equipped in the Nezha S, which was launched in July 2022, is a structured light solution.

The gesture recognition sensor in the Nezha S cabin is located above the rearview mirror and can recognize 6 gestures, including: moving the palm forward and backward to adjust the transmittance of the skylight, making a "hush" gesture to mute the sound, adjusting the volume by moving the fingers clockwise/counterclockwise, opening the palm and moving it left and right to switch audio and video programs, making a "V" gesture to take selfies in the car, and liking and collecting programs.

ToF Technology

ToF technology measures distance based on the time of flight of light, and is implemented by constructing a 3D image from the underlying photosensitive element. It can obtain effective and real-time depth information within 5 meters, and is applicable to a wider range of scenarios. Whether the ambient light is strong (such as sunlight) or weak, effective depth of field information can be obtained. The gesture recognition solutions installed in mass-produced models such as the BMW iX, Ideal L9, and the Alpha S new HI version of the Polar Fox are all ToF solutions.

The gesture recognition sensor in the BMW iX cabin is located on the roof light above the central control screen and can recognize 8 gestures, including: ① Waving your hand left and right to reject a call/ignore a prompt;

② Use your index finger to answer calls/confirm prompts;

③ Rotate clockwise to increase the volume or zoom in on the navigation map;

④ Rotate counterclockwise to reduce the volume or zoom out the navigation map;

⑤ Make a fist and extend your thumb to wave left and right to adjust the previous song or the next song;

⑥ Two-finger pointing to the display can be customized;

⑦ Make a five-zero-five gesture (customizable);

⑧ Make a fist with your index finger and thumb touching each other, and then swing and pull left and right to view the images around the vehicle (the vehicle must be equipped with the Automatic Parking Assist System PLUS).

BMW iX gesture recognition function diagram

Source: Aptiv

To ensure that both drivers and passengers can use gesture recognition control, Ideal L9 has gesture recognition sensors installed in the front and rear cabins of the car. The front cabin sensor is located above the rearview mirror, while the rear cabin sensor is located above the rear entertainment screen.

The front cabin can recognize 2 gestures, including: ① Control windows/sunroof/sunshades (combined with voice interaction);

② On the playback page, make a fist and move it up and down to adjust the volume.

The rear cabin can recognize 7 gestures, including: ① Spread your five fingers and place your inner elbow on the armrest for 2 seconds to perform gesture positioning;

② Spread your five fingers and wave your hand downward to open the screen;

③ Spread your five fingers and use your palm to move the cursor;

④ Make a fist and grab the icon;

⑤ Make a fist, grab and move the image to flip the screen;

⑥ On the playback page, make a fist and move left and right to adjust the playback progress;

⑦ On the playback page, open five fingers and wave upwards to exit the current content.

Binocular stereo imaging technology

Binocular stereo imaging technology is based on the principle of parallax and is achieved by obtaining three-dimensional geometric information of objects from multiple images. This technology has low hardware requirements and does not require additional special equipment. It is a cost-effective solution. The gesture recognition solution equipped in the Mercedes-Benz EQS, which was launched in May 2022, is a binocular stereo imaging solution.

The gesture recognition sensor in the Mercedes-Benz EQS cabin is located at the roof reading light and can recognize three gestures, including: ① Make a “V” gesture to call up favorites;

② Wave your hand forward/backward under the rearview mirror to control the sunroof to open/close;

③ Wave your hand towards the inside of the car and the door will close automatically (optional four-door electric opening and closing is required).

At present, gesture recognition technologies such as radar-based millimeter waves, ultrasound, and bioelectric myoelectricity have not been widely used in the gesture recognition function in the car cabin. Compared with traditional visual gesture recognition technology, these technologies still have some limitations and challenges.

Millimeter wave radar

Millimeter-wave radar is a radio wave sensor that can accurately detect the position and movement of hands even in the presence of obstacles. In 2020, Ainstein, a US subsidiary of Muniu Technology, established a joint venture brand RADAC with ADAC Automotive of the United States, and officially released an in-vehicle gesture recognition solution based on millimeter-wave radar at the CES. The gesture recognition sensor in this solution is located above the tailgate, and the door can be opened by waving the hand left and right.