Wearable sonar tracks facial expressions using sound, not cameras

Cornell engineers have developed a new wearable device that can monitor a person's facial expressions through sonar and recreate them in a digital avatar. Removing the camera from the equation could ease privacy concerns.

The device, which the team calls EarIO, consists of an earpiece with a microphone and speakers on either side that can be connected to any regular headset. Each speaker emits a sound pulse beyond the range of human hearing toward the wearer's face, and the echo is picked up by the microphone.

As the user makes various facial expressions or speaks, the echo profile will change slightly due to the way the user's skin moves, stretches and wrinkles. A specially trained algorithm can recognize these echo profiles and quickly reconstruct the expressions on the user's face and display them on the digital avatar.

"Through the power of artificial intelligence, the algorithm discovered complex connections between muscle movements and facial expressions that are invisible to the human eye," said study co-author Li Ke. "We can use this to infer complex information that is even harder to capture -- the entire frontal face."

The team tested the EarIO system on 16 participants, running the algorithm on a regular smartphone. Sure enough, the device was able to reconstruct facial expressions just as well as a regular camera. Background noise, such as wind, talking or street noise, did not interfere with its ability to recognize faces.

The team says sonar has some advantages over using cameras. Acoustic data requires much less energy and processing power, which also means the equipment can be smaller and lighter. Cameras can also capture a lot of other personal information that users may not intend to share, so sonar could be more private.

As for what this technology might be used for, it could be a handy way to replicate your physical facial expressions on a digital avatar in games, VR, or virtual worlds.

The team said further work is still needed to tune out other distractions, such as when the user turns their head, and to simplify the system for training the AI ​​algorithms.

The research was published in the Association for Computing Machinery's Journal on Interactive, Mobile, Wearable and Ubiquitous Technologies.