iFlytek comprehensively deploys multi-modal interaction in intelligent cockpits

In November 2020, iFlyTek's new product launch conference for iFlyAuto was held in Guangzhou. iFlytek's multimodal interaction capabilities attracted the attention of the guests as soon as they were introduced.

At the 4th China Automotive Enterprise Innovation Conference held in Shanghai in March 2021, Mr. Zhao Yi, Deputy General Manager of iFlytek Intelligent Vehicle, attended the conference and gave a keynote speech on "iFlytek's Multi-Mode Perception of Automotive Cockpits", which triggered heated discussions among experts and scholars present.

In April 2021, the first SAIC Zero-beam SOA Platform Developer Conference was grandly held in Shanghai. iFlytek showcased three innovative technology products in the SOA artificial intelligence exhibition area: the smart cockpit multi-mode perception system demo, the in-vehicle multi-language voice assistant, and the smart audio management system hardware, which attracted widespread attention from the guests and media.

This series of actions all demonstrate iFlytek's strategic blueprint of using multimodal interaction capabilities as a starting point to improve smart cockpit interaction solutions and realize overtaking in the smart car race.

Smart cockpit multi-mode interaction is in the ascendant

At present, the global automotive industry is facing a major change that has not been seen in a century. Various technologies such as 5G, big data, artificial intelligence, and cloud computing are accelerating the reconstruction of the industry landscape, driving the automobile from a "hardware-based" industrial product to a personalized smart terminal that can self-learn, self-evolve, and self-grow, and a different kind of life that carefully accompanies every car owner. Being able to truly provide car owners with more personalized, more intelligent, and more emotional services will become the core competitiveness of future automotive products, which will inevitably bring a series of new challenges.

From the perspective of the automotive industry chain, for a car to truly understand people and provide precise services based on the environment, it needs to have strong interaction and scene realization capabilities. The current voice assistants in cars are more functional and do not have much emotional expression. To build dialogue logic for users and form a dialogue style for cars, it is necessary to combine services and interactive interfaces, and to consider the design of the connection between car services and people from the interaction level.

From the consumer's perspective, consumers no longer want a car to be just a means of transportation, but a personalized mobile space that meets entertainment and office needs. At the same time, it needs to have the ability to "perceive" and "understand" people, so as to provide a comfortable and intelligent experience during driving.

The core of improving user experience is to achieve more accurate perception. Machines need to understand human intentions very accurately to achieve more accurate interaction, which requires the improvement and upgrading of interaction capabilities. In the planning of the domestic and foreign automotive industries in the past five years, it can be clearly seen that multimodal interaction in smart cockpits has become a key planning product. iFlytek is undoubtedly at the forefront in this regard.

iFLYTEK has achieved a full-link technology upgrade of "listening, speaking, watching and displaying"

The first step to realize the mass production of in-vehicle multi-modal interaction is to increase the popularity of in-vehicle voice. Only when voice interaction becomes a strong demand of users, it will be meaningful to combine it with other modes.

iFlytek has entered the automotive OEM market for ten years. In the past ten years, intelligent voice has always been a banner of iFlytek's smart cars, from the initial command word voice recognition system to the later microphone array noise reduction technology, AIUI conversational human-computer interaction technology... iFlytek's smart cars have gradually built a full-loop automotive intelligent voice interaction core technology including voice wake-up, voice recognition, natural language understanding, speech synthesis, voiceprint recognition, sound source localization, etc.

Based on the powerful voice capability, iFlytek's multimodal human-computer interaction and driving safety protection technology solution (hereinafter referred to as iFlytek's multimodal interaction solution) innovatively adds visual capabilities:

1. Scenario services are a channel to personalize user experience, which naturally requires vehicles to have the ability to identify drivers. iFlytek's infrared heterogeneous FaceID technology, trained with a large-scale face database, can achieve liveness detection and senseless login, and based on this, realize personalized services, allowing cars to better understand users and provide more thoughtful services.

2. Just as the combination of voice and vision is the most important way for people to interact with each other, visual interaction has better directionality. To further upgrade vehicles to become true "artificial intelligence", it is necessary to use a combination of auditory and visual perception technologies. iFlytek's automotive-grade eye tracking technology allows users to wake up the interaction by just looking at it, and everything can be controlled by the gaze, which greatly reduces the interaction steps and avoids the problems caused by voice interaction.

The innovative addition of lip sound fusion voice technology has three core capabilities: lip sound wake-up-free, lip sound detection, and lip sound enhancement. You can speak directly and even whispers can be heard clearly, making in-car interactions faster, more accurate, and warmer.

At the same time, screen gaze perception and eye position perception can adjust the brightness of the car screen, make auxiliary devices such as head-up display easier to use, and enhance the driving experience.

3. At present, the driver monitoring system related to vision has become a rigid demand from the regulatory level: Euro-NCAP requires DMS to be one of the necessary conditions for a five-star rating, and China is currently following up on the formulation of relevant standards and regulations. iFlytek's perception technology based on the degree of eye and mouth opening and closing can determine the user's fatigue level; combined with head posture estimation and tracking, it can accurately determine whether the line of sight has left the road; for user behaviors such as making phone calls and smoking, it can achieve corresponding vehicle control and reminders. Multiple information fusion is processed to make vehicle driving safer.

In terms of multi-mode output, iFLYTEK's multi-modal interaction solution will eventually be presented in a more technological way, including image replication, lip synchronization, motion display, and ambient lighting rhythm.

The full-link technology upgrade of "listening, speaking, seeing and displaying" has built a complete multimodal system, which can realize the integrated processing of multiple information such as voice, image, and living body in the entire vehicle use cycle of getting on-driving-getting off the vehicle, and understand the passenger information more actively and deeply, so as to actively care for them, push relevant content/services, and change vehicle settings, thereby bringing a subversive interactive experience.

In the next decade, the new era characteristics of market globalization, industry intelligence, product technology, and brand rejuvenation will accelerate the reconstruction of the industry value system. iFlytek will strive to play the role of explorer, organizer, and leader in promoting the innovative development of the smart car industry, and hand in the "iFlytek answer sheet" for the construction of smart cockpit interaction and Internet of Vehicles systems.