Can AI “roll up” the final answer to the smart cockpit?

At the end of 2022, ChatGPT emerged, which not only ignited an unprecedented storm of technological public opinion, but also made all walks of life face an AI industry revolution.

The automotive industry is no exception. Many car companies have embraced AI and actively deployed and embraced big models. New forces such as Wei, Xiaoli, and Li Auto have successively announced their own big model plans, and suppliers such as SenseTime and Haomo Zhixing have also launched their own big models. It can be said that big models have become a trend in cars.

In 2023, Baidu released the Chinese version of ChatGPT - Wenxin Yiyan. Many car companies such as Great Wall, Geely, and Lantu followed suit and announced their joining of the Wenxin Yiyan ecosystem. Behind this is the car companies' sensitive sense of applying AI big models to cars.

In the highly homogenized field of smart cockpits, AI big models have the best opportunity to be installed on board.

From in-vehicle voice systems to intelligent interactions, and then to cabin-driver integration, the AI ​​big model can enable the smart cockpit to evolve again and make it even smarter.

01

Smarter in-car interaction

The system in the car that is easiest to introduce AI big model technology into should be the voice interaction system, which is not only easy to access but also can achieve relatively good results.

AI's advanced natural language processing technology can analyze complex language structures and contexts, enabling the voice system to understand the user's natural language instructions, making voice interaction smoother and more natural. Different from traditional voice assistants, AI can maintain the context information of the conversation, support multiple rounds of conversation, and provide a more coherent interactive experience.

Taking the new force that embraces big models most actively as an example, in June 2023, Ideal released its self-developed cognitive big model Mind GPT, making the in-car voice assistant Ideal classmate smarter.

Through the high knowledge reserve of the large model, Mind GPT can improve the conversation and comprehension abilities of Ideal Classmates, and enable them to have logical reasoning ability, making the question-and-answer experience smoother. Mind GPT's built-in memory network also gives Ideal Classmates the ability to remember personalized preferences and habits based on historical conversations, so as to better understand users.

Not only that, as AI large models gradually move from single modality to multi-modality, the in-vehicle interactive system can also transform from single voice interaction to interaction that combines multiple sensory inputs such as voice, vision and touch.

In terms of tactile interaction, the multimodal large model can use pressure sensors to detect touch force and position, use capacitance changes to detect touch events, and provide tactile feedback through devices such as vibration motors to enhance the intuitiveness of the interaction.

In terms of visual interaction, the multimodal large model can use computer vision technology to process images and videos captured by the camera, identify facial features, gaze direction and gestures, and then authenticate the driver through facial recognition, analyze the driver's gaze direction, provide corresponding interactive feedback or auxiliary functions, and capture and analyze gesture movements through the camera to achieve contactless control.

If we use structured light, ToF (Time of Flight) and other technologies to obtain visual information in three-dimensional space, the multimodal large model can also achieve more accurate gesture and facial recognition.

Face, gesture, vein, and even eye tracking... the in-vehicle interaction equipped with a multimodal large model is like using black technology.

Take Ideal L7 as an example. It is equipped with a 3D depth sensor, such as Melexis' MLX75027, which can recognize the user's gestures and allow the user to control the navigation, air conditioning, etc. in the car through gestures. As long as the index finger points to the window, it can automatically rise and fall.

There is also the Weipai Blue Mountain DHT PHEV, which can combine voice and head posture, capture the driver's head movements through the on-board camera, and give a confirmatory or negative response.

02

Smarter cabin-driver integration

In the second half of intelligence, as the automotive electronic and electrical architecture develops towards central integration, cabin-driver integration has undoubtedly become a recognized development trend of electric vehicles.

Qian Qian, deputy director of the Central Research Institute of Hangsheng Group's Technology Center, believes that "cross-domain integration can solve the problem of efficient use of computing power through integrated design; different departments can also collaborate on processes to improve development efficiency. This is reflected in the improvement of product strength in vehicle products, with faster data transmission between systems, higher system efficiency and response speed; the system's adaptability in complex scenarios will be improved, and the user's human-machine co-driving experience will be improved; and the platform scalability of vehicle models can also be improved with the help of cabin-driver integration."

Under this trend, large AI models can play even greater advantages.

The cockpit has transformed from the original mechanical cockpit to a smart space with touch technology and a large screen. Under the trend of cabin-driver integration, many smart driving functions will be delivered to users by the smart cockpit. The smart cockpit is taking on more and more usage scenarios.

Under normal circumstances, AI can collect the driver's driving habits and preferences, and provide personalized settings during intelligent driving, such as customized driving modes, driving routes, etc.

Moreover, while driving, AI can also use multimodal interaction to collect sensor data, transmit information about vehicle performance and road conditions to the driver in the form of voice or visualization, and take corresponding actions based on the driver's instructions.

Taking SenseTime’s DriveAGI as an example, users can not only ask DriveAGI to explain its decision-making process by asking questions, but also control autonomous driving behavior through voice or gesture commands.

For example, in the future, when driving autonomously, the navigation system may instruct the vehicle to turn around at the next intersection to reach the destination, but the driver knows that there is a shortcut ahead where he can turn directly. In this case, he only needs to say "turn left directly" to the system, and the system will execute this instruction based on the current road conditions.

At present, ADAS and smart cockpits are not deeply integrated. In the future, integrating the two may become a new trend, which can create many new functions and interaction modes, thereby improving driving experience and safety.

In the Banyan 3, which was recently released at NIO's Technology Day, we can also feel that AI is increasingly affecting the integration of cabin and driver. For example, if you say "find a good parking lot" to NOMI, it will actively analyze the walking distance between the parking lot and the destination, recommend underground parking lots when the weather is hot, and even provide information such as whether the parking space is a mechanical parking space.

Not only that, NOMI can also call the vehicle-side recorder to infer and summarize eight scenarios including people approaching, cars approaching, people staying, and door opening.

At the 8th ATC Automotive Human-Computer Interaction Technology Summit in 2024, Ecarx also demonstrated their large-scale cabin-driver integrated model architecture based on the central computing platform, including super-human AI travel assistant and humanoid driving technology. These technologies not only enhance the user's personalized experience, but also realize multi-dimensional and multi-modal human-like decision-making based on central computing.

Huawei's new generation Hongmeng cockpit also has the AI ​​large model Qianwu engine installed, which can not only achieve millimeter-level precise perception in the cabin, adaptive adjustment of the driver's rearview mirror, steering wheel, HUD height, etc., but also supports skeleton-level human perception of the entire cabin and multi-modal fusion control of the vehicle, and can control the sunshades, doors, and air-conditioning wind direction with a wave of the hand.

It can be seen that AI big model empowering smart cockpits has become an industry trend, and the era of smart cockpit 4.0 is coming rapidly. AI will make smart cockpits smarter and create a multi-modal and more personalized interactive experience.