New system analyzes camera data in real time to detect not only occupants’ facial features but also their postures

Is the driver tired or asleep? The camera inside the car can monitor it, especially in self-driving vehicles, where the camera is not only important but also required by law. According to foreign media reports, the Fraunhofer Institute for Optronics, System Technologies and Image Exploitation (IOSB) in Germany is the first in the world to be able to use image data to draw conclusions about the driver's activities and analyze how fast the driver can control the vehicle.

Not only body posture but also activity can be detected (Image source: IOSB)

In autonomous driving, the vehicle needs to decide its own behavior, whether to turn, brake, or accelerate. However, before the vehicle completely does not need a driver, semi-autonomous driving will only provide some assistance to the driver, giving the driver greater freedom. In semi-autonomous driving, if you are in a construction area on a highway or turning from a highway to a city street, you still need to hand over control of the vehicle to the driver. Therefore, the vehicle must have two levels of intelligence: 1. It must be able to understand traffic conditions; 2. It must be able to monitor the situation inside the car and interact with the driver. What is the driver doing? How fast can the driver control the vehicle? Although driver monitoring systems have appeared, such systems use almost no camera image data and can only detect driver fatigue.

Artificial intelligence technology detects what the driver is doing

Researchers at IOSB are looking at the interaction between drivers and vehicles and trying to bridge the gap. "Our technology is not only able to recognize facial features, but also the current posture of the driver and passengers," said Michael Voit, head of the research team. "We can then determine from these postures what the driver and passengers are doing."

At the heart of the research are machine learning algorithms and processes, such as artificial intelligence (AI). Algorithms analyze camera data in real time to determine whether the driver is talking on the phone, playing with his children, or looking at a passenger's phone. As a result, IOSB technology goes beyond image recognition and is also able to interpret various activities based on specific contexts. First, the researchers trained the system by manually annotating a large number of camera photos: Where are people's hands, feet and shoulders? Where are mobile phones, books or other objects? Then, the researchers evaluated the algorithm using new images and corrected or verified the results.

The system extracts an image of the driver or passenger to form a digital skeleton, a kind of stick figure that recreates the posture of a human body. It can therefore use the movement of the skeleton, aided by the objects it recognizes, to infer the activities of the driver and passengers. "Thus, the algorithm can recognize if someone is sleeping or looking at the road, how distracted they are, and how long it takes to refocus on the road," explains Voit. The system supports both traditional video cameras and infrared cameras that can see in the dark, as well as 3D cameras that can measure the distance of objects from the camera. In addition, the system gives designers freedom to install cameras in the car.

Driving simulator (Image source: IOSB)

The researchers have been investigating issues related to detecting movement inside cars in numerous research projects, working with well-known car manufacturers such as Audi and Volkswagen as well as automotive suppliers such as Bosch and Continental. Such projects are also supported by the German Federal Ministry of Education and Research (BMBF), the German Federal Ministry for Economic Affairs and Energy (BMWi) or the German Federal Ministry of Transport and Digital Infrastructure (BMVI). "We are not only able to detect the movement of the driver, but also of all passengers in the front and rear seats of the vehicle. The technology is already ready for pilot production and we are in contact with companies that want to use our technology. We use our own driving simulator for development, and industry customers can benefit from such development projects. Simulating traffic situations is the basis for collecting relevant driving and behavioral data, and the various sensor technologies in the simulator mean that the behavior of all passengers can be studied," confirms Voit.

Data protection is a priority

In terms of data protection and security, the researchers made it a top priority from the outset. Pascal Birnstill, senior scientist at IOSB, said: "The camera data is analyzed in real time, not saved, not sent outside the car, and no personalized mode is required, so no personal data is collected." Therefore, the technology respects privacy from the beginning of its development and complies with strict EU regulations and a high level of data protection awareness.

Wide range of applications, even beyond activity detection

A new EU regulation emphasizes the importance of activity detection, and vehicles equipped with autonomous driving technology must be equipped with driver monitoring technology regardless of the degree of automation of the vehicle. With IOSB technology, automakers can not only meet this requirement, but also realize various visions about autonomous driving. For example: voice recognition used for communication between cars and people is rapidly approaching the limit of development. The voice command "stop here" itself is meaningless, but combined with body gesture recognition and activity detection, the system can determine which parking space the current user is pointing to. In addition, the system also helps to solve the safety issues of driverless cars. Although it is still the driver's responsibility to ensure that all occupants comply with safety regulations and fasten their seat belts, it will soon become the job of driverless cars, such as self-driving taxis or buses.