Driver fatigue monitoring system based on multi-layer deep learning framework and motion analysis

Abstract: The latest developments in the automotive industry have aroused the interest of researchers in the monitoring of fatigue driving, with the intention of developing an effective driver monitoring system that can detect abnormal psychophysical states in a timely manner and reduce traffic accidents caused by fatigue driving. Many literatures now focus on the study of physiological signals in particular, and obtain information about heart movement by measuring heart rate variability (HRV). In fact, HRV is also an effective indicator for assessing physiological stress because it can provide information related to the activity of the cardiovascular system controlled by the autonomic nervous system. This paper aims to extract facial feature points, analyze the subtle skin movements caused by blood pressure, and then reconstruct the photoplethysmogram (PPG) signal in a robust way. It is concluded that the PPG signal detected by the sensor is strongly correlated with the PPG signal reconstructed using facial feature points, and we have obtained evidence to support this conclusion from the experimental results.

1 Introduction

Drowsiness is a physiological state characterized by a reduced level of consciousness and difficulty staying awake. According to the National Safety Council, the proportion of fatal accidents caused by drowsy driving in the United States is increasing significantly. Therefore, it is important to develop an effective warning system that can detect in advance that the driver's physiological condition is not suitable for driving. It is reported that studies have shown that heart rate variability (HRV) is associated with the driver's attention level. Specifically, heart rate variability is an important indicator of an individual's physiological adaptability and behavioral flexibility. The method of evaluating cardiac activity is to measure blood pressure using PPG signals, and then evaluate heart rate variability. Specifically, the PPG signal consists of the peak value of vascular volume representing successive cardiac cycles. The PPG detection method is to use an LED light source to illuminate different parts of the skin and then use a photodiode to evaluate the intensity of light reflection. Although physiological signals allow us to monitor drowsiness, recent research has focused on using computer vision technology to evaluate driver fatigue. Although it is certainly challenging to develop a face detection system in an automotive environment, there are still many methods that use cameras to determine the blink rate and thus evaluate fatigue. Different from other studies, our method focuses on using computer vision technology to detect and extract facial landmarks, and defines the time series of facial landmarks by analyzing the changes in pixel intensity of previously recorded video sequences. More specifically, the basic principle of our method is also to reveal the subtle facial movements caused by blood pressure changes through "video zooming". The purpose of this study is to construct a PPG signal by defining the time series of facial landmarks instead of using sensors.

The rest of this paper is organized as follows: Section II presents related research results; Section III gives an overview of PPG signals and introduces our pipeline based on long short-term memory and convolutional neural networks. Section IV explains the experimental procedure. Finally, Section V discusses the advantages of our approach and future research directions.

2 Related research

In the papers and works published in the past, most of them detected driver drowsiness through physiological signals and achieved high detection accuracy. In fact, many studies have shown that driver fatigue monitoring solutions based only on computer vision technology may not be effective, especially visual methods that focus on analyzing traffic signs, which often fail when the road conditions are bad.

Some researchers have published a research result on photoplethysmography (PPG) detection. The authors used low-power wireless PPG sensors to achieve good detection results. Another method is that the authors use low-frequency and high-frequency PPG signals detected on fingers and earlobes to evaluate fatigue. The research results cited in this article mainly evaluate HRV signals by studying ECG and PPG signals. However, the method cited in this article has high requirements for computing performance and requires expensive detection equipment to be integrated in the car. Although the integrated sensor is not necessarily a direct measurement tool, in order to accurately obtain physiological signals, the driver still needs to place his hand or other parts of the body (such as earlobes or fingers) on the sensor, which is a limitation for the promotion of application in cars. This article takes a different approach and proposes an innovative framework. The basic principle is to capture the driver's facial image, collect facial feature points, and reconstruct the PPG signal to evaluate the HRV signal and fatigue level.

3 Background and pipeline scheme

As mentioned above, we proposed an innovative method for monitoring driver drowsiness without using sensors to obtain PPG signals. Some scholars have described how video magnification methods can reveal facial motion changes by magnifying ordinary video images, because blood pressure changes during successive cardiac cycles can cause color changes in different parts of the skin. Studies have shown that autonomic nervous system activity can regulate certain physiological processes, such as blood pressure and respiratory rate, which can be indirectly measured by evaluating heart rate variability signals, because heart rate variability signals change during physiological stress, extreme fatigue, and drowsiness.

Evaluating HRV heart rate variability requires the use of biofeedback tools or software, as well as high-quality sensors to detect ECG signals, and powerful processors to manage large amounts of data. ECG signals are the traditional method for evaluating heart rate variability, but this method has certain drawbacks in use. Although the detection effect is good, subtle movements of the human body during the data acquisition (data sampling) process will cause some noise and artifacts in the signal. In order to overcome the problems of ECG, the industry has proposed that PPG signals are a reliable solution. The ability to detect changes in blood volume enables PPG to effectively detect subtle skin movements that are difficult to observe with the naked eye. In particular, by analyzing PPG signals, we can define heart rate changes within a specific period of time and show whether both branches of the autonomic nervous system (parasympathetic and sympathetic) are working properly. Generally, a small HRV value indicates a constant heart rate interval, while a large HRV value indicates an abnormal heart rate interval. A very normal heart rhythm and subtle changes in heart rate can determine whether attention is reduced due to chronic physiological stress. However, there is no standard HRV value because HRV values ​​vary from person to person.

With this in mind, we developed a driver drowsiness monitoring system using a combination of Long Short Term Memory (LSTM) neural networks and Convolutional Neural Networks (CNN). The proposed pipeline represents an advancement in cardiac motion assessment methods because it uses a low frame rate (25fps) camera to detect and extract key feature points in face images and analyze pixel changes in each video frame. Specifically, LSTM is a powerful solution for assessing hidden nonlinear correlations between data.

Specifically, the output of the LSTM pipeline is the predicted time series of facial landmarks after integrating the raw PPG target data detected by the sensor.

Furthermore, the accurate classification of the CNN model indicates that the LSTM prediction is effective in determining the level of attention of the car driver.

4 Experiments

A total of 71 subjects participated in our LSTM-CNN pipeline run. More specifically, the dataset is PPG samples from patients/drivers of different genders, ages (between 20 and 70 years old), and pathologies. In this case, we not only collected data from healthy subjects, but also collected data from patients with diseases such as hypertension and diabetes. Considering the differences between the two sleepiness states, PPG signal samples for each of the two sleepiness states were measured separately. Specifically, we simulated two scenarios of full wakefulness and sleepiness confirmed by synchronized ECG sampling signals, where Beta and Alpha waveforms confirmed the brain activity states when awake and sleepy, respectively. The simulation interval for each scenario was set to 5 minutes to ensure that the system had sufficient time to complete preliminary calibration and real-time continuous learning. At the same time, we used a low frame rate (25fps) full HD camera to record a video of the driver's face. As mentioned above, we first used the dlib library based on the Kazemi and Sullivan machine learning algorithm to detect the previously recorded video frames and extract the facial feature points of the face. Then, we calculated the pixel intensity associated with each feature point and the change in pixel intensity per frame, determined the time series of facial feature points, and input them into the LSTM neural network.

4.1 CNN Pipeline

This section describes the CNN model architecture used in the experiments in more detail. The proposed CNN architecture provides strong evidence to validate the LSTM prediction results. Specifically, our CNN model is able to track and learn the facial expressions of car drivers, thereby improving the level of drowsiness detection. To train the model, we set the batch size to 32 and the initial learning rate to 0.0001. In addition, we used 32 neurons in the hidden layer and 2 output neurons in the binary classification.