How to use AI to leverage sensor data

By Kaustubh Gandhi, Software Product Manager, Bosch Sensortec

Artificial intelligence (AI) is currently revolutionizing all aspects of society. For example, by combining the advantages of data mining and deep learning, AI can now be used to analyze large amounts of data from various sources, identify various patterns, provide interactive understanding, and make intelligent predictions.

An example of this innovative development is the application of AI to data generated by sensors, especially data collected by smartphones and other consumer devices. Motion sensor data, along with other information such as GPS addresses, provide a large and diverse set of data. So the question is: “How can AI be used to fully leverage these synergies?”

Sports data analysis  

An illustrative real-world application would be to analyze usage data to determine the user's activities during each time period, whether sitting, walking, running, or sleeping.

In this case, the benefits of smart products are self-evident:

1. Increase customer lifetime value Increasing user engagement can reduce customer churn. 

2. More competitive product positioning The next generation of smart products meets the growing expectations of consumers. 

3. Create real value for end users
Accurate detection and analysis of indoor movements can enable sensitive navigation functions, monitor health risks, and improve the efficiency of devices. In -depth understanding of the actual usage scenarios of multiple smartphones and wearable platforms will greatly help product designers understand users' repetitive habits and behaviors, such as determining the right battery size or the right time to push notifications.  

Smartphone manufacturers are showing interest in AI capabilities, which highlights the importance of identifying simple daily activities, such as steps, which will inevitably develop into more in-depth analysis, such as sports activities. For popular sports like football, product designers will not only focus on athletes, but will also provide convenience for a wider range of people, such as coaches, fans and even large companies such as broadcasters and sportswear designers. These companies will benefit from deep data analysis to accurately quantify, improve and predict sports performance.

Data acquisition and preprocessing

After identifying this business opportunity, the next logical step is to think about how to efficiently collect these huge data sets.

For example, in the case of activity tracking, raw data is collected through axial motion sensors such as accelerometers and gyroscopes in smartphones, wearables, and other portable devices. These devices acquire motion data on three axes (x, y, z) in a completely covert manner, i.e., continuously tracking and evaluating activity in a way that is convenient for user applications.

Training the model

For supervised learning of artificial intelligence, labeled data is needed to train the "model" so that the classification engine can use this model to classify actual user behavior. For example, we collect motion data from test users who are running or walking, and provide this information to the model to help it learn.

Since this is essentially a one-time approach, simple apps and camera systems can accomplish the task of “labeling” users. Our experience shows that as the number of samples increases, the human error rate in classification decreases. Therefore, it makes more sense to obtain a larger sample set from a limited number of users rather than a smaller sample set from a large number of users. 

It is not enough to just have raw sensor data. We observed that achieving highly accurate classification requires careful identification of features, i.e. the system needs to be told which features or activities are important to distinguish between sequences. The process of manual learning is iterative, and in the pre-processing stage it is not clear which features are most important. Therefore, the device must make some guesses based on expert knowledge that may affect the classification accuracy.

For activity recognition, indicative features can include “filtered signals” such as body acceleration (raw acceleration data from sensors) or “derived signals” such as Fast Fourier Transform (FFT) values ​​or standard deviation calculations.

For example, the Machine Learning Database at the University of California, Irvine (UCI) created a dataset that defined 561 features based on six basic activities performed by 30 volunteers: standing, sitting, lying down, walking, descending stairs, and ascending stairs.

Pattern recognition and classification

After collecting the raw motion data, we need to apply machine learning techniques to classify and analyze it. The machine learning techniques available to us range from logistic regression to neural networks.

Support vector machines (SVMs) are one such learning model used in artificial intelligence. Physical activities such as walking involve sequences of multiple movements, and since support vector machines are good at classifying sequences, they are a logical choice for classifying activities.

Support vector machines are simple to use, train, scale, and predict, so multiple sample collection experiments can be easily set up in parallel for nonlinear classification of complex real-life datasets. Support vector machines also come in a variety of different size and performance optimizations.

After deciding on a technology, we had to choose a software library for the support vector machine. The open source library LibSVM is a good choice, it is very stable and well documented, supports multi-class classification, and provides extensions for all major developer platforms from MATLAB to Android.

The Challenge of Continuous Classification

In practice, users move around and the devices they use need to be classified in real time to identify activities. In order to minimize product costs, we need to balance the costs of transmission, storage, and processing without affecting the results, that is, the quality of information.

Assuming we can afford the data transmission costs, all data can be stored and processed in the cloud. In fact, this will bring huge data charges to users. Of course, the user's device must be connected to the Internet, and the cost of wireless network, Bluetooth or 4G module will inevitably further increase the cost of the device.

To make matters worse, 3G network access is often suboptimal in non-urban areas, such as when hiking, biking, or swimming. This reliance on the cloud for large amounts of data transfers would slow updates and require regular synchronization, greatly offsetting the actual benefits of AI motion analysis. In contrast, processing these operations solely on the device's main processor would significantly increase power consumption and reduce execution cycles for other applications. Similarly, storing all data on the device would increase storage costs.

Squaring the circle

To resolve these conflicting issues, we can follow four principles:

1. Split – Split feature processing from the execution of the classification engine.

2. Reduce - Intelligently select the features required for accurate activity recognition to reduce storage and processing requirements.

3. Use – The sensors used must be able to acquire data with low power consumption, perform sensor fusion (combining data from multiple sensors), and be able to pre-process features for continuous execution.

4. Retention - Retain a model of system-supporting data that can determine user activity.

By splitting feature processing from the execution of the classification engine, the processor that interfaces with the accelerometer and gyroscope sensors can be much smaller. This effectively avoids the need to continuously transfer real-time data blocks to a more powerful processor. Feature processing such as high-speed Fourier transforms used to transform time domain signals into frequency domain signals would require a low-power fused processor to perform floating-point operations.

Furthermore, in the real world, individual sensors have physical limitations and their outputs deviate over time, for example due to offsets and nonlinear scaling caused by welding and temperature. To compensate for such irregularities, sensor fusion is required, along with fast, inline, and automatic calibration.

Figure 1: Functional flow of activity classification (Source: Bosch Sensortec)

此外，所选择的数据捕获速率可以显著影响所需的计算和传输量。通常来说，50Hz采样率对于正常的人类活动就足够了。但在对快速移动的活动或运动进行分析时，需要200 Hz的采样率。同样地，为了取得更快的响应时间，可以安装2 kHz单独加速计来确定用户目的。

为了迎接这些挑战，低功耗或者应用特定传感器集线器可以显著降低分类引擎所需的CPU周期。比如Bosch Sensortec的BHI160和BNO055两个产品就是这种传感器集线器。相关软件可直接以不同的传感器数据速率直接生成融合后的传感器输出。

The initial choice of features to process subsequently greatly impacts the size of the trained model, the amount of data, and the computational power required to train and perform inline predictions. Therefore, the selection of features required to classify and distinguish a specific activity is a critical decision and can also be a significant business advantage.

Looking back at the UCI Machine Learning Database we mentioned above, with its full dataset of 561 features, a model trained with the default LibSVM kernel achieved a test accuracy of 91.84% for activity classification. However, after training and feature ranking, selecting the most important 19 features was enough to achieve a test accuracy of 85.38% for activity classification. Upon closer inspection of the ranking, we found that the most relevant features were the frequency domain transform and the mean, maximum, and minimum values ​​of the sliding window acceleration raw data. Interestingly, none of these features can be achieved through preprocessing alone, and sensor fusion is necessary to ensure that the data is sufficiently reliable and therefore particularly useful for classification.

in conclusion

In summary, technology has now reached the point where advanced AI can be run on portable devices to analyze data from motion sensors. These modern sensors operate at low power, while sensor fusion and software partitioning significantly improve the efficiency and feasibility of the entire system, while also greatly simplifying application development.

To complement the sensor infrastructure, we leverage open source libraries and best practices to optimize feature extraction and classification.

Providing users with truly personalized experiences has become a reality, and through artificial intelligence, systems can use data collected by sensors in smartphones, wearables and other portable devices to provide people with more in-depth functions. In the coming years, a range of devices and solutions that are still unimaginable today will be further developed. Artificial intelligence and sensors open up a new world of exciting opportunities for designers and users.

来源：UCIhttp://archive.ics.uci.edu/ml/datasets/Smartphone-Based+Recognition+of+Human+Activities+and+Postural+Transitions

About Bosch Sensortec

Bosch Sensortec GmbH is a wholly owned subsidiary of Robert Bosch GmbH, developing and providing customized MEMS sensors and solutions for smartphones, tablets, wearable devices and IoT products. The product portfolio includes 3-axis accelerometers, gyroscopes and geomagnetic sensors, integrated 6-axis and 9-axis sensors, environmental sensors, and a comprehensive software portfolio. Since its establishment in 2005, Bosch Sensortec has become a leader in MEMS technology in the above markets. Bosch has been a pioneer and global market leader in the field of MEMS sensors since 1995, and the number of MEMS sensors sold to date has exceeded 8 billion. One out of every two smartphones in the world uses a Bosch Sensortec sensor.