ISM330IS sensor with built-in intelligent sensor processing unit brings more powerful artificial intelligence to edge devices

At this year's SENSOR + TEST 2022 conference in Nuremberg, Germany, attendees had the opportunity to see the ISM330IS, the first sensor with a built-in intelligent sensor processing unit (ISPU). STMicroelectronics released this technology in early 2022. Simply put, the ISPU is a C-programmable embedded digital signal processor (DSP) capable of running machine learning and deep learning algorithms. As such, it is the next evolution of edge AI, or what ST calls the "Onlife Era". The ISM330IS has a single-precision computing floating-point unit, a first for motion sensors.

What challenges did ST overcome from an idea to a new generation of sensors?

At the start of this research, ST published a paper on the feasibility of integrating a machine learning core into an inertial sensor. In the past, the role of sensors was to collect data, and all computing tasks had to be done on the microcontroller. The reason behind this architecture was relatively simple. Inertial sensors are small, low-power devices. Adding a powerful processor would not only violate these design constraints, but also bring huge system integration and manufacturing challenges. Although the DSP and accelerometer and gyroscope are integrated into one module, ST has not compromised processing performance, memory capacity, and sensor accuracy.

What’s so special about the first sensor with machine learning at its core?

This paper published by ST in 2018 is groundbreaking because it solves the market promotion problem of LSM6DSOX, the first inertial sensor with eight parallel decision trees running machine learning algorithms. Running applications locally while consuming very low power has gone from impossible to possible. After the product was launched, new applications began to emerge, especially after the module was integrated into the SensorTile.box. For example, the baby crying detector developed with it can alert the driver when a baby is forgotten in the car. Similarly, two projects at University College London developed automated standing/sitting monitoring and a more convenient digital stethoscope based on this, which received good market reviews.

What other sensors have machine learning cores?

The LSM6DSOX also marks the emergence of a new developer community. ST has made the machine learning core library available on GitHub and improved the accessibility of the Unico GUI software tool to help more programmers who want to take advantage of the machine learning core of the LSM6DSOX. In addition, we have released sensors with more powerful performance. The LSM6DSRX has better performance for more demanding applications such as virtual reality headsets. ST also launched the LSM6DSV16X, which has an enhanced machine learning core and better performance-power ratio for systems with stricter power constraints. Therefore, ST sensors with machine learning cores have accelerated the arrival of the next era of automation to a certain extent, and the ISM330IS has opened an important new chapter in this saga.

From new processing cores to new applications

What is ISPU made of?

The ISPU of the ISM330IS provides 8 KB of data RAM and 32 KB of application RAM, runs at 10 MHz, has a 32-bit RISC Harvard architecture, a four-stage pipeline, a floating-point unit, and a 16-bit length instruction set optimized for neural network processing. In addition, although the processor only takes four clock cycles to issue an interrupt request (the Arm Cortex core usually takes 15 clock cycles), it can also process 16-bit multiplication operations in one clock cycle. It uses SPI or I2C to communicate with the main control MCU, and developers only need to load the C code into the volatile memory of the ISPU when the main processor starts.

The FPU enables applications to run inference algorithms more flexibly on edge devices. Once the conditions are met, the program will issue an interrupt request to the microcontroller. Similarly, compared with the previous generation core, the architecture enables the ISPU to improve performance while maintaining power consumption at the microwatt level. Therefore, compared with the decision tree of the previous machine learning core, the new product is a major leap and a more efficient system. In addition, despite its powerful computing power, the ISM330IS still fits into the market-standard 3 mm x 2.5 mm x 0.83 mm LGA package. Therefore, with the new sensor, designers do not need to significantly modify the PCB layout.

How does ISM330IS stand out?

Machine learning applications are becoming more and more popular. When many people think of machine learning, they think of high storage capacity, cloud computing servers, or highly parallel GPU architectures. There are many programs that have very high computing performance requirements, which makes them incompatible with edge processing. However, some companies also know that not all deep learning systems require this level of computing power. Therefore, applications that run inference algorithms locally on mobile products such as mobile phones or industrial equipment are becoming more and more popular. Image recognition, anomaly detection, and predictive maintenance all require reliable AI performance under low power conditions. Similarly, home wireless security system cameras can use artificial intelligence to recognize faces or pets, and the ISPU core can provide intelligence for the mobile system's always-on display.

The ISM330IS is a new solution to the challenge of performance and power consumption , as its high-performance mode consumes only 0.59 mA, while the ISM330DHCX consumes up to 1.5 mA under the same conditions. The higher power consumption of the ISM330DHCX is partly due to the module's integration of features such as a more powerful gyroscope. However, these figures also show the optimization of the new product and the energy efficiency of the processor core. In fact, low-power microcontrollers rarely have FPUs because they usually require a lot of power, but the ISM330IS successfully controls power consumption to a level suitable for battery-powered systems.