Why choose i.MX 8M Plus? Because these two new features are so awesome!

Generally speaking, whether it’s using a voice assistant through a smartphone or smart speaker, or a social media app or even how a phone can group photos of a certain person together, machine learning deployed in the cloud is the key technology behind all of this.

However, these use cases all rely on machine learning running in servers in the cloud. The real challenge that NXP solves is machine learning at the edge, such as running machine learning and inference locally on edge processors such as the i.MX 8M Plus .

Running machine learning inference at the edge means that applications will continue to run even if network access is compromised—critical for applications such as surveillance or smart home alarm centers, or when operating in remote areas without network access. Edge computing also allows for much lower latency when making decisions than if data had to be sent to a server for processing and then the results sent back. For example, low latency is important when performing visual inspections on an industrial factory floor and needing to decide whether to accept or reject a rapidly moving product.

Another key advantage of machine learning at the edge is user privacy. The collected data, such as voice communications and commands, faces, videos, and images captured by edge devices, is processed at the edge and saved locally at the edge. The information is not sent to the cloud for processing, where it may be recorded and tracked. The user's privacy is not affected, and it is up to the individual to decide whether to share personal information in the cloud.

Today, given the need for edge machine learning, the question has slowly shifted from whether edge machine learning is needed to what level of performance is needed for edge machine learning. One metric for measuring the performance of machine learning accelerators is the number of operations per second (usually 8-bit integer multiplication or addition), commonly referred to as TOPS, or trillion operations per second. This is a preliminary benchmark, as overall system performance will also depend on many other factors, but it is one of the widely cited machine learning metrics.

Generally speaking, performing full speech recognition (not just keyword retrieval) on the edge takes about 1-2TOPS (depending on the algorithm, more if you want to understand what the user is saying, not just convert from speech to text). Performing object detection (using algorithms such as Yolov3) at 60fps also takes about 2-3TOPS. This makes machine learning acceleration (such as the 2.3TOPS of the i.MX 8M Plus) an advantage for such applications.

Any camera-based system requires an ISP, and sometimes it may be built into the camera module, or integrated into the application processor and may be hidden from the user. The ISP typically performs many types of image enhancement, with its primary purpose being to convert the raw image sensor's per-pixel output monochrome components into RGB or YUV images that are more commonly used elsewhere in the system.

Application processors without ISPs work well in vision-based systems when the camera input comes from the cloud or a webcam (usually connected to the application processor via Ethernet or USB). In such cases, the distance between the camera and the application processor may be far, even up to 100 meters. The camera itself has an ISP and processor built in to convert the image sensor information and encode the video stream, which is then sent over the network.

Application processors without an ISP are also suitable for relatively low-resolution cameras. When the resolution is 1 megapixel or lower, the camera's image sensor usually has an embedded ISP and can output RGB or YUV images to the application processor, which means that an ISP is not required in the processor.

However, when the resolution is 2 megapixels (1080p) or higher, the embedded ISP of most image sensors is no longer competent and needs to rely on an ISP elsewhere in the system - either a standalone ISP chip (which can be implemented, but increases the power and cost of the system) or an ISP integrated in the application processor.

NXP chose the i.MX 8M Plus to provide high-quality images and enable image optimization, especially for resolutions of 2 megapixels and above.

By integrating a 2.3TOPS machine learning accelerator and ISP, the i.MX 8M Plus application processor will become a key player in edge embedded vision systems and is widely used in smart homes, smart buildings, smart cities, and industrial IoT applications.

With an embedded ISP, the application processor can be used to create high image quality optimized systems that interface directly with local image sensors, and even feed that image data to the latest machine learning algorithms, all offloaded to local edge-side machine learning accelerators.

Thanks to the i.MX 8M Plus application processor's optimized architecture for machine learning and vision systems, edge device developers can take a different approach and become market leaders and innovation drivers like NXP. They can benefit from powerful machine learning capabilities and high-definition camera systems to enable devices to see more clearly and farther. New innovation opportunities have emerged in the embedded product landscape.