Edge computing AI chips drive the development of the next generation of IoT

AI-based machine learning techniques are already moving beyond cloud-based data centers as processing of vital IoT sensor data moves closer to where the data is created.

The move will be enabled by new chips equipped with artificial intelligence (AI), including embedded microcontrollers, which have smaller memory and power requirements than GPUs (graphics processing units), FPGAs (field programmable gate arrays) and other specialized IC types, and can be connected to the cloud.

It is in these clouds that machine learning and related neural network applications have exploded, but the rise of the Internet of Things has created a big data impact that also requires edge-based machine learning.

Now, cloud providers, Internet of Things (IoT) platform makers, and other companies see the benefits of processing data at the edge before handing it off to the cloud for analysis.

Making AI decisions at the edge can reduce latency and make real-time responses to sensor data more feasible. But what we call “edge AI” comes in many forms. How can we power it with the next generation of IoT and output high-quality, actionable data?

Edge computing workload growth

Edge-based machine learning could drive significant growth in the AI ​​in IoT market, which Mordor Intelligence estimates will grow at a compound annual growth rate of 27.3% by 2026.

This is supported by 2020 research from the Eclipse Foundation IoT Group, which found that 30% of the most commonly cited edge computing workloads for IoT were for AI.

For many applications, replicating an always-on server that enables parallel machine learning on the cloud is not feasible. IoT edge cases that benefit from local processing are numerous and increasingly being monitored through a variety of operational use cases. For example, a processor could monitor events triggered by changes in a pressure gauge on an oil rig, an anomaly detected on a remote power line, or video footage captured in a factory.

The last scenario is the most widely used one. The application of AI to analyze image data at the edge has proven to be very meaningful. However, there are many complex processing requirements for event processing using data collected by IoT devices.

The value of edge computing

Cloud-based IoT analytics are here to stay, said Steve Conway, senior consultant at Hyperion Research. However, there is a natural lag in moving data in and out of the cloud, and the round trip takes time.

"Electromagnetic waves travel at best at the speed of light," Conway quipped.

In addition to device and board-level implementations, this hierarchy includes IoT gateways and data centers in manufacturing that extend the architecture available for next-generation IoT system development.

Saurabh Mishra, senior manager of product marketing for SAS's IoT and Edge division, said edge AI architecture is another generational shift in the focus of data processing, and a critical shift in the long run.

“The idea is to centralize your data. You can do it for certain industries and certain use cases,” he said.

“It’s not really possible to effectively and cost-effectively move it to the cloud for analytics,” Mishra said. “SAS has created a proven edge IoT reference architecture on which customers can build AI and analytics applications. Striking a balance between cloud and edge AI will be an essential requirement.”

Finding the balance starts with considering the amount of data needed to run machine learning models, said Frédéric Desbiens, program manager for IoT and edge computing at the Eclipse Foundation. That's what the new intelligent processors are responsible for.

“Edge AI accelerators can process data locally before sending it elsewhere. However, this requires you to consider the functional requirements, including the required software stack and storage,” Desbiens said.

Edge AI Chips

The rise of cloud-based machine learning has been fueled by the rise of high-memory-bandwidth GPUs, a success that has caught the attention of other chipmakers.

First came in-house processors dedicated to AI, followed by hyperscale cloud processors from Google, AWS, and Microsoft.

The AI ​​chip war has now been joined by companies including AMD, Intel, Qualcomm and ARM.

At the same time, mainstream suppliers of embedded microprocessors and system-on-chips such as Maxim Integrated, NXP, Silicon Labs, STMicroelectronics, etc. began to work on adding AI capabilities to the edge.

Today, IoT and edge processing demands have attracted AI chip startups, including EdgeQ, Graphcore, Hailo, Mythic, etc. Steve Conway of Hyperion emphasized that the main obstacles for edge AI chips include available memory, power consumption, and cost.

"Embedded processors are very important because energy consumption is very important," Conway said. "GPUs and CPUs are not small, and GPUs in particular consume a lot of energy."

Making neural networks fit for the edge

Data movement is a significant power factor, suggests Kris Ardis, executive director of the microcontroller and software algorithm business at Maxim Integrated, which recently released the MAX78000, a device that pairs a low-power controller with a neural network processor that can run on battery-powered IoT devices.

“If you can do the computation at the very edge, you can save bandwidth and communication power. The challenge is taking a neural network and making it fit on an edge processor,” Ardis said.

He said a single IoT device based on the chip could power an IoT gateway, which could also play an important role in aggregating data from devices and further filtering data that could potentially flow to the cloud.

Other semiconductor equipment makers are also adapting to a trend that moves computing closer to where the data is. They are part of an effort to extend capabilities for developers even as their hardware grows.

Bill Pearson, vice president of Intel's Internet of Things division, acknowledged that there was a time when "the CPU was the answer to everything." Trends like edge AI now overshadow that.

He used the term "XPU" to refer to a variety of chip types that support different uses. However, he added that this diversity should be supported by a single software application programming interface (API).

To help software developers, Intel recently released the 2021.2 version of the OpenVINO toolkit for reasoning on edge systems. It provides a common development environment for Intel components including CPUs, GPUs, and Movidius vision processing units. Pearson said Intel also provides DevCloud for edge software to predict the performance of neural network reasoning on different Intel hardware.

The same is true for NVIDIA GPUs.

“The industry has to make it easier for people who are not AI experts to do their jobs,” said Justin Boitano, vice president and general manager of enterprise and edge computing at NVIDIA.

This could take the form of the NVIDIA Jetson, which includes a low-power ARM processor. Named after the 60s sci-fi cartoon series, the Jetson is designed to provide GPU-accelerated parallel processing for mobile embedded systems.

Recently, to simplify the development of vision systems, NVIDIA launched Jetson JetPack 4.5, which includes the first production version of its Vision Programming Interface (VPI).

Over time, Boitano said, AI development chores at the edge will be handled more by IT departments and less by AI researchers with a deep understanding of machine learning.

The rise of Tiny ML

In the past, it was cumbersome to migrate machine learning methods from the cloud with unlimited computing power to constrained edge devices, but now new software technologies are being applied to enable compact edge AI while reducing the workload of developers.

In fact, the industry has already seen the rise of “Tiny ML” approaches that achieve functionality with lower power consumption and use limited memory while enabling inference operations.

A variety of machine learning tools have emerged to reduce edge processing requirements, including Apache MXNet, Edge Impulse’s EON, Facebook’s Glow, Foghorn Lightning Edge ML, Google TensorFlow Lite, Microsoft ELL, OctoML’s Octomizer, and others.

Downsizing neural network processing is the main goal here, and there are multiple implementations of the technology. These include quantization, binarization and pruning, said Sastry Malladi, CTO of Foghorn, a software platform vendor that supports multiple edge and on-premises implementations.

Quantization of neural network processing focuses on the use of low-bitwidth mathematics, while binarization, in turn, is used to reduce the complexity of the computation and pruning is used to reduce the number of neural nodes that must be processed.

Malladi acknowledged that this is a daunting task for most developers, especially across a range of hardware. He said the effort behind Foghorn’s Lightning platform is designed to abstract away the complexity of machine learning at the edge.

For example, the goal is to develop software that uses a drag-and-drop interface rather than application programming interfaces and software development kits, which are less intuitive and require more coding knowledge.

Simplifying embedded machine learning development Platform maker Edge Impulse is also focusing on making it easy to develop software that runs on multiple types of Edge AI hardware.

According to Zach Shelby, CE, the maturation of machine learning will eventually mean the miniaturization of some models.

“The direction of research was toward increasingly complex and larger models,” Shelby said. “But as machine learning reached prime time, people started to care about efficiency again, so Tiny ML came along.”

There has to be software that can run on existing IoT infrastructure while supporting new hardware, he said. Shelby went on to say that the Edge Impulse tool allows for cloud-based modeling of algorithms and events on available hardware so that users can try out different options before making a selection.

Facing new challenges

On the edge, computer vision has become an important use case for AI, especially in the form of deep learning, which employs multi-layered neural networks and unsupervised techniques to achieve results in image pattern recognition.