ExecuTorch Beta Released to Accelerate Generative AI Development on the Edge on Arm Platforms

Click on the Arm Community above to follow us

Arm is working with the PyTorch team at Meta to advance the launch of the new ExecuTorch Beta, which aims to provide artificial intelligence (AI) and machine learning (ML) capabilities to billions of edge devices and millions of developers around the world, thereby ensuring that the true potential of AI can be used by the widest range of devices and developers.

Arm computing platforms are ubiquitous, powering many edge devices around the world, and ExecuTorch is a PyTorch native deployment framework designed for deploying AI models on mobile and edge devices. The close collaboration between the two enables developers to enable smaller and more optimized models, including the new Llama 3.2 1B and 3B quantized models. These new models can reduce memory usage, improve accuracy, enhance performance, and provide portability, making them ideal for generative AI applications on small devices, such as virtual chatbots, text summarization, and AI assistants.

Developers can seamlessly integrate the new quantization models into their applications without additional modifications or optimizations, saving time and resources, allowing them to quickly develop and deploy more intelligent AI applications at scale on a wide range of Arm devices.

In the mobile space, Arm’s collaboration with ExecuTorch means that many generative AI applications, such as virtual chatbots, text generation and summarization, real-time voice and virtual assistants, can run at higher performance on devices equipped with Arm CPUs. This achievement is due to KleidiAI, which introduces a microkernel optimized for 4-bit quantization and integrates it into ExecuTorch through XNNPACK, so that when running 4-bit quantized LLMs on Arm computing platforms, the execution of AI workloads is seamlessly accelerated. For example, with the integration of KleidiAI, the execution speed of the Llama 3.2 1B quantization model pre-filling phase can be increased by 20%, allowing text generation on some Arm-based mobile devices to exceed 400 tokens per second. This means that end users will get a faster and more responsive AI experience from their mobile devices.

In the field of IoT, ExecuTorch will improve the real-time processing capabilities of edge AI applications, including smart home appliances, wearable devices, and automated retail systems. This means that IoT devices and applications can respond to environmental changes at millisecond speeds, which is critical to ensuring safety and functional availability.

ExecuTorch runs on Arm ^® Cortex ^® -A CPUs and Ethos™-U NPUs to accelerate the development and deployment of edge AI applications. In fact, by integrating ExecuTorch with the Arm Corstone™-320 reference platform (also available as an emulated fixed virtual platform (FVP)), Arm Ethos-U85 NPU drivers and compiler support into a single package, developers can start developing edge AI applications months before the platform is available.