NPUs & Neural Engines

Every modern smartphone, laptop, and many IoT devices now contain a dedicated chip for AI inference: a Neural Processing Unit (NPU) - also called a Neural Engine, AI Engine, or AI Accelerator depending on the vendor. These chips do one thing well: run matrix multiplications (the core operation of neural networks) at high throughput while consuming a fraction of the power that a GPU would require for the same workload.

Why a Dedicated Chip?

Neural network inference is dominated by one operation: multiply-accumulate (MAC) - multiply two numbers and add the result to a running total. This happens billions of times per inference. A general-purpose CPU handles this poorly (it is optimized for diverse, sequential workloads). A GPU does it well but consumes too much power for mobile use. An NPU is designed from the ground up for high-throughput, low-power MAC operations:

TOPS = Trillion Operations Per Second. The NPU achieves 5–10× the AI throughput of the GPU at 20–30% of the power.

Apple Neural Engine

Apple's Neural Engine (ANE) has been present in iPhones since the A11 Bionic (iPhone X, 2017). It is tightly integrated with the rest of Apple Silicon:

• M4 ANE: 38 TOPS, handles Apple Intelligence on-device model inference
• Runs Core ML models - Apple's on-device ML framework
• Tight integration with GPU and CPU via unified memory - no data copy overhead
• Not directly programmable by third parties - accessed via Core ML, Vision, Natural Language frameworks
• Apple uses the ANE for: Face ID, Siri voice recognition, camera computational photography, and Apple Intelligence

Qualcomm Hexagon NPU

Qualcomm's Hexagon NPU is inside most premium Android phones (Samsung Galaxy, OnePlus, Xiaomi flagships):

• Snapdragon 8 Gen 3 Hexagon: 75 TOPS - the highest mobile NPU throughput as of 2024
• Runs models via Qualcomm AI Engine Direct SDK, ONNX Runtime, and TensorFlow Lite
• Supports INT4, INT8, and FP16 quantized models
• Qualcomm "AI Hub" provides pre-optimized model versions (Llama 3, Phi-3, Gemma, Stable Diffusion) certified for Snapdragon deployment

Google Tensor - Pixel's Custom Silicon

Google designs the Tensor chip exclusively for Pixel phones, with a custom NPU optimized for Gemini Nano inference and Google's specific ML workloads:

• Tensor G4 (Pixel 9 series): optimized for Gemini Nano Nano-X, live speech recognition, and real-time translation
• Tighter integration with Android's AI stack than third-party chips
• Enables Google's "Live Translate" and "Interpreter" features that run entirely on-device

Windows AI PCs - NPUs Come to Laptops

Microsoft's "AI PC" initiative requires a minimum 40 TOPS NPU in any PC that carries the "Copilot+" branding:

• AMD Ryzen AI 300 series: up to 50 TOPS NPU
• Intel Core Ultra (Lunar Lake): up to 48 TOPS NPU
• Qualcomm Snapdragon X Elite: 45 TOPS NPU - the first to ship Copilot+ features
• Enables: Windows Studio Effects (background blur, eye contact correction), Cocreator in Paint, Recall (screenshot search), and local Phi-3 inference

Accessing NPUs as a Developer

Developers typically access NPUs through framework abstractions, not directly:

• iOS/macOS: Core ML - Apple's framework automatically routes operations to ANE, GPU, or CPU based on availability and efficiency
• Android: Android Neural Networks API (NNAPI) or MediaPipe - routes to NPU on capable devices
• Windows: ONNX Runtime with DirectML - automatically targets NPU on supported hardware
• Cross-platform: TensorFlow Lite delegates (NPU delegate, NNAPI delegate) route specific operations to the NPU

The key practical point: if you export your model to the right format (Core ML, ONNX, TFLite) and use the right framework, the NPU is used automatically without any explicit NPU programming on your part.