The Edge AI Spectrum
AI does not only run in hyperscale data centers. It runs on the phone in your pocket, inside your web browser, on industrial sensors, and on microcontrollers smaller than a postage stamp. Understanding where AI can run - and what the trade-offs are at each point - is essential for building AI systems that are practical, private, and efficient.
The Full Spectrum
Cloud ◄──────────────────────────────────────────────────► Deep Edge Hyperscale On-Premise Desktop Mobile/Tablet IoT/Embedded Data Center Server Workstation Phone Microcontroller ───────────────────────────────────────────────────────────────────────────── H100/B200 A100/L40S RTX 4090/M4 Max NPU/GPU Cortex-M, ESP32 100s GB VRAM 40-80GB VRAM 24-192GB RAM 8-16GB RAM 256KB-2MB RAM GPT-4, Claude 70B models 7-70B models 1-7B models Tiny (<1M params) Network req. Network opt. Network opt. Network opt. No network needed
Cloud AI - Maximum Capability
The left end of the spectrum: hyperscale data centers running the largest frontier models. This is what you access via API (Claude, GPT-4, Gemini Ultra). Characteristics:
- Capability: Maximum - 100B–trillion parameter models with multimodal capabilities
- Cost: Per-token billing; expensive at scale
- Latency: Network round-trip adds 100–2000ms
- Privacy: Data leaves the device and is processed by a third party
- Availability: Requires internet connection
On-Premise - Control Without Cloud
Enterprise servers or workstations running large open-weight models (Llama 3 70B, Mixtral, Falcon). Used when data privacy requirements prohibit sending data to external APIs, or when usage volumes make per-token costs prohibitive:
- Typically 1–4× A100 or L40S GPUs
- Serves 7B–70B parameter models
- No ongoing API costs after hardware purchase
- Used by hospitals, legal firms, government agencies, and enterprises with strict data governance
Desktop / Developer Machine - Local Experimentation
Consumer GPUs (RTX 4090, AMD 7900 XTX) or Apple Silicon (M4 Max with unified memory) running quantized open-weight models via Ollama, LM Studio, or Foundry Local. This is the "Local AI" landscape already covered elsewhere in this site:
- 4B–70B models with 4-bit quantization
- Practical for development, testing, and privacy-sensitive personal workflows
- Apple M4 Max (128GB unified memory) can run 70B models comfortably
Mobile / Tablet - The Billions-Scale Edge
The most commercially significant edge tier: billions of phones and tablets running 1–7B parameter models on dedicated Neural Processing Units (NPUs). This is where Apple Intelligence, Google Gemini Nano, and Qualcomm's AI platform play:
- Apple Neural Engine (ANE): up to 38 TOPS (trillion operations per second) on M4 chips
- Qualcomm Hexagon NPU: up to 75 TOPS on Snapdragon 8 Gen 3
- Google Tensor G4: custom NPU optimized for Gemini Nano inference
- 1B–7B models run comfortably; some 13B models with aggressive quantization
- Zero network latency for on-device tasks; complete privacy
IoT / Embedded - The Deep Edge
Microcontrollers (STM32, ESP32, Arduino) and small single-board computers (Raspberry Pi, NVIDIA Jetson Nano) running tiny ML models with kilobytes of RAM. This is TinyML territory:
- Models under 1MB - sometimes under 100KB
- Tasks: keyword detection, anomaly detection, gesture recognition, image classification
- No internet required - complete local operation
- Ultra-low power: run on batteries for months
- Real-time response: inference in under 10ms
Choosing the Right Tier
The right deployment tier depends on four factors:
- Task complexity - simple classification → edge; complex reasoning → cloud
- Privacy requirements - sensitive data → on-device or on-premise
- Latency requirements - real-time response → edge or local
- Scale and cost - billions of low-complexity requests → edge is dramatically cheaper
The emerging pattern: hybrid routing. Simple tasks run on-device; complex tasks escalate to cloud. Apple Intelligence is the most polished example of this architecture deployed at consumer scale.