On-Device vs Cloud Decision

One of the most consequential architectural decisions in any AI product is where inference runs: on the user's device, on your server, or some combination. There is no universally correct answer - the right choice depends on your specific requirements across privacy, latency, capability, cost, and connectivity.

Decision Framework

Is the data privacy-sensitive?
  (medical, legal, financial, personal communications)
       │
       YES ──▶ Strong preference for on-device or self-hosted
       │
       NO
       │
Is real-time response required? (< 200ms)
       │
       YES ──▶ On-device required (cloud adds 200ms+ network)
       │
       NO
       │
Does the task require complex reasoning?
  (multi-step analysis, long-form generation,
   novel task generalization, multilingual)
       │
       YES ──▶ Cloud (frontier models significantly better)
       │
       NO
       │
Is the task narrow and well-defined?
  (classification, extraction, summarization,
   routing, keyword detection)
       │
       YES ──▶ On-device SLM likely sufficient
       │
       NO
       │
What is the usage scale and cost sensitivity?
  High volume + cost-sensitive ──▶ On-device amortizes cost
  Low volume + capability-critical ──▶ Cloud

Privacy - The Clearest Signal

When data cannot leave the device, the decision is made. Categories where on-device is often required:

• Healthcare - patient data, medical records, diagnostic information (HIPAA in US, similar regulations globally)
• Legal - privileged communications, case files, client information
• Financial - personal financial data, trading information
• Personal communications - messages, emails, voice notes
• Enterprise IP - source code, product roadmaps, confidential documents

Note: "on-premise" (self-hosted on your own server) is not the same as "on-device" but achieves similar privacy goals for enterprise use cases.

Latency Requirements

Cloud inference adds at minimum 100–300ms of network latency (TLS handshake + data transfer + server queuing). On fast connections with low server load, real-world cloud latency is typically 200–800ms to first token. On-device inference starts generating immediately. This matters for:

• Voice interfaces - human speech response time expectation is under 300ms; cloud LLM latency makes responsive voice difficult
• Real-time subtitles / translation - needs to keep up with speech; only on-device achieves this reliably
• Game AI - NPC responses need to feel immediate
• Keyboard suggestions - next-word prediction needs sub-20ms response; only on-device achieves this

For tasks where 1–5 seconds of latency is acceptable (document summarization, research queries, long-form generation), cloud latency is irrelevant.

Capability Gap

Be honest about what on-device models can and cannot do:

The Hybrid Architecture

Apple Intelligence is the most visible example of an increasingly common pattern: complexity-based routing between on-device and cloud inference.

• Classify each request by complexity and privacy requirements
• Simple, privacy-sensitive tasks → on-device SLM
• Complex tasks without sensitive data → cloud frontier model
• Complex tasks with sensitive data → on-premise or specialized private cloud

For most consumer applications, 70–90% of requests can be served by the on-device model, with only the hard requests escalating to cloud. This dramatically reduces cloud API costs while maintaining capability for complex tasks.

Cost Comparison

At scale, on-device inference has zero marginal cost (hardware is already paid for). Cloud inference scales linearly with usage. The break-even depends on:

• Cloud API cost per request (typically $0.001–$0.05 per request depending on model and length)
• Number of requests per user per day
• On-device model quality loss (if quality drops, some users may churn, which has its own cost)

For consumer apps with millions of daily active users making 10+ requests per day, on-device inference can save $1M–$10M per month in cloud costs - a significant business motivation independent of privacy or latency considerations.