Small Language Models
The AI industry is moving in two directions simultaneously: frontier labs building ever-larger models for maximum capability, and a parallel revolution in Small Language Models (SLMs) - models under 7B parameters that can run on consumer hardware, mobile devices, and even microcontrollers. The SLM revolution has one surprising core insight: data quality beats model size.
Why SLMs Matter
- On-device deployment - 1–7B models run on phones, tablets, and laptops without cloud APIs
- Cost - 10–100× cheaper per token than frontier models; often free for on-device use
- Latency - no network round-trip; responses can be real-time on fast NPUs
- Privacy - data never leaves the device
- Offline capability - works without internet connection
- Narrow task efficiency - many tasks (classification, extraction, routing, summarization) do not require a 100B+ model
Microsoft Phi - The Data Quality Benchmark
The Phi family from Microsoft Research is the canonical example of what focused, high-quality training data can achieve:
| Model | Params | Notable Achievement |
|---|---|---|
| Phi-1 | 1.3B | Competitive with 10× larger models on Python coding benchmarks |
| Phi-2 | 2.7B | Outperformed Llama 2 7B on many reasoning benchmarks |
| Phi-3-mini | 3.8B | Competitive with Mixtral 8×7B (47B active params) on many tasks |
| Phi-3.5-mini | 3.8B | 128K context window, strong multilingual performance |
| Phi-4 | 14B | State-of-the-art for its size class across STEM and reasoning |
The Phi insight: train on "textbook-quality" data - carefully curated educational content, synthetic data generated by GPT-4 to be clear and instructive - rather than raw web crawl data. Small models trained on clean, informative data outperform much larger models trained on noisy internet data.
Google Gemma
Gemma is Google's open-weight SLM family, derived from the same research as Gemini:
- Gemma 2B / 7B (2024) - competitive with models 2–3× their size; widely used for on-device experimentation
- Gemma 2 (2B / 9B / 27B) (2024) - significant quality improvement with new architecture choices (sliding window attention, logit soft-capping)
- Gemma 3 (1B–27B) (2025) - multimodal, 128K context, strongest Gemma generation
- CodeGemma - code-specialized variant; strong at code completion and generation
- PaliGemma - vision-language variant; takes image + text, outputs text
Gemma models are certified to run on Android via MediaPipe and on Google AI Edge, making them the natural choice for Android developers building on-device AI.
SmolLM - Hugging Face's Ultra-Small Models
SmolLM pushed the boundary on how small an LLM can be while remaining useful:
- SmolLM 135M - runs in under 300MB RAM; useful for keyword classification and simple extraction
- SmolLM 360M - small enough for most microcontrollers with sufficient RAM
- SmolLM 1.7B - strong general capability; fits easily on any modern phone
- SmolLM2 (2025) - significantly improved data quality; 135M SmolLM2 outperforms original 1.7B on several benchmarks
Meta Llama - The Open-Weight Giant
While not strictly "small," Meta's Llama family offers strong models across a wide size range. The smaller variants are widely used for edge deployment:
- Llama 3.2 1B / 3B - purpose-built for on-device; multimodal (vision) variants available
- Llama 3.1 8B - the dominant 7–8B class model; strong across tasks; fits on most developer laptops
- Available on Qualcomm, MediaPipe, and Apple Neural Engine via Llama.cpp quantized versions
When SLMs Beat Large Models
- Classification - spam detection, sentiment, topic classification - 1–3B models often match frontier performance after fine-tuning
- Extraction - named entity recognition, structured data extraction from text - SLMs with fine-tuning are competitive
- Summarization - short document summarization - Phi-3 or Gemma 2 9B is often sufficient
- Code completion - single-function completions - CodeGemma 7B rivals GPT-3.5 for most tasks
- Routing / triage - decide which model or tool to call - a 1B model is more than sufficient
SLMs are poor choices for: multi-step reasoning, complex instruction following, novel task generalization, and multilingual tasks with low-resource languages.