Efficiency Strategies for Builders

The architectural decisions you make when building AI applications have direct consequences on energy use and cost. This is not about being "green" for PR - these are straightforward optimizations that typically reduce cost by 50–90% while also reducing environmental impact. The good news: the most energy-efficient design is almost always the most cost-effective one.

Model Routing - The Highest-Leverage Strategy

The single highest-impact efficiency strategy is routing requests to the smallest model that can handle them adequately:

Incoming request
       │
       ▼
Classifier (small model or rules)
  "What kind of request is this?"
       │
  ┌────┴──────────────────────────────┐
  │                                   │
Simple task              Complex task
(classification,         (reasoning, analysis,
 summarization,           multi-step,
 extraction,              novel task)
 routing)
  │                                   │
  ▼                                   ▼
Small SLM                 Frontier model
(Phi-3, Gemma 2B)         (Claude Opus, GPT-4o)
~$0.0001/request          ~$0.005-0.05/request
~0.00003 kWh              ~0.002-0.01 kWh

Typical distribution: 70-85% of requests → small model
Cost reduction: 5-10× vs sending everything to frontier

Semantic Caching

Many real-world AI applications serve similar or identical queries repeatedly. Caching the results of previous queries eliminates the inference entirely for those queries - zero energy, zero cost:

• Exact cache - hash the prompt; if same prompt seen before, return cached response. Simple and effective for FAQs, documentation lookups, standardized reports.
• Semantic cache - embed the incoming prompt; find similar past prompts; return the cached response if similarity exceeds a threshold. Handles paraphrased versions of the same question.
• Result - typical cache hit rates of 20–40% for consumer AI applications; each hit is 100% energy savings for that request.

Tools: GPTCache, Redis with vector search, Momento Semantic Cache, or build your own with pgvector.

Batching - GPU Utilization

A GPU generating a response for 1 user and a GPU generating responses for 16 users simultaneously consume nearly the same power. Batching is the practice of grouping multiple inference requests together to process them in parallel:

• A GPU at 30% utilization (one user at a time) generates 30 tokens while using 700W
• A GPU at 90% utilization (batched requests) generates 3× the tokens using the same 700W
• Energy cost per token: 3× lower with good batching
• Continuous batching (as implemented in vLLM) dynamically adds requests to in-progress batches, achieving much higher GPU utilization than naive batching

Quantization

Running INT4 quantized models instead of FP16 models reduces memory bandwidth requirements by ~4×. Since AI inference is often memory-bandwidth-bound (the bottleneck is loading weights from VRAM, not compute), this translates to roughly 2–4× better energy efficiency:

• INT4 model: ~3.5GB for 7B model → fits in 8GB GPU; serves more users per GPU
• FP16 model: ~14GB for 7B model → requires 16GB+ GPU; serves fewer users per GPU
• Quality difference: typically <3% on standard benchmarks; often imperceptible in real use
• Tools: llama.cpp (GGUF), vLLM (AWQ/GPTQ backends), Ollama (handles quantization automatically)

Prompt Optimization

You pay per token (input + output). Prompt design has direct cost and energy implications:

• Remove unnecessary context - audit system prompts for verbosity. Every 100 tokens of unnecessary system prompt adds ~$0.001 per request; at 1M requests/day this is $1,000/day.
• Control output length - use `max_tokens` to cap output length for tasks that don't need long responses. Summarize to 3 bullet points, not "be concise."
• Reduce context window - don't pass the entire conversation history if only the last 2 exchanges are relevant.
• Structured output - JSON schema outputs are typically shorter than prose descriptions of the same data.

Streaming vs Batch Latency

Streaming responses (tokens delivered as generated) improves perceived latency without changing energy use - but it changes what is possible:

• Streaming allows users to start reading before generation completes - better UX at same cost
• For backend pipelines that do not show output to users until complete, disable streaming - it reduces overhead
• Speculative decoding (running a small draft model to propose tokens, verified by the large model) can reduce latency by 2–3× with marginal energy overhead

Infrastructure Selection

• Spot/preemptible GPU instances - for batch inference jobs (nightly reports, data processing), spot instances can be 60–90% cheaper and often use otherwise idle capacity - better utilization of existing infrastructure
• Region selection - choose cloud regions with high renewable energy penetration (us-west-2 for AWS, europe-west4 for GCP) for lower carbon intensity per kWh without paying more
• Time-of-day routing - if your workload allows it, shift batch processing to off-peak hours when the grid has more renewable energy and electricity is cheaper