Batching & Throughput Strategies
A GPU can process multiple matrix operations in parallel - but LLM inference systems must explicitly organise requests into batches to exploit this. The batching strategy determines how much of the GPU's throughput capacity you actually use. The difference between naive static batching and continuous batching can be a 10× improvement in requests-per-second for the same hardware.
Static Batching
Static batching groups a fixed number of requests, runs them together, and waits until all complete before accepting new requests. Simple to implement, but wasteful: if one request in the batch generates 500 tokens and another generates 20, the GPU sits partially idle waiting for the long request to finish before the batch slot frees up.
Problem
Variable output lengths cause "head-of-line blocking" - short requests that finish early must wait for long ones. GPU utilisation drops significantly compared to the theoretical maximum.
Continuous Batching
Continuous batching (also called iteration-level scheduling) allows new requests to join a running batch at each decode step. As soon as a request in the batch completes, a new request from the queue fills its slot - the GPU is almost never idle between requests.
This is the core scheduling technique used by vLLM, TGI, and TensorRT-LLM. It is the primary reason production-grade servers achieve 10–20× the throughput of naive implementations.
Dynamic Batching
Dynamic batching collects requests arriving within a configurable time window (e.g. 10–50ms) and sends them as a batch. This reduces per-request overhead and improves GPU efficiency at the cost of adding a small latency for each request. Used by Triton Inference Server and cloud serving systems.
High traffic (QPS > 50)
Dynamic batching pays off - batches form naturally from concurrent requests. Set window to 20–50ms; batch overhead is negligible vs request latency.
Low traffic (QPS < 10)
Dynamic batching adds latency without much benefit - requests do not overlap enough to fill batches. Use continuous batching without a wait window.
Priority Queuing
For multi-tenant or mixed-SLA systems, priority queuing ensures high-priority requests (interactive user queries) are not blocked behind low-priority batch jobs (asynchronous document processing). Implement priority queues at the application layer before the serving system, or use vLLM's built-in priority scheduling.
Prefill/Decode Separation
An emerging architecture (used by large cloud providers) separates the prefill and decode phases onto different GPU pools:
- Prefill workers process the input prompt (parallel, compute-bound) - run at high batch size
- Decode workers generate output tokens (sequential, memory-bandwidth-bound) - optimise for latency
This prevents long prefill operations from blocking the decode stage and vice versa, reducing tail latency at scale. It is complex to implement and primarily relevant for very high-throughput deployments.
Key Metrics to Monitor
| Metric | What it tells you |
|---|---|
| TTFT (time to first token) | Prefill latency - determines when user sees first response character |
| ITL (inter-token latency) | Decode speed - determines streaming smoothness |
| Throughput (tokens/sec) | Total GPU output capacity - primary cost efficiency metric |
| KV cache utilisation | % of GPU VRAM used by KV cache - if >95%, add more GPUs or reduce context length |