Advanced

vLLM & Production LLM Serving

Running an open model in production is not the same as running it for development. A naive approach (single-request, synchronous inference) leaves 80%+ of GPU throughput on the table. Production LLM serving is about maximising utilisation of expensive GPU hardware while maintaining low latency for users. vLLM is the dominant open-source solution; knowing how it works tells you what all serving systems must solve.

The Inference Throughput Problem

LLM inference has two phases: prefill (processing the input prompt - parallel, fast) and decode (generating output tokens one at a time - sequential, slow). The memory bottleneck is the KV cache - the key-value attention matrices that must be stored for every token in every active request. Naive systems pre-allocate a fixed KV cache slot per request, leaving most GPU VRAM unused while requests wait for that slot.

PagedAttention: How vLLM Works

vLLM (developed at UC Berkeley, 2023) introduced PagedAttention - inspired by operating system virtual memory paging. Instead of one contiguous KV cache block per request, the KV cache is divided into fixed-size pages allocated dynamically as tokens are generated. Multiple requests share the GPU memory pool efficiently, with no pre-allocation and minimal fragmentation.

Result: vLLM achieves 24× higher throughput than naive HuggingFace Transformers inference on the same hardware in benchmark conditions. In production, 10–20× throughput improvements over naive approaches are typical, depending on request patterns.

Continuous Batching

Traditional batching waits for a fixed batch of requests before starting processing. Continuous batching (iteration-level scheduling) allows new requests to join a running batch mid-generation as slots free up. This dramatically improves GPU utilisation under variable load - the GPU is rarely idle.

Serving Options Compared

vLLM

Open source, Python

Best open-source throughput. OpenAI-compatible REST API. Supports GPTQ, AWQ, FP8, multi-GPU tensor parallelism. The default choice for GPU servers.

Use when: deploying to NVIDIA GPU servers at scale

TensorRT-LLM (NVIDIA)

NVIDIA only

NVIDIA's inference engine - highest throughput on NVIDIA hardware, typically 30–40% faster than vLLM for supported models. More complex to set up; requires model compilation step. Used via Triton Inference Server.

Use when: maximum throughput on NVIDIA A100/H100 is critical and you have MLOps capacity

Text Generation Inference (TGI) - Hugging Face

Open source, Rust+Python

HuggingFace's production serving framework. Rust core for throughput, OpenAI-compatible API, built-in quantization support. Tight integration with HuggingFace Hub for pulling models. Slightly behind vLLM in raw throughput benchmarks but excellent ecosystem fit.

Use when: you use HuggingFace Hub and want tight ecosystem integration

Ollama

Open source, local

Single-user local serving via llama.cpp. Zero configuration, Docker-like model management (ollama pull llama3). Not designed for high-throughput multi-user serving - single request at a time.

Use when: development, personal use, or single-user private deployment

Deployment Architecture

A minimal production vLLM deployment:

# Start vLLM server (OpenAI-compatible API)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --quantization awq \
  --tensor-parallel-size 2 \   # 2 GPUs
  --max-model-len 32768 \
  --port 8000

# Clients use OpenAI SDK with base_url="http://localhost:8000/v1"

Production Tips

Set --max-model-len to the 95th percentile of your actual request lengths - not the model's maximum
Use AWQ quantization for the best throughput/quality tradeoff
Enable speculative decoding for latency-sensitive workloads (see Speculative Decoding page)
Monitor GPU VRAM utilisation - if consistently above 95%, add tensor parallelism across more GPUs
Use prefix caching to avoid re-computing the KV cache for shared system prompts

Serving LLMs at Scale