Inference & Serving
Getting a model to produce a good output is one challenge. Getting it to do so in under 500 ms, at thousands of requests per second, at a cost you can afford - that is inference engineering. This section covers the techniques that close the gap between a model that works in a notebook and one that runs in production.
In This Section
Quantization Explained
FP32 → INT8 → INT4 - how weight quantization reduces memory and increases throughput, and the quality trade-offs at each precision level.
Serving LLMs at Scale
vLLM, TensorRT-LLM, llama.cpp, and Triton Inference Server - architecture patterns for high-throughput, low-latency LLM serving.
Speculative Decoding
How draft-model speculation speeds up autoregressive generation, token acceptance rates, and when the technique pays off.
Batching Strategies for Throughput
Static batching, dynamic batching, continuous batching - GPU utilisation trade-offs and how to tune batch size for your SLO.