LLM Primer VI Scaling AI Systems: Architecting Low-Latency LLM Inference for Production Scale
English | June 17, 2026 | ASIN: B0H5PRDMPK | 389 pages | Epub | 6.76 MB
A single user generating real-time text leaves a high-end H100 GPU 99.7% idle. Inference engineering is the discipline of finding work for those idle units.LLM Primer VI is the practitioner's playbook for architecting LLM inference at production scale. It walks the full stack between "the model fits in memory" and "the system serves millions of requests a day under contract latency" - token mechanics, KV cache management, hardware selection, model compression, batching and speculation, serving engines, Kubernetes orchestration, and the unit economics of every layer.What's insideThe autoregressive bottleneck - why decoding is memory-bandwidth-bound and prefill is compute-boundKV cache math: the formula, MHA/GQA/MQA trade-offs, and the invention of PagedAttentionNVIDIA (H100/H200/B200/L40S) vs AMD MI300X vs specialized silicon (Groq, Inferentia, TPU, Gaudi)Quantization with AWQ, GPTQ, GGUF, SmoothQuant - and Blackwell's FP4 hardware accelerationPruning, 2:4 sparsity, and Patient Knowledge DistillationContinuous batching, prefix caching, and KV cache eviction (H2O, InfiniGen)Speculative decoding: EAGLE, Medusa, MTP, Lookahead, n-gram, and the verification bottleneckvLLM, TensorRT-LLM, SGLang, TGI, Ollama - when each fitsRay Serve, KServe, Triton, BentoML - the platform layerDisaggregated prefill/decode pools with LeaderWorkerSet, NVIDIA Grove, KAI SchedulerKEDA + Knative autoscaling, CRIU cold-start mitigation under 5 secondsToken economics: input-vs-output asymmetry, conversation accumulation, invisible reasoning tokensServerless vs dedicated breakeven math, microVM sandboxes for agent codeIntelligent model routing, semantic caching, batch APIs, context compactionIncludes four reference appendices: math formulas and cost modeling, hardware specifications guide, deployment configurations with Python and Kubernetes YAML, and benchmarking methodology with TTFT, ITL, and TPS definitions.Code examples in Python and YAML - runnable, not pseudo-code. Nearly 400 pages of real engineering.Written for platform engineers, SREs, and infrastructure architects responsible for LLM inference at scale. Assumes general LLM familiarity at the level of Volume I and comfort with distributed systems.Volume VI of the LLM Primer series.
Links are Interchangeable - Single Extraction
