Production Inference Systems

A model service normally receives 70 requests each second and can complete 95 under its measured workload. One GPU replica fails while a traffic burst raises arrivals to 110 requests each second. Queue length grows, first-token latency exceeds its limit, clients retry timed-out calls, and retry traffic increases the overload. A dashboard that reports only GPU utilization or total tokens per second can look healthy while the fraction of useful requests falls.

Production evidence must connect the user-visible request to internal work. Service logs count offered, admitted, rejected, canceled, failed, and completed requests. Histograms report first-token and inter-token latency by request class. Scheduler counters report queue length and KV-block reserve. GPU metrics report model work, memory use, and power. Cost records attach accelerator, host, network, and idle-reserve expense to the same measurement interval.

Chapter 20 defines goodput as completed work that satisfies declared constraints per unit time. For example, request goodput can mean the number of requests per second that finish successfully, preserve required model quality, deliver the first token within one second, and keep subsequent token gaps below 80 milliseconds. The numerator is deliberately narrower than raw throughput because late, incorrect, or failed output consumed resources without meeting the service objective.

Each replica contains host central processing unit (CPU) processes and one or more graphics processing units (GPUs). Scheduler counters include free key-value (KV) blocks. Request metrics include time to first token (TTFT) and time per output token (TPOT). A service-level objective (SLO) converts those metrics into explicit success criteria before the operator calculates goodput or cost.

The chapter connects service-level objectives, observability, capacity planning, overload control, and rollout review. Each calculation names the request population, time window, measurement location, and completion condition. The final investigation asks whether a kernel or scheduler improvement changes useful service outcomes after traffic variation, replica loss, warm-up, cache behavior, and operating cost are included.