IIntroduction to AI Systems Performance EngineeringTechnical textbook
Chapter 20: Production Inference Systems
20.2

Inference observability

Design signals that support causal diagnosis

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

  • Request timeline
  • Metrics, logs, and traces

A dashboard shows GPU utilization rising from 70 to 92 percent while first-token latency doubles. The utilization number does not say whether longer prompts, larger batches, additional key-value reads, scheduler gaps, or retry traffic caused the change. Join client request timestamps, gateway logs, scheduler decisions, batch identifiers, GPU profiler events, memory counters, and deployment-version labels for the same interval.

The joined evidence establishes an ordered sequence and supports a causal hypothesis. A long-prompt burst appears in the request log before scheduled prefill tokens rise in the scheduler log; those iterations are longer in the GPU timeline; interactive requests then receive decode opportunities less often. Every measurement names its producer and unit. The sequence does not by itself prove that long prefills caused the latency change. Replay the same arrivals with a smaller prefill budget, or otherwise change the suspected mechanism while holding the workload fixed, and test whether iteration duration and interactive latency respond as predicted.

A model server crosses boundaries faster than a human can inspect them: gateway, scheduler, tokenizer, cache allocator, distributed workers, kernels, sampler, streamer, and downstream tools. Observability must preserve causality across those boundaries without recording sensitive content or adding enough overhead to change token cadence.

Metrics summarize populations, traces connect one request's dependencies, and logs capture discrete events and context. Each has a role. Aggregate counters without shared identifiers cannot explain one tail request. Recording a complete trace for every token can consume enough CPU, network, and storage to change the serving result.

Current inference engines expose many useful metrics, including queue time, prefill time, decode time, TTFT, TPOT, prompt tokens, generation tokens, KV reuse, and speculative acceptance. Names and availability change across versions, so the service should map engine metrics into a stable internal data model and record engine version.

Instrument request, scheduler, engine, and hardware scopes

At request scope, record class, arrival, prompt and output lengths, queue reason, cache-hit tokens, TTFT, token gaps, completion, cancellation, retry, and status. At scheduler scope, record active sequences, scheduled prefill and decode tokens, admissions, preemptions, rejections, priorities, and free-block reserve.

At engine scope, record iteration duration, batch shapes, graph or kernel variant, fallback path, prefill and decode throughput, memory high-water marks, and distributed timing. At hardware scope, record power, clocks, thermal limits, memory errors, link counters, and device health. Deployment and configuration changes should be available as timeline events.

Each signal should answer a diagnosis question. Queue reason explains why execution did not begin. Scheduled token mix explains iteration duration. Clock and thermal data distinguish workload changes from throttling. Remove signals that have no owner or use; uncontrolled telemetry creates overhead without improving decisions.

Preserve causality across many-to-many batching

Use distinct identifiers for request, logical sequence or branch, scheduler iteration, execution batch, and distributed step. One request participates in many decode iterations. One batch contains tokens from many requests. A flat request-to-kernel identifier cannot represent both relations.

Record scheduler decisions before launch and completion events after execution. The recorded event trace can then show that request R skipped iteration 410 because of priority, joined batch B at iteration 411, and received its next token after that batch's collective. The joined evidence is stronger than inferring policy from a gap in GPU activity.

Distributed clocks can drift. Synchronize them to the accuracy required for the analysis, record uncertainty, or use logical sequence and local durations when ordering is more reliable than absolute time. Device event timing and host wall-clock timing also have different semantics; do not subtract them without a correlation method.

Control overhead, cardinality, sampling, and privacy

Do not place request IDs, raw prompts, or other unbounded values in metric labels. Use metrics for bounded dimensions and distributions. Store high-cardinality identifiers in sampled traces or structured logs with retention and access controls. Align histogram buckets with SLO thresholds and diagnostic ranges.

Combine a small unbiased baseline sample with tail- and error-triggered samples. Tail sampling retains the requests needed for diagnosis, while the baseline prevents a dataset containing only failures from being treated as the workload distribution. Record sampling decisions so denominators remain correct.

Avoid raw content unless a separately governed debugging workflow requires it. Token counts, model and adapter IDs, cache-match lengths, policy reasons, and privacy-preserving request classes usually provide enough performance context. Benchmark telemetry on and off to quantify its effect on CPU gaps, network traffic, storage, and token cadence.

Define retention and aggregation by purpose. Fine-grained batch traces may be needed for hours during a rollout, while SLO metrics must remain available for longer trend analysis. Schema versions should accompany telemetry so engine upgrades do not silently change units, phase definitions, or label meanings. Alert rules need explicit owners and response actions.

Worked example: Explaining periodic TPOT spikes

Every few minutes, p99 TPOT spikes while GPU utilization and average load appear stable.

  1. Tail-triggered traces show decode iterations delayed behind cache allocator work and a burst of prefix evictions.
  2. Join batch and cache events. A large prompt cohort consumes free blocks, pushing reserve below a threshold and causing synchronous reclamation.
  3. Move reclamation off the critical path where safe, preserve a free-block reserve, or change admission before pressure becomes urgent.
  4. Verify spike frequency, cache hit rate, memory headroom, and total goodput. Avoid fixing tails by disabling valuable reuse without an economic comparison.

Conclusion: The joined trace establishes that the long-prompt cohort precedes free-block depletion, synchronous reclamation, delayed decode iterations, and p99 TPOT spikes. This ordering supports the reclamation hypothesis. A controlled replay that preserves a free-block reserve or moves reclamation off the decode path must reduce the predicted iteration gaps and TPOT spikes before the service treats the explanation as causal.

Common errors

  • Adding labels until every event is unique. Unbounded cardinality can overwhelm telemetry systems and obscure useful aggregation.
  • Tracing only healthy requests. Tail and failure sampling must be deliberate, while retaining an unbiased baseline.
  • Logging sensitive prompt or output content for convenience. Performance causality rarely justifies the privacy risk.

Reference summary

Telemetry must connect one request to many scheduler iterations and one execution batch to many requests. Use request, sequence, batch, iteration, and distributed-step identifiers. Record policy decisions as events instead of inferring them later from GPU activity.

  • Request: tenant/class, input/output length, queue time, TTFT, TPOT, E2E, cancellation, retry, and result status.
  • Scheduler: active sequences, scheduled tokens, prefill/decode mix, preemption, priority, and rejection.
  • Cache: pages used/free, fragmentation, prefix hit value, eviction, and copy-on-write.
  • Engine: batch shapes, kernel/graph variant, tokens/s, memory peaks, and fallback paths.
  • Hardware: power, clocks, thermals, memory errors, link counters, and device health.
  • Outcome: useful tasks completed, quality, cost, and the declared service-value measure where applicable.

Use high-cardinality identifiers in traces and sampled logs, not unbounded metric labels. Preserve privacy by avoiding raw prompt content unless explicitly governed. Correlate deployments and configuration changes so performance shifts have context.

Knowledge check

Which signals distinguish slow decode caused by a larger batch from slow decode caused by thermal throttling?

Show the answer guide
  • For a batch-driven slowdown, scheduler records show more scheduled sequences or tokens per decode iteration at the same time TPOT rises. Iteration duration and matrix multiplication (GEMM) shapes increase with batch, while GPU clock remains near its normal value.
  • For thermal throttling, hardware telemetry shows temperature reaching a limit, power or thermal throttle reasons becoming active, and streaming multiprocessor (SM) or memory clocks falling. The slowdown can occur even when scheduled batch size and shapes remain constant.
  • Normalize kernel or iteration latency by batch and context length, then correlate it with both scheduler variables and clocks. A batch explanation follows workload changes; a thermal explanation follows clock changes after temperature accumulation.
  • Confirm causality with a controlled intervention: hold batch and shape fixed while changing cooling or power state, or hold temperature conditions stable while changing the batch cap. A single utilization percentage cannot distinguish the two causes.
Practice

Distinguish scheduler growth from clock throttling

Run a fixed-length decode workload in two controlled experiments. In experiment A, keep the GPU cool and increase the active batch from 1 to 64. In experiment B, hold batch and context length fixed while creating a safe sustained load that changes temperature or power state within vendor limits. Sample scheduler batch, iteration time, kernel shapes, GPU temperature, power, clocks, and throttle reasons at one-second or finer intervals.

Evidence to keep: Submit the workload and safety limits, synchronized scheduler and hardware telemetry, two time-aligned plots, and profiler traces for equal-shape iterations before and during each slowdown. Give a causal diagnosis that cites the signal which changes while the competing variable remains controlled.

Observable demand and service behavior feed a capacity model that translates tokens, requests, memory, and reserve into unit economics.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.

  1. MLPerf Inference Rules↗
  2. PagedAttention / vLLM↗
  3. SGLang↗
  4. NCCL RAS↗
  5. NVIDIA Nsight Systems User Guide↗

    Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.