End-to-end performance review
Evaluate a proposed change before rollout
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- All production-serving subsections
- Experimental method
An engineer proposes a fused quantized projection that reduces one decode kernel from 120 to 90 microseconds. The local measurement is real, but the production decision also needs model-quality results, the fraction of calls eligible for the kernel, effect on complete token-iteration time, scheduler operating curves, goodput under latency objectives, energy, compilation behavior, rollout risk, and a rollback trigger.
An end-to-end review assembles those artifacts in causal order. The baseline defines the current request workload and service objective. A falsifiable prediction states how removed bytes or faster instructions should change a measured GPU interval. Kernel profiles test the mechanism. Load tests evaluate request outcomes. A canary rollout compares the observed fleet result with the prediction and records any new active limit.
The final review joins every abstraction encountered in the book. A proposal may begin with a slow kernel, expensive cache, compiler break, collective tail, or scheduling policy. The proposal is complete only when the mechanism connects to useful work, correctness, representative workload, operational risk, and economics.
A local result can be valid and still reduce service value. A quantized kernel can lower weight bandwidth but change quality and batching. A larger cache block can improve attention access while wasting admission capacity. A faster collective can interfere with compute. The review traces each consequence before a wide rollout.
The document is a decision record, not a profiler transcript. The review should let another engineer understand the workload, prediction, mechanism, evidence, limitations, rollout, and rollback without repeating the entire investigation. Raw profiler captures and benchmark outputs remain attached for verification.
Define the decision and falsifiable prediction
State the exact decision requested. Examples include enabling a quantized artifact for one traffic class, changing KV block size, increasing a prefill token budget, or deploying a new collective path. Define the user or capacity objective and the constraints that may not regress.
Describe the workload distribution, SLOs, quality gates, hardware, software, model version, and baseline operating point. Include offered load, prompt and output lengths, cache warmth, and parallel topology. A result without these conditions cannot be applied safely to another pool.
Write a quantitative prediction before presenting measurements. If block size is halved, estimate saved tail tokens and additional metadata or lookups. State uncertainty and a rejection condition. For example, reject the proposal if attention slowdown or scheduler cost consumes more goodput than the added KV capacity provides.
Connect changes across affected layers
Review model representation, graph and compiler, kernels, memory layout, parallel topology, scheduler, gateway, and operations. Most proposals affect only several layers, but the remaining layers can impose constraints. A new packed weight layout can change kernel dispatch and model loading without changing the logical graph.
Present correctness evidence first enough to establish that measured work is valid. Then present mechanism evidence, such as lower transferred bytes or tail waste; component evidence, such as attention or allocator timing; and end-to-end evidence from a representative operating curve. All should move in the direction predicted by the model.
Show surfaces rather than one selected point. Include relevant shapes, context lengths, concurrency, load, cache-hit rate, and hardware variants. Record regressions and fallback frequency. A dispatch or rollout policy needs the boundary of the supported region, not only its best case.
Decide value, rollout, and next limiting resource
Translate the end-to-end result into additional goodput, reduced cost per useful output, lower energy, latency headroom, or reliability. Weight the effect by traffic that uses the path. Include engineering and operational costs when they materially affect the decision.
Define canary population, comparison period, telemetry, automatic and manual rollback conditions, compatibility, and failure tests. A change that affects cache ownership or model state needs stress, cancellation, preemption, and recovery tests in addition to numerical checks.
State the remaining limitation expected after the change. Removing KV tail waste may make attention address overhead or prefill compute more important. Predicting the next limiting resource provides a consistency check: if an unrelated limit appears, the original model may be incomplete and should be investigated before expansion.
Define the rollout comparison before the canary begins. Assign traffic randomly within valid compatibility constraints, preserve equivalent request classes, and prevent cache warmth or regional load from biasing one group. Run long enough to observe tail lengths, failure events, and allocator steady state. Report both absolute results and the difference from the control.
Close the review with a decision, owner, date, and re-evaluation trigger. A model upgrade, engine rewrite, new GPU, workload shift, or SLO change can invalidate the result. The review should state which assumptions are expected to change and which measurements must be repeated when they do.
Worked example: Reviewing a paged-cache change
A proposal halves KV page size to reduce tail waste and admit more concurrent long-context requests.
- Model saved bytes from the real length distribution and translate them into additional admissible tokens and expected goodput.
- Measure new page-table and allocator overhead, attention address cost, kernel locality, and scheduler CPU time. Small pages may increase metadata and reduce vector efficiency.
- Validate shared-prefix ownership, preemption, eviction, and in-flight safety. Run stress and failure tests as well as numerical comparison.
- Replay load and report TTFT, TPOT, goodput, cache reserve, cost, and tails. Define rollout guards and the metric that triggers rollback.
Conclusion: The proposal is justified only if saved tail capacity produces more admissible tokens and goodput after page-table, allocator, attention, and scheduler costs. The rollout must also preserve shared-prefix ownership and provide a measurable rollback condition.
Common errors
- Presenting mechanism counters without an end-to-end operating point. The business or research system consumes useful capacity, not profiler percentages.
- Presenting end-to-end speed without a causal explanation. Without mechanism evidence, another engineer cannot validate or maintain the improvement.
- Omitting rollback and observability. Production optimization includes the ability to detect and reverse harm.
Reference summary
An end-to-end review begins with a decision, baseline, workload, service-level objective, quality constraint, and falsifiable prediction. The review then connects mechanism evidence, named kernel and request-phase timing, correctness, the online operating curve, fleet value, rollout, and rollback.
After the rollout, compare the observed result with the original prediction. Record where the model was accurate, where an unmodeled cost appeared, and which limit became active next. The decision record improves later capacity models and prevents the same unsupported assumption from returning in another proposal.
- Objective: which goodput, latency, quality, cost, or energy metric should move?
- Mechanism: which bytes, operations, launches, waits, or failures are removed?
- Scope: which models, shapes, hardware, and traffic classes use the path?
- Evidence: microbenchmark, operating curve, shadow/canary, and long-run stability.
- Correctness: numerical, model-quality, safety, isolation, and rollback checks.
- Economics: fleet-weighted value and engineering/operational cost.
- Next bottleneck: what limit should appear if the model is right?
Knowledge check
Why must every optimization proposal predict the next bottleneck?
Show the answer guide
- An optimization removes or reduces one limiting term, so the remaining time and resource demands become a larger fraction of the system. Amdahl's law prevents the eliminated term from determining the complete gain by itself.
- The change can directly increase pressure elsewhere: larger batches improve weight reuse but add KV traffic and latency; quantization reduces weight bytes but adds dequantization; fusion removes high-bandwidth memory (HBM) traffic but adds registers; more replicas reduce queueing but add communication or cost.
- Predicting the next bottleneck specifies what should flatten the speedup and which counter should rise. The prediction turns a benchmark result into a falsifiable system model and determines the next capacity or reliability risk.
- Without the prediction, a local win can move failure to memory, network, scheduler, quality, or recovery and reduce end-to-end goodput. The review is complete only when the new limiting resource is measured or the prediction is revised.
Run a cross-layer optimization review
Choose one controlled serving change, such as increasing decode batch cap from 8 to 32, enabling weight-only quantization, or fusing a decode epilogue. Before running it, write a byte, operation, memory, and latency model for a fixed request trace and predict the first bottleneck after the targeted cost falls. Then execute baseline and changed systems across at least four loads.
Evidence to keep: Submit the prediction made before measurement, implementation or configuration diff, correctness or quality tests, a benchmark matrix containing TTFT, TPOT, throughput, goodput, memory, and cost, and profiler or trace evidence for both the removed and next bottleneck. Finish with a causal conclusion that states whether the prediction survived and what now limits further gain.
The final chapter defines a reproducible method for continued study, experiments, technical reports, and evaluation of new hardware and software.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- MLPerf Inference Rules↗
- PagedAttention / vLLM↗
- SGLang↗
- NCCL RAS↗
- NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.