Inference load testing
Produce a load-versus-goodput curve instead of a vanity maximum
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Benchmark contract
- Serving metrics
- Arrival processes
A benchmark driver sends prompts to the 7-billion-parameter service on one 80 GiB GPU. At 20 offered requests per second, the server completes about 20 and queues remain short. At 85 requests per second, batching improves device use and the server still meets its latency limits. At 120 requests per second, completion rate flattens, queue length grows throughout the run, and many first tokens arrive late.
The driver must continue offering the declared arrival rate even when responses slow if the goal is to expose overload. The driver records arrival time, prompt length, requested output limit, actual output length, cache state, status, and every response timestamp. Server counters record queue depth, active sequences, scheduled tokens, free key-value blocks, and failures. The joined records produce the offered-load curve studied in this lesson.
An offline benchmark asks how much work a server can finish when requests are always available and latency is unimportant. An online benchmark asks how the server behaves under an arrival process and service objectives. Both are legitimate, but they answer different questions. Mixing them creates impressive numbers that cannot size a product.
Measure serving capacity across a range of offered loads. At low load, latency is close to service time and hardware can be underused. Increased load can improve batching and throughput. Near saturation, queue delay and tail latency increase quickly. Above sustainable capacity, completion rate stops increasing while backlog continues to grow. Select an operating region from goodput, latency, memory reserve, and recovery requirements.
A reproducible benchmark also fixes model behavior. Tokenization, prompt formatting, sampling parameters, stopping rules, quantization, and quality checks can change both output length and work. Two engines that generate different numbers of tokens or use different quality settings have not processed the same benchmark contract.
Define workload and model behavior
Build a request trace with arrival time, prompt tokens, requested output limit, actual stopping behavior, priority class, cancellation, and any repeated prefixes. Preserve correlations. Long prompts may belong to particular tenants or arrive in bursts, and output length can correlate with prompt type. Independent random draws can remove the exact structure that causes production tails.
Fix the model artifact, tokenizer, chat template, adapter, quantization, decoding algorithm, temperature, top-p or top-k parameters, random seed policy, stop sequences, and maximum lengths. If exact output matching is not possible across engines, use identical inputs and parameters and report actual generated lengths and a declared quality evaluation.
MLPerf Inference separates scenarios and uses LoadGen to control submission and measurement. A general service benchmark can adopt the same discipline even when it is not an MLPerf submission: define the scenario, enforce latency and accuracy constraints, control query generation, and preserve logs needed for audit.
Use open-loop load to expose saturation
An open-loop generator sends requests according to the selected arrival process without waiting for earlier completions. Offered load therefore remains independent of server speed. If the server falls behind, queue growth and missed deadlines become visible. A closed-loop client waits for a response before sending more work; when the server slows, the client automatically reduces arrivals and can conceal overload.
Sweep load from an underused region through the expected capacity point and beyond it. Each point should run long enough to reach steady behavior and sample long requests, cache eviction, and scheduler interactions. Record queue growth. A test that ends while backlog is still increasing has not demonstrated a sustainable completion rate.
Continue beyond the first SLO failure so the shape of overload is visible. One configuration may degrade gradually and recover quickly. Another may enter preemption, retry, or memory-pressure behavior that collapses goodput. The overload curve determines safe admission thresholds and alert conditions.
Closed-loop concurrency tests remain useful when the question is how a fixed population of clients behaves. Report achieved request rate and client think time with them. Do not use a closed-loop result to claim a sustainable offered-arrival rate, because slower responses reduce the generated load. Load reduction caused by slow responses is the coordinated-omission problem in capacity tests.
Report an operating curve
For each request, record TTFT, inter-token or TPOT behavior, end-to-end latency, prompt and generated lengths, status, cancellation, cache reuse, and queue reasons. For each load point, report input tokens per second, output tokens per second, completed requests, goodput under each SLO, rejection, errors, memory reserve, and power or cost.
Warm and cold states need separate labels. Prefix-cache warmth, graph and kernel compilation, model loading, autoscaling, and allocator state can change latency. A warm steady-state test sizes established replicas. A cold-start test sizes scale-out and recovery. Do not average the states into one number.
Provide replay files, random seeds, exact software and hardware configuration, startup flags, and raw per-request results. Plot percentiles and goodput against offered load rather than reporting only the chosen point. The curve lets a reader select another operating point when SLOs or workload mix change.
Repeat selected points to estimate run-to-run variation. Report confidence intervals or the observed spread when comparing close results. Small differences near the saturation boundary can come from arrival randomness, output-length variation, compilation, thermal state, or cache history. A claimed improvement should be larger than this measurement uncertainty or supported by enough repetitions to distinguish it.
Worked example: Finding sustainable capacity
A service must keep p99 TTFT below one second and p99 TPOT below 80 ms. Load tests increase offered requests per second.
- At each load, use an open-loop arrival trace and run through steady state. Track queue length to ensure it is not continuously growing.
- Raw output tokens per second continue rising up to 120 requests/s, but the TTFT SLO fails above 85 and the TPOT SLO fails above 95.
- Calculate goodput as completed requests meeting both objectives. Measured goodput peaks near 85 requests/s and falls as more requests miss constraints.
- Provision below that knee with reserve for burst and failure, then validate mixed priority and realistic prefix-cache conditions.
Conclusion: The 120-request/s point reports higher raw output rate but violates the first-token requirement. The 85-request/s region is the highest tested load that meets both latency constraints. Production capacity must remain below that boundary by enough margin to handle bursts and replica loss.
Common errors
- Using closed-loop clients for an overload claim. Client backpressure can prevent the server from seeing the offered demand.
- Reporting only average latency. Saturation and head-of-line effects appear first in tails.
- Comparing engines with different output lengths, decoding, cache warmth, or quality. They are not processing the same workload.
Reference summary
A serving load test fixes model behavior and request behavior. Preserve the model artifact, tokenizer, prompt format, sampling parameters, stopping rules, prompt and output distributions, arrival trace, cache state, and quality checks.
Use an open-loop arrival generator when the test must expose saturation: the generator continues to offer the declared load even when the server slows. A closed-loop concurrency test waits for completions and can reduce its own request rate during overload. The closed-loop test measures a different operating condition and must be labeled as such.
- Declare model, weights, quantization, hardware, engine version, parallel plan, and quality check.
- Use prompt/output length distributions and request arrival patterns representative of the target.
- Sweep offered load through saturation; do not report only one chosen point.
- Measure warm and cold prefix/cache behavior when both occur in production.
- Report TTFT/TPOT/E2E percentiles, achieved tokens/s, rejection, failures, power, and memory.
- Separate input and output tokens because their compute profiles differ.
Knowledge check
Why should a capacity test continue past the point where latency objectives first fail?
Show the answer guide
- The first latency-objective failure marks the service's admissible operating boundary, but it does not reveal the saturation knee, maximum service rate, or how quickly queueing accelerates beyond the boundary.
- Continuing the controlled load sweep shows whether the service rejects work promptly, builds an unbounded queue, times out after performing wasted work, exhausts KV memory, or enters an unstable throughput collapse.
- The overload region provides the headroom needed for burst planning and distinguishes a policy that degrades gracefully from one that fails abruptly. Recovery after load falls is part of capacity behavior.
- The test should stop at a predefined safety limit, not at the first SLO miss. Post-SLO measurements are diagnostic capacity data and must not be reported as acceptable production operation.
Measure a complete serving operating curve
Replay one fixed request-length distribution at arrival rates from 10% of estimated capacity upward in 10% increments until throughput plateaus, errors exceed 10%, or another declared safety limit is reached. Hold each rate for at least five times the longest expected request duration, then reduce load to 20% and observe recovery.
Evidence to keep: Submit the arrival trace and stop conditions, plots of offered load against completed throughput, goodput, p50/p99 TTFT, p50/p99 TPOT, queue depth, rejection, timeout, and KV use, plus a marked SLO boundary and saturation knee. Include a recovery trace and a causal conclusion about the first resource or policy that fails.
The operating curve is shaped by scheduler and allocator decisions made at every iteration. We now study those mechanisms directly.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- MLPerf Inference Rules↗
- Orca: Iteration-Level Scheduling↗
- PagedAttention / vLLM↗
- NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.