SLOs and goodput
Calculate goodput from throughput and service constraints
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Latency percentiles
- Quality constraints
- Admission and load
During one minute, clients offer 6000 requests to the 7-billion-parameter service. The server rejects 200 before GPU allocation, admits 5800, completes 5700, and reports 95 completed requests per second. Only 5100 completions deliver the first token within one second and keep every later token gap below 80 milliseconds. Raw completion throughput is 95 requests per second, while constrained useful completion is 85 requests per second.
The 85-request value is goodput under the declared constraints: 5100 qualifying requests divided by 60 seconds. A service-level objective (SLO) states the request population, the measured quantity, the boundary where timestamps are taken, the threshold, the required percentile or fraction, and the evaluation window. Request logs must retain rejected, canceled, failed, late, and successful populations so the numerator cannot improve merely by discarding inconvenient outcomes.
Throughput counts completed work; goodput counts completed work that met the promises defining usefulness. A response whose first token arrives after a user has abandoned it consumed resources without serving the objective. A fast malformed tool call may create negative value. Service-level objectives turn latency, quality, and reliability from secondary charts into the numerator.
An SLO should correspond to a user experience or system obligation and have a precise measurement boundary. Time to first token, inter-token delay, end-to-end completion, availability, and correctness often need separate objectives. Different request classes can require different thresholds. The scheduler and capacity plan then optimize the number of requests that meet all requirements for their class.
Goodput is evaluated over an offered workload. A system cannot improve it by dropping slow requests from measurement, generating less useful output, or allowing an unbounded queue to defer failures beyond the test window. Arrival, admission, cancellation, completion, and quality populations must reconcile.
Write SLOs with populations and boundaries
Specify the eligible request class, measurement start and end, percentile or rate, threshold, and evaluation window. For example: among admitted interactive-chat requests below a stated prompt limit, 99 percent receive the first response byte within one second over a five-minute window. The definition names both the population and the user-visible measurement points.
Use separate objectives for TTFT, ongoing token cadence, completion, availability, and quality when each affects usefulness. A request can meet TTFT and then stall during decode. Another can stream quickly and fail a tool-call schema. Goodput requires every objective selected for that request class.
Low-volume classes may not produce enough observations for a short-window p99. Use a longer window, a maximum count of misses, or another statistically appropriate rule. Do not combine classes solely to obtain a stable percentile when their service requirements differ.
Calculate goodput from complete request outcomes
Count requests or useful tasks that completed and met latency, quality, correctness, and availability requirements. Divide by elapsed time or provisioned resource as required by the decision. Name this request goodput or task goodput. Token goodput uses qualifying tokens in the numerator and answers a different question. Report offered, admitted, rejected, canceled, timed-out, failed, and successful counts so the numerator can be audited.
Early explicit rejection can protect active work when demand exceeds capacity. Rejection still affects availability and must be counted according to the SLO, but early rejection can preserve more useful completions than accepting requests that will time out after consuming prefill and KV resources. Compare goodput and rejected load together.
Generated-token throughput is useful for hardware accounting, but it is not a substitute for request or task goodput. One configuration can emit more low-quality or abandoned tokens. Another can complete more useful tasks with fewer generated tokens. Select the numerator that represents service value.
Report coverage of the offered population when admission can select requests. Otherwise a policy can appear to improve goodput by accepting only short or easy work. Per-class goodput, rejection, and length distributions reveal this selection effect. Compare policies on the same offered trace rather than only on the requests each policy chose to admit.
Use error budgets to manage operating risk
An error budget allows a bounded fraction of requests to miss an SLO over its window. Burn rate compares recent consumption with the rate that would exhaust the budget. A rollout that improves median throughput while sharply increasing tail misses can consume the budget faster and require rollback.
Reserve capacity and admission thresholds influence budget use. Running near saturation may lower unit cost during steady demand but leave no protection for bursts, long prompts, or a lost replica. Choose an operating point that keeps expected burn within policy under both normal and specified failure conditions.
SLOs do not replace diagnosis. Decompose every miss by phase, policy, dependency, and configuration. The decomposition connects the service objective to scheduler, cache, kernel, network, or model behavior and identifies the change that can increase goodput.
Quality objectives need a monitoring cadence that matches available feedback. Some outcomes can be validated automatically at request time, such as schema validity or tool execution. Other outcomes require sampled evaluation, delayed labels, or model-assisted review. State how delayed quality failures enter the goodput and release decision so system speed is not measured against an unchecked numerator.
Worked example: Choosing capacity by goodput
Configuration A completes 1,200 requests/s but only 88% meet latency and error SLOs. Configuration B completes 1,080 requests/s and 99% meet them.
- A's goodput is 1,056 requests/s; B's is about 1,069. B produces more useful completions despite lower raw throughput.
- Inspect why A misses: saturation queueing, retries, or overly large batches. Some failed work may also consume more downstream resources.
- Compare provisioned cost and headroom. If B uses the same hardware, it dominates on goodput and reliability; if it uses more, cost per useful completion decides.
- Load test burst and replica-failure conditions to ensure the selected point remains inside the SLO envelope.
Conclusion: Configuration B completes fewer requests in total but produces about 1,069 requests/s within the declared constraints, compared with 1,056 for A. Cost and failure-headroom measurements determine whether B remains the better deployment, but raw throughput does not.
Common errors
- Counting only successful completed records. Timed-out and canceled requests can disappear and make the service look healthier.
- Treating admission rejection as identical to silent timeout. Both affect availability, but early explicit rejection can protect useful work and enable retry elsewhere.
- Using one latency objective for every class. Interactive chat, batch summarization, and tool control have different experience boundaries.
Reference summary
An SLO names a request population, measurement boundary, threshold, statistic, and evaluation window. Separate objectives can cover availability, TTFT, inter-token delay, end-to-end completion, and quality. Goodput counts requests or tasks that satisfy every required objective for their class.
Reconcile offered, admitted, rejected, canceled, failed, and completed populations. Requests that disappear after cancellation or timeout still consumed capacity and cannot be omitted from the service outcome.
request goodput = completed requests meeting all declared SLOs / second- Use workload-specific classes; short interactive chat and long batch summarization need different objectives.
- Measure at the user boundary and decompose internally.
- Track cancellations and abandoned work; they consume resources without user value.
- Use error budgets to govern risky rollouts and capacity headroom.
- Report quality gates beside system SLOs when routing among model variants.
Knowledge check
How can total request throughput rise while goodput falls?
Show the answer guide
- Total throughput counts every completed request or token, even when completion occurs after the service deadline. Goodput counts only correct work that satisfies the declared TTFT, TPOT, completion, and reliability objectives.
- As offered load rises, larger batches can increase raw completions per second. The same load creates longer queues and scheduler iterations, so a growing fraction of those completions can miss latency limits.
- Timeouts, retries, cancellations, or degraded outputs can consume capacity and sometimes raise recorded work while contributing no goodput. Completion-only samples can also hide abandoned slow requests.
- Therefore throughput can rise from 100 to 120 requests per second while requests meeting the complete SLO fall from 95 to 70 per second. Report the numerator and failure reasons from arrival, not only successful completions.
Separate throughput from goodput under load
Define an SLO of TTFT below 800 ms, p-request TPOT below 80 ms, correct completion, and no timeout. Replay one fixed request distribution at six increasing arrival rates through and beyond saturation. Label every arrival as admitted, rejected, canceled, timed out, failed, completed within SLO, or completed outside SLO.
Evidence to keep: Submit the event log, the exact goodput calculation, a table and plot of offered load, raw throughput, goodput, queue delay, and each failure class, and one load interval where raw throughput and goodput move in opposite directions. Explain the scheduler or queue mechanism that causes the divergence.
Goodput measures requests that meet the service requirements. Request, scheduler, cache, engine, and hardware records can locate a failure in the gateway, admission controller, scheduler, KV allocator, model worker, GPU kernel, collective, sampler, or response streamer. The records establish timing and association; a replay or controlled intervention is still needed when the diagnosis makes a causal claim.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- MLPerf Inference Rules↗
- PagedAttention / vLLM↗
- SGLang↗
- NCCL RAS↗
- NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.