Inference roofline model
Relate batch size to weight reuse and latency
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Roofline
- Prefill/decode shapes
- Model memory calculation
Profile the 7-billion-parameter model twice on the same 80 GiB GPU. The first run processes a 2048-token prompt and produces large matrix operations with thousands of token rows. The second run advances one active request by one token and produces matrix operations with one token row. Nsight Compute reports different arithmetic throughput and memory traffic even though both runs execute the same model weights.
Repeat decode with 1, 8, 32, and 64 active requests. Record iteration milliseconds, total output tokens per second, per-request token gaps, model-weight bytes, and key-value-cache bytes. Weight reads can be shared across the token rows in a larger batch, while every request still reads its own retained attention state. The measurements reveal why total throughput can rise while each request waits longer between tokens.
The ordinary roofline uses arithmetic intensity per operation. Serving needs another layer: how often model and cache bytes can be reused across scheduled tokens. A decode step at batch one may read billions of weight bytes to produce one token. The same weights serving a larger token batch are reused, raising effective intensity and improving matrix shape. But batching also consumes cache and makes requests wait.
Prefill and decode therefore occupy different hardware regimes. Prefill processes many prompt positions together and usually creates larger matrix dimensions. Decode processes one new position per active sequence and must repeat the model for every generated token. Increasing the number of active decode sequences improves weight reuse, but it also increases KV traffic, cache capacity, and the duration of each scheduler iteration.
There is no single arithmetic intensity for a served model. The value changes with phase, scheduled token count, context-length distribution, datatype, tensor parallelism, and cache representation. The model must be written for one scheduler iteration or for a stated workload class.
Model one prefill iteration
For prefill, the scheduled token count can include hundreds or thousands of positions from one or several prompts. Linear layers reuse each weight tile across many token rows, so their arithmetic intensity and matrix efficiency can be high. Attention also processes relationships among prompt positions, and its work and memory behavior depend on the attention algorithm and prompt length.
A long prefill can achieve strong GPU throughput while producing poor service latency for other requests. Kernel efficiency describes device work during the iteration; it does not include the delay imposed on decode sequences waiting for the iteration to finish. Keep the hardware model and the scheduler critical path in the same analysis.
Chunking changes prefill shapes. Smaller chunks reduce the maximum interruption of decode but can turn efficient GEMMs into smaller operations and add scheduling overhead. Larger chunks improve matrix efficiency and prompt completion rate but increase head-of-line blocking. The token budget determines this trade.
Model one decode iteration
A decode iteration advances each selected sequence by one token. The linear layers read the model weights and apply them to a token matrix whose first dimension is the number of selected sequences. At batch one, a large weight traversal produces one token. At batch 32, the same weights can support 32 token rows, so weight bytes per output token fall substantially.
KV traffic does not amortize in the same way. Each sequence attends to its own retained context. A batch containing long and short sequences can have similar weight work per sequence but very different cache reads. As contexts grow, KV bandwidth or attention work can become the active limit even when weight-only reasoning predicts further benefit from batching.
Larger decode batches also lengthen each iteration. Total tokens per second can rise while the time between tokens for each request rises. If the scheduler waits to form a large batch, time to first token can increase as well. The useful operating point is the largest batch behavior that still meets the latency and fairness constraints, not the highest device throughput without those constraints.
Add fixed, distributed, and scheduler costs
Small token batches may be limited by launch overhead, framework dispatch, graph selection, sampling, or scheduler metadata rather than arithmetic or HBM. CUDA graph capture and kernel fusion can reduce some fixed work, but graph variants consume memory and dynamic shapes can miss the captured path. Record fallback frequency.
Tensor-parallel serving adds collectives within model layers. Communication volume and latency can be poorly amortized at small decode shapes. A larger batch can improve compute efficiency while increasing collective duration. A roofline for one GPU is therefore incomplete for a distributed replica; include link bytes and synchronization on the iteration critical path.
Use the byte and operation model to predict a regime, then measure per-iteration weight traffic, KV traffic, arithmetic throughput, launch gaps, collective time, and scheduler gaps. Plot these quantities against scheduled token count and context length. The changes show where an optimization will stop scaling.
Worked example: Effect of batch size on decode throughput
For this simplified example, profiler memory counters show approximately 4 GB of model-weight traffic during one complete decode iteration: the total bytes requested while the token rows pass through every transformer layer once. Compare an iteration with one scheduled token with an iteration containing 32 scheduled tokens. Ignore KV-cache traffic for the first calculation, then add it back explicitly.
- At batch one, roughly 4 GB of weights support one token's model work. With 3 TB/s of sustainable memory bandwidth, weight reads alone impose an ideal lower bound near 4 GB / 3 TB/s = 1.33 ms per iteration. The reciprocal gives an upper bound near 750 output tokens/s before KV reads, arithmetic, collectives, launches, or host gaps.
- At 32 tokens, the same ideal 1.33 ms weight traversal serves 32 token positions, reducing the weight-only contribution from 1.33 ms to about 0.042 ms per output token. The estimate assumes the weights are read once and reused across all rows; generated instructions and memory counters must confirm that assumption.
- Add reality: active requests have different context lengths, so KV traffic is not equally amortized. The batch also needs 32 cache allocations and may increase queue delay.
- Measure the operating curve: per-token device time, TPOT distribution, total tokens per second, and goodput at several arrival loads and batch limits.
Conclusion: A 32-token iteration can amortize the weight traversal across 32 outputs, but it also reads 32 separate KV histories, reserves more cache, and takes longer than a batch-one iteration. The selected batch limit must be justified by goodput and latency curves under the actual context distribution.
Common errors
- Using one arithmetic intensity for the model. Phase, active tokens, cache length, and quantization change it continuously.
- Treating batch size as free reuse. Requests must wait and reserve state to form that batch.
- Ignoring fixed launch and host cost at small decode shapes. The active limit may sit outside the arithmetic roofline.
Reference summary
Prefill processes many prompt positions and commonly forms large matrix operations. Decode processes one new position per selected sequence and repeats the model for every generated token. A larger decode batch amortizes weight reads across more output tokens, but each sequence also waits for the longer batch iteration.
KV traffic grows with each sequence's attended context and is not amortized like one shared weight traversal. Model weight bytes, KV bytes, arithmetic, collective time, and launch or scheduler gaps for one iteration. The active limit can move as batch and context change.
decode weight arithmetic intensity grows roughly with concurrently processed tokens- Throughput mode: large token batches, relaxed latency, high weight reuse.
- Latency mode: small batches, short queue, less reuse and lower hardware efficiency.
- Mixed service: explicit priorities or separate capacity for latency classes.
- Disaggregated prefill/decode: independent resource pools when phases need different operating points.
Knowledge check
Why can a larger decode batch reduce cost per token while increasing both TTFT and TPOT?
Show the answer guide
- A larger decode batch applies one traversal of model weights to more token rows. Weight bytes and fixed launch costs per output token fall, and matrix shapes can use the GPU more efficiently, so cost per token can decrease.
- Forming the larger batch can hold a newly arrived request in the scheduler before prefill begins. The batching or queueing delay raises time to first token.
- Once decoding starts, an iteration containing more token rows usually takes longer. Each request receives one token per selected iteration, so the longer iteration raises time per output token even while the server completes more total tokens per second.
- Longer contexts also add separate KV-cache traffic for each request. The economic gain stops when KV traffic, compute, collectives, or latency limits replace weight bandwidth as the binding constraint.
Plot the decode batch trade-off
Serve a fixed model with prefix caching disabled and decode batch caps of 1, 4, 16, 32, and 64. Replay requests with a declared 1024-token prompt and 128-token output at enough arrival rates to keep each cap populated. Record scheduler wait, prefill time, decode-iteration duration, output tokens per second, TTFT, TPOT, energy if available, and total GPU time.
Evidence to keep: Submit the server configuration and workload, a benchmark matrix for every batch cap, plots of cost or GPU time per token, p99 TTFT, and p99 TPOT versus batch, plus one profiler report comparing batch 1 with the throughput-optimal batch. Identify the measured batch at which latency fails before token cost reaches its minimum.
The roofline predicts regimes. A serving benchmark must sweep load and concurrency to reveal where those regimes become an operational curve.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- MLPerf Inference Rules↗
- Orca: Iteration-Level Scheduling↗
- PagedAttention / vLLM↗
- NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.