Quantization quality and performance
Compare quantized systems under equal quality constraints
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Serving benchmark contract
- Quantization methods
- Statistical evaluation
Two 4-bit artifacts of the 7-billion-parameter model both fit comfortably on one 80 GiB GPU. Artifact A generates 15 percent more output tokens per second but fails a required structured-output test twice as often. Artifact B preserves the test result but dispatches fallback BF16 kernels for several projections and provides little speedup. File width alone cannot rank the deployments.
Run the same prompt set, sampling settings, request-arrival sequence, and latency objectives against the BF16 baseline and each quantized artifact. Record model-quality results first. For artifacts that satisfy the quality gate, record weight memory, key-value memory, time to first token, inter-token gaps, throughput, goodput, power, and kernel fallbacks. The resulting table makes the quality constraint and the performance mechanism independently inspectable.
Quantization decisions have several objectives. Lower bit widths can reduce memory and improve throughput while reducing task quality. A format can improve batch capacity without reducing latency. Another format can improve decode without changing prefill. Report the Pareto frontier for quality, latency, throughput, memory, and cost.
A candidate lies on the Pareto frontier when no other measured candidate is at least as good on every selected dimension and strictly better on one. Service constraints then select a point: minimum tool accuracy, maximum p99 TPOT, fixed device memory, or a cost target. The frontier depends on hardware and workload, so preserve raw measurements instead of assigning one permanent score.
Quantization can also change which deployment topologies are possible. A compressed model may fit on one GPU instead of two, leave more memory for KV, or permit more replicas per node. Deployment-capacity changes can matter more than the speed of one GEMM. Evaluate the complete replica and serving pool.
Evaluate numerical and behavioral quality in layers
Begin with operator and layer checks to catch implementation errors. Compare a quantized linear operator with the reference over representative inputs, then compare layer outputs and logits. Perplexity provides a scalable model-level signal, but it does not cover every downstream capability.
Add task and product evaluations: reasoning, code, tool selection and argument formation, structured output validity, retrieval, safety, multilingual use, and target domains. Include long contexts when KV or activation behavior changes with length. A small average change can hide a severe regression in a low-frequency capability that the product requires.
Use the intended decoding configuration. Quantization changes logits, and sampling can convert small distribution changes into different generations. Exact text equality is not a general requirement, but task success and quality thresholds must use enough samples to quantify uncertainty.
Measure phase, shape, and memory effects
Report model artifact size, device weight allocation, runtime reserved memory, KV capacity, and peak workspace. Then measure prefill and decode separately across prompt lengths, output lengths, scheduled-token counts, and supported shapes. A weight-only format can improve batch-one decode while providing little gain for compute-efficient prefill.
Measure maximum stable concurrency and load-versus-goodput. Extra KV capacity can admit more requests even when individual request latency is unchanged. A topology change from two GPUs to one can remove collectives and free another GPU for a replica, producing a system effect larger than one kernel measurement.
Record which operators use the intended quantized kernel and which fall back to wider precision. Mixed-precision islands can be required for quality or unsupported shapes. Include their weight copies, conversions, and execution time in the reported candidate.
Compare equal service outcomes
Hold prompt set, decoding policy, output requirements, and SLOs constant. If one candidate generates shorter outputs because quality deteriorates or stop behavior changes, its lower latency and token count do not represent equivalent service. Report task completion and generated length beside performance.
Apply quality gates before selecting the cheapest point. Among candidates that pass, calculate goodput, cost per useful request or token, energy, and headroom. A mixed-precision candidate can dominate a uniformly lower-bit candidate when a few sensitive layers preserve quality with small resource cost.
Quantization support changes with GPUs, kernels, and engines. Store the artifact and raw quality results so a new runtime can be evaluated without repeating every modeling decision. Re-run performance after engine, driver, or hardware changes; do not carry forward an old ranking as a property of the method alone.
Include uncertainty in the frontier. Quality scores from sampled generation and latency percentiles from variable arrivals both have sampling error. When two candidates are close, repeat evaluation or use paired inputs and bootstrap intervals. Do not declare one point dominant when the measured differences are smaller than the uncertainty relevant to the decision.
Worked example: Selecting a serving format
BF16, INT8 weight-and-activation, and 4-bit weight-only candidates are available for an interactive model.
- Run the same 10,000 evaluation prompts through BF16, INT8 weight-and-activation, and 4-bit weight-only artifacts. Record task success, structured-output validity, generated length, and uncertainty for each artifact.
- Replay the same open-loop arrival file on the same GPU. Measure p99 time per output token, maximum stable concurrency, weight memory, goodput, and quantized-kernel fallback count for every artifact.
- Apply the declared constraints: tool-use success must remain within 0.5 percentage points of BF16, p99 time per output token must remain below 70 ms, and stable concurrency must reach the service target. Report the complete comparison table.
- Reject the 4-bit candidate if the measured tool-use regression exceeds the quality limit even when decode capacity is highest. Select INT8 only if the recorded quality, latency, concurrency, and cost results all satisfy the requirements.
- Preserve the hardware, kernels, calibration inputs, workload file, raw request results, and confidence intervals. Re-run the evaluation after the model, engine, or task distribution changes.
Conclusion: The four-bit candidate is rejected because it violates the tool-use quality gate. INT8 satisfies the quality, p99 TPOT, and concurrency requirements and reduces cost relative to BF16. The decision remains specific to the tested hardware, kernels, and workload distribution.
Common errors
- Collapsing quality into one average benchmark. Important capability cliffs can hide in subdomains.
- Comparing token rates when candidates generate different lengths or error rates. Goodput needs a stable numerator.
- Calling a dominated point a tradeoff. If another candidate is better on every relevant dimension, there is no reason to choose it.
Reference summary
Apply quality gates before comparing service cost. Evaluate operator and layer error, perplexity, product tasks, long-context behavior, structured output, and required domains. Then measure memory, prefill, decode, concurrency, power, and goodput with identical service settings.
- Hold model behavior or an explicit acceptable-quality threshold constant.
- Measure model load time, weight memory, KV memory, TTFT, TPOT, throughput, and power.
- Sweep prompt/output shapes and batch sizes because bottleneck regimes change.
- Inspect whether parallel communication bytes or cache capacity change.
- Report fallback operators and mixed-precision islands.
- Plot a Pareto frontier; do not collapse quality and speed into a hidden subjective score.
A method that is slower on one GPU can become faster on hardware with native support for its format. A method with low numerical error can still be unsuitable when efficient kernels are unavailable. Preserve raw quality and performance data so that you can evaluate the method on new hardware.
Knowledge check
Why should quantization benchmarks include both batch-1 decode and high-throughput operation?
Show the answer guide
- Batch-1 decode often has small matrix shapes and little weight reuse. The batch-1 path can be dominated by weight bandwidth, launch overhead, and quantization unpacking, so weight compression can produce a large gain or expose kernel overhead.
- High-throughput operation reuses weights across many token rows and can become compute-, KV-, or collective-bound. Quantized matrix kernels can have different tensor-core efficiency, workspace, and supported shapes in this regime.
- A format that wins at batch one can therefore have little gain or regress at a saturated batch, while another format has the opposite result. One benchmark cannot establish both interactive latency and fleet capacity.
- Quality is a common acceptance gate, but performance must be measured separately for batch-1 TTFT and TPOT, high-load throughput and goodput, memory capacity, and the actual prompt and context distribution.
Evaluate one quantized model in two serving regimes
Compare a BF16 model with one quantized candidate under two declared workloads: isolated batch-1 requests with a 1024-token prompt and 128-token output, and a load sweep that keeps 32 or more sequences active with the same length distribution. Hold decoding policy and hardware constant and run a fixed quality evaluation before performance tests.
Evidence to keep: Submit the quantization recipe, quality results, batch-1 TTFT and TPOT distributions, the high-load throughput-goodput curve, peak memory and maximum admitted concurrency, and profiler reports from both regimes. Give separate deployment conclusions for interactive latency and saturated capacity.
Quantization reduces cost per model pass. Speculative decoding attempts to reduce the number of serial target passes needed per accepted token.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.