Quantization formats and granularity
Describe a quantized path completely
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Scale and zero-point
- Weight, activation, and KV dataflow
Seven billion weights stored at two bytes each have a 14-billion-byte payload. The same payload is 14 GB in decimal units or approximately 13.0 GiB in binary units because one GiB equals 1,073,741,824 bytes. File headers, alignment, tied or duplicated tensors, and runtime workspaces can change the observed artifact or allocation size, so report those terms separately. A weight-conversion tool produces a smaller file containing packed 4-bit values, one scale per group of 128 weights, shape metadata, and alignment padding. File size and GPU memory use confirm that storage decreased, but a profiler must still identify the kernel that unpacks or consumes the representation during matrix multiplication.
Inspect one group of weights before and after conversion. Record the original floating values, the selected scale and optional zero point, the integer codes, the packed bytes, and the reconstructed values consumed by a reference calculation. The complete record is a quantization map for that path. A label such as '4-bit' describes only the payload width and omits scale storage, grouping, activation format, accumulation, and kernel support.
Quantization is a family of representation choices, not a single switch. Weights, activations, KV state, and accumulators can use different formats. Scales may apply to a tensor, channel, group, or block. Mapping may be symmetric or asymmetric, calibrated from data or learned during training. Hardware may consume one combination natively and emulate another through unpacking and conversion.
A complete quantization path follows values from storage to arithmetic and back. The path accounts for compressed payload and metadata, identifies where scales are loaded, names the matrix instruction and accumulator, and shows whether unpacking or dequantization is fused. The phrase four-bit weights describes one storage property; the phrase does not specify the execution path.
Keep the quantization method separate from the representation format. GPTQ or AWQ can produce weights for a particular group size and packing scheme. A runtime kernel must support that exact combination on the target hardware. Another engine can use the same method name but select a different packing or fallback path.
Define the mapping and scale granularity
Uniform affine quantization maps a real value x to an integer q using a scale s and, for an asymmetric scheme, a zero-point z. A typical expression rounds x / s + z and clamps it to the representable integer range. Dequantization reconstructs approximately s × (q - z). Symmetric schemes usually set the represented zero at the center and can simplify metadata and arithmetic.
One scale can cover a complete tensor, one output channel, or a smaller group or block. Finer granularity adapts to local ranges and often reduces error. Finer granularity also increases scale metadata, scale loads, address calculations, and kernel specialization. Padding and alignment can increase the effective bits per weight beyond the nominal payload width.
State the accumulator and output formats. Narrow input multiplication often accumulates in INT32, FP16, BF16, or FP32 before an epilogue scales and converts the result. Overflow, rounding, and reduction order depend on these choices. A complete numerical contract names every boundary rather than labeling the whole operator INT8 or FP8.
Distinguish weight-only and weight-activation paths
Weight-only quantization stores compressed weights while activations remain in a wider format. During GEMM, the kernel unpacks or dequantizes weight tiles and multiplies them with the wider activations. The weight-only path reduces persistent storage and weight bandwidth, which can be valuable in small-batch decode. The conversion instructions and scale traffic must remain small relative to the saved bytes.
Weight-and-activation quantization also converts activations and can use narrow matrix instructions with higher throughput. Static activation scales come from calibration, while dynamic scales require runtime range calculation and reduction. Activation outliers can force a wide scale and poor resolution, so methods such as SmoothQuant change the distribution before W8A8 execution.
Batch size changes the expected gain. At batch one, decode often streams weights and can benefit greatly from weight compression. At a larger token batch, each weight tile is reused across more rows and arithmetic can become limiting. A four-bit weight-only kernel can therefore save memory without improving a compute-bound operating point.
Treat KV quantization as an online dataflow
KV values are produced for every new token, stored for the sequence lifetime, and read by later attention operations. Quantizing the cache reduces per-token capacity and can reduce long-context attention bandwidth. Unlike weights, these values cannot all be transformed once during model conversion.
The runtime must select or calculate scales as tokens arrive, write the compressed representation, load scale metadata during attention, and reconstruct values with sufficient accuracy for logits and softmax. Per-token, per-head, or per-block granularity changes both error and metadata. The quantization work lies directly on prefill and decode paths.
Measure maximum cache tokens, active-request concurrency, KV write time, attention read time, memory traffic, and long-context quality. Accumulators and softmax generally remain wider even when K and V are compressed. A capacity improvement can be useful without a kernel speedup if it raises goodput by admitting more sequences.
Worked example: Comparing two four-bit weight schemes
Scheme A uses one symmetric scale per 128 weights. Scheme B uses one scale and zero-point per 32 weights.
- Scheme B adapts more closely to local ranges but carries four times as many scale groups plus zero-points. Calculate effective bits per weight including metadata.
- Inspect packing and kernel support. If Scheme A maps to a mature fused GEMM while Scheme B requires extra unpack instructions or separate dequantization, nominal reconstruction error may not predict speed.
- Evaluate layer- and task-level quality across representative calibration and inference data, paying attention to outlier channels.
- Benchmark prefill and decode separately. Weight bandwidth matters differently as token batch and matrix shape change.
Conclusion: Scheme B may reduce reconstruction error with smaller asymmetric groups, but it carries more metadata and may lack an equally efficient fused kernel. Select between the schemes using effective bits, generated instructions, phase-specific performance, and quality on the deployment workload.
Common errors
- Reporting compressed payload without scales, zero-points, padding, and packing. Effective size can differ materially.
- Assuming dequantization happens for free. Its instructions and metadata traffic must be fused and amortized.
- Using one quality score for all workloads. Long context, tool use, rare domains, and generation settings can expose different sensitivity.
Reference summary
A quantization specification names the stored payload, scale and zero-point granularity, symmetric or asymmetric mapping, packing and padding, activation format, accumulator format, output conversion, and consuming kernel. The method used to create the artifact does not by itself specify these runtime properties.
Weight-only, weight-activation, and KV quantization change different traffic. Weight-only formats mainly reduce persistent storage and weight reads. Weight-activation formats can use narrow matrix instructions. KV formats reduce growing request state but add online quantization and attention dequantization.
- Weight-only: reduces weight traffic and capacity; activations remain wider.
- Weight-and-activation: can use integer or narrow floating matrix pipelines but must handle activation outliers.
- KV-cache: reduces long-context capacity and bandwidth with per-token runtime quantization/dequantization.
- Static scales come from calibration; dynamic scales adapt at runtime with reduction overhead.
- Per-tensor scales are cheap but coarse; per-channel/group/block scales preserve accuracy with metadata and extra operations.
Knowledge check
Why can a weight-only 4-bit model use far less memory yet show little speedup at batch sizes that are already compute-bound?
Show the answer guide
- Weight-only 4-bit quantization can reduce resident model bytes and weight-transfer bytes substantially because packed weights replace FP16 or BF16 weights, with smaller scale and grouping overhead added.
- At sufficiently large batch size, matrix multiplication (GEMM) reuses each weight tile across many token rows and becomes limited by tensor-core arithmetic, activation traffic, or another resource rather than by reading weights from high-bandwidth memory (HBM).
- Reducing weight bytes cannot accelerate a kernel whose active limit is no longer weight bandwidth. On-the-fly unpacking, dequantization, scale application, and a less efficient quantized kernel can consume the saved time.
- Activations, KV cache, collectives, sampling, and scheduler work also remain. Memory capacity can improve even when end-to-end speed changes little, so memory and latency must be reported separately by batch regime.
Locate the batch where weight compression stops helping
Benchmark one 4096-by-4096 linear layer and one complete decode pass in BF16 and weight-only 4-bit form at token batch sizes 1, 2, 4, 8, 16, 32, 64, and 128. Use the same output dtype and a fixed 2048-token context for the model run. Verify numerical error and record actual packed-weight storage.
Evidence to keep: Submit the benchmark code and quantization configuration, correctness or quality measurements, a table of latency, tokens per second, weight bytes, and peak memory by batch, and profiler reports for batch 1 and the largest batch with memory and compute utilization. Plot speedup versus batch and identify the measured regime change.
The quantization configuration defines the available format and granularity choices. Post-training methods use calibration data and outlier handling to preserve required model quality.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.