Outliers, calibration, and post-training methods
Explain the problems addressed by SmoothQuant, GPTQ, and AWQ
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Quantization map
- Activation distributions
- Error metrics
Collect activation values from representative prompts as the 7-billion-parameter model runs. A few channels can contain values much larger than most channels. Mapping the complete tensor to a narrow integer range then forces either large clipping error for the outliers or coarse steps for ordinary values. Histograms and per-channel maximum reports make the outlier problem visible before any named quantization method is introduced.
Use one original checkpoint and one recorded calibration set for three separate conversions. Apply SmoothQuant first and inspect how one outlier activation channel and its corresponding weight channel change. Apply GPTQ to a fresh copy and inspect how calibration inputs make some weight errors more consequential than others. Apply activation-aware weight quantization (AWQ) to another fresh copy and inspect which activation channels cause associated weights to receive additional protection through scaling. Preserve each conversion tool's version, calibration data, group size, scales, packing format, and exact runtime kernel. Perplexity and task evaluations measure resulting behavior; kernel profiles measure execution.
Quantization error is not evenly distributed. A small set of channels or weights can have much larger magnitude, forcing a shared scale to cover a wide range and wasting resolution on ordinary values. Activations can produce input-dependent outliers. Practical methods reshape, protect, or calibrate these difficult regions rather than reducing every value identically.
Post-training quantization avoids full model retraining, but it still requires calibration samples and an objective. Minimizing unweighted parameter error is not the same as preserving layer output or model loss. SmoothQuant, GPTQ, and AWQ address different sources of error, so their mechanisms should be learned separately.
The methods also target different execution forms. SmoothQuant prepares weights and activations for 8-bit weight by 8-bit activation execution, commonly written W8A8. GPTQ and AWQ produce weight-only artifacts: the stored weights are narrow while the layer inputs ordinarily remain in a floating-point format. The next sections derive each intervention from a measured tensor before comparing their names or headline bit widths.
Use calibration to observe relevant distributions
Calibration runs representative samples through the model and records ranges or optimization statistics. The sample should include target domains, prompt forms, lengths, modalities, and activation patterns. Random token sequences can produce distributions unlike actual inference and can miss the channels that dominate real error.
Min-max scaling preserves the observed extreme but can leave few quantization levels for common values. Clipping sacrifices outliers to improve resolution elsewhere. Per-channel or per-group scales adapt locally. The selected objective can minimize weight error, layer-output error, reconstruction over calibration inputs, or a proxy for model loss.
Calibration is not a quality evaluation. Calibration configures the transformation. Use separate data for perplexity, tasks, long-context tests, structured outputs, and domain capabilities. Record the calibration dataset and conversion software so the quantized artifact can be reproduced.
Follow one activation channel through SmoothQuant
Consider one linear layer whose input matrix has 4096 channels. Calibration shows that channel 731 repeatedly reaches magnitude 20 while most channels remain near magnitude 1. A single INT8 activation scale must cover the largest magnitude in its group. The scale needed for 20 gives the ordinary values relatively few distinct integer levels, so their rounding error increases.
The layer output is the product of activation matrix X and weight matrix W. For one input channel, multiply the values in X's channel by 1/s and multiply the corresponding row of W by s. The two factors cancel in the product, so the floating-point layer function is unchanged before quantization. Choosing s greater than 1 shrinks the difficult activation channel and enlarges its associated weight row. SmoothQuant chooses these channel scales from observed activation and weight ranges, with a smoothing parameter controlling how much range moves from X to W.
After rescaling, quantize both activations and weights to the intended 8-bit formats and use a W8A8 matrix kernel. Check the transformed activation range, transformed weight range, layer-output error, full-model quality, and actual kernel dispatch. Excessive scaling can make the weight row difficult to quantize, so the method redistributes error pressure rather than deleting it.
Follow one weight decision through GPTQ
Begin with a weight matrix W and a calibration matrix X containing real inputs to that layer. Replacing one weight with a nearby quantized value changes the layer output WX for every calibration example. An error on a weight connected to inputs that are usually near zero has less output effect than the same numerical error on a weight connected to large or strongly correlated inputs.
GPTQ summarizes this sensitivity with a matrix derived from the calibration inputs. The matrix approximates the curvature, or second-order change, of the layer reconstruction objective near the original weights. In practical terms, the summary tells the algorithm how a quantization error in one weight direction affects output error and how remaining unquantized weights can compensate for part of that error. The phrase approximate second-order information refers to that locally estimated change in reconstruction error; the runtime does not supply an unexplained importance score.
GPTQ processes portions of the weight matrix, selects quantized values, and updates the weights not yet finalized to compensate for the introduced reconstruction error. Group size, processing order, damping, and packing determine both the artifact and its runtime compatibility. Inspect one group before and after the update, then evaluate layer reconstruction and model behavior on data not used for calibration.
Follow activation evidence through AWQ
AWQ also records layer inputs, but it uses the activation observations to find weight channels whose errors have unusually large effects on layer outputs. In this context, a salient weight is not merely a weight with a large stored magnitude. The associated activation pattern indicates that accurately representing that weight or channel matters disproportionately for the observed workload.
AWQ searches per-channel scaling choices that reduce quantization error for those important weights while keeping a regular weight-only representation. Scaling changes which values share the available integer levels, much as changing the unit on a ruler changes where rounding occurs. The scale is folded into the execution path so the selected weights do not require a separate high-precision matrix multiplication.
Inspect the activation statistic, chosen channel scale, quantized integer codes, and reconstructed layer output for one protected channel. Then verify that the serving engine consumes the resulting group size and packing through its intended weight-only kernel. Activation evidence comes from a calibration workload, so domain coverage and held-out quality tests determine whether the protection generalizes.
Connect the offline method to an online kernel
The conversion produces packed weights, scales, possible zero-points, group structure, and configuration metadata. The serving engine must select a kernel that understands all of them. If a supported shape falls back to dequantizing a complete tensor or to a generic GEMM, the expected bandwidth or matrix-throughput benefit can disappear.
Kernel support is architecture-specific. Group size, symmetric versus asymmetric mapping, activation dtype, output dtype, tensor-parallel sharding, and matrix dimensions can change dispatch. Record fallback operators and benchmark aligned, narrow, and ragged shapes separately.
Keep numerical and systems results separate until the final comparison. First show that the artifact meets the quality requirement. Then show that the runtime uses the intended packed kernel. Finally, measure memory, prefill, decode, concurrency, and goodput. A strong method without a suitable kernel is a quality result, not yet a serving optimization.
Quantized model conversion is a release process. Record source weights, calibration data version, conversion code, numerical settings, packing format, and checksums. Validate every shard after conversion and loading. Artifact validation prevents an unexplained quality or performance change from being attributed to the quantization method when the artifact or loader changed.
Worked example: Handling an activation outlier channel
One channel has magnitude 20× typical channels. Per-tensor INT8 activation quantization gives ordinary values poor resolution.
- Measure output sensitivity and confirm the channel appears consistently in representative data rather than as a corrupted sample.
- Use per-channel scaling if the target kernel supports it, or apply an equivalent rescaling that moves some range into the associated weight channel.
- Quantize and measure layer output error, model quality, scale metadata, and transformed weight distribution. Check that the new weight range remains quantizable.
- Benchmark the actual fused GEMM path. Extra scales or a high-precision exception path must not overwhelm bandwidth saved.
Conclusion: Rescaling can reduce the activation channel's range while increasing the associated weight channel's range. The method succeeds only if activation error falls, transformed weights remain quantizable, model quality passes, and the serving kernel performs the scaling through an efficient supported path.
Common errors
- Calibrating on convenient random inputs. Activation ranges depend on real tokens, domains, and sequence behavior.
- Optimizing weight error alone. Downstream activation and task sensitivity determine which errors matter.
- Selecting a published method without verifying a native kernel for the exact format and granularity.
Reference summary
SmoothQuant applies an equivalent channel scaling that reduces activation outliers and transfers range into weights, supporting W8A8 execution. GPTQ uses approximate second-order information from layer inputs to quantize weights while compensating later choices. AWQ uses activation observations to identify salient weights and applies scaling that reduces their weight-only quantization error.
SmoothQuant, GPTQ, and AWQ solve different numerical problems. Each result must still be packed into a representation supported by the target runtime. Calibration data, group size, conversion settings, artifact checksums, and kernel dispatch belong in the release record.
- Calibrate on representative domains, prompt lengths, modalities, and activation ranges.
- Evaluate layer reconstruction only as a diagnostic; measure end-task and generative quality.
- Check rare capabilities and long-context behavior, not only average perplexity.
- Separate algorithm accuracy from the exact runtime kernel/packing format.
- Record conversion reproducibly; quantized artifacts are derived model releases.
Knowledge check
What problem does SmoothQuant move between activations and weights, and why can that help hardware efficiency?
Show the answer guide
- Activation quantization is difficult when a few input channels have much larger magnitudes than the rest. One scale for the group must cover the outliers, leaving too few integer levels for ordinary activation values.
- For a linear layer Y = XW, SmoothQuant chooses a per-channel scale s and rewrites the equivalent computation as (X divided by s) times (s times W). The transformation reduces activation-channel range and moves the corresponding magnitude into weight channels without changing the exact floating-point product apart from rounding.
- Weights are fixed after calibration and can be quantized or packed offline with channel-aware scales. The transformed activations then have a smoother dynamic range that is more suitable for efficient integer kernels at runtime.
- The method helps hardware efficiency when it enables a supported INT8 activation-times-weight path and avoids costly outlier handling. Excessive migration can make weight quantization error worse, so calibration selects the balance and validation measures accuracy.
Move an activation outlier into the weights
Create a linear layer with input shape [256,1024] and output width 4096. Construct calibration inputs whose channels 17 and 603 have magnitudes 50 times larger than typical channels. Implement the SmoothQuant rescaling identity, then compare naive per-tensor INT8 activation quantization with the transformed INT8 activation and weight path across at least five smoothing strengths.
Evidence to keep: Submit the program, calibration procedure, per-channel maximum plots before and after smoothing, tests that verify floating-point equivalence before quantization, an error table after quantization, and a kernel benchmark on supported hardware or a clearly labeled operation-and-byte model. Identify the smoothing strength that best satisfies declared error and latency constraints.
Each quantization method produces a model candidate. Compare candidate quality, latency, throughput, memory use, and hardware support under the deployment constraints.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.