Quantization and Speculative Decoding

The chapter continues with the same approximately 7-billion-parameter model used in the serving examples. Its BF16 weight payload uses about 14 GB in decimal units, which is approximately 13.0 GiB, before file headers, alignment, duplicated tensors, or runtime workspaces are added. BF16 stores each value in 16 bits and preserves the 8-bit exponent width used by FP32, while using fewer fraction bits. A converted artifact stores groups of weight values as packed 4-bit integers plus floating-point scale values, reducing the file and GPU allocation. During generation, a GPU kernel must unpack the integers, load the scales, reconstruct suitable values or use a supported narrow matrix instruction, and accumulate the products. A memory report can show a large reduction even when the profiler shows little speed improvement because conversion work or an unsupported kernel path consumes the saved time.

A second optimization addresses the serial nature of token generation. An ordinary decoder runs the large model once for token 1, again for token 2, and again for token 3. Speculative decoding lets a smaller draft model propose several tokens and lets the large target model score the proposed positions together. A server log can record proposed tokens, accepted tokens, draft milliseconds, verification milliseconds, and committed output for every cycle.

Chapter 19 first specifies a complete quantized data path: stored bits, scales, grouping, packing, activation format, accumulator format, output conversion, and the GPU kernel that consumes the representation. The chapter then connects SmoothQuant, GPTQ, and AWQ to the numerical problems each method addresses. Quality measurements precede performance claims because a faster model that no longer satisfies the task requirement does not provide equivalent service.

The host central processing unit (CPU) loads and schedules the artifact, while the graphics processing unit (GPU) runs quantized matrix multiplication (GEMM). Attention may also store key-value (KV) cache entries in a narrower format. The service reports time to first token (TTFT), time per output token (TPOT), and useful completions under a service-level objective (SLO) so smaller storage is not mistaken for a complete performance result.

The final lesson derives speculative-decoding progress from one proposal-and-verification cycle. Draft accuracy alone cannot predict speed. Proposal length, acceptance position, target verification time, draft time, cache updates, and lost batching opportunities all appear in the measured cycle. The service must also preserve the declared sampling distribution or explicitly state that the model behavior changed.