Speculative decoding

Follow one proposal and verification cycle

A draft model or other proposal mechanism generates k candidate tokens autoregressively and records the proposal probabilities needed by the acceptance algorithm. The target model evaluates the candidate positions together. Because the full proposed prefix is known, target logits for several positions can be computed in a more parallel pass than k separate target decode calls.

The acceptance test proceeds from the first proposed position. In classical exact speculative sampling, a token can be accepted according to a probability ratio derived from target and draft distributions. At the first rejection, later draft tokens are not committed, and a correction token is sampled according to the method. If every proposal is accepted, the target can provide an additional token. Greedy variants use a corresponding equality or verification rule.

The previous progress rule is not universal. Tree-based proposals, multiple draft heads, relaxed verification, and other speculative algorithms can expose different numbers of candidate positions and use different commit rules. State the algorithm before defining accepted tokens, bonus tokens, or speedup. Distribution preservation applies only when the selected acceptance and correction rule provides it; an approximate rule requires a separate quality evaluation.

Only the accepted prefix and required target token become visible output. Target and draft KV state must commit the same accepted positions and discard or overwrite speculative suffix state. Cache commit and discard operations must be coordinated with paged blocks, batch membership, cancellation, and any prefix sharing.

Calculate progress and cycle cost

Let P be the number of committed tokens from a cycle. The no-speculation baseline requires approximately P serial target steps. The speculative cycle costs draft time D(k), target verification time V(k), and overhead H for acceptance, sampling, cache commit, and scheduling. An initial speed estimate is P / (D + V + H) when times are normalized to one target step.

For one simple model of classical speculative sampling, assume each draft position has the same conditional acceptance probability alpha and proposal length k. Progress is at least one token and at most k + 1 tokens. Its expectation is 1 + alpha + alpha^2 + ... + alpha^k, which equals (1 - alpha^(k + 1)) / (1 - alpha) when alpha is not one. Real acceptance events are position- and context-dependent, so measure the complete progress distribution instead of fitting every workload to one alpha.

Longer windows increase maximum progress but make more work depend on early proposals. If token two is rejected in an eight-token proposal, later draft generation and much verification work produce no committed progress. A shorter window gives up some maximum progress but can reduce waste and produce a more regular batch shape.

A more accurate draft raises acceptance but can raise D enough to erase the benefit. V(k) is not automatically the cost of one ordinary target decode step: verification has a longer query dimension, writes speculative KV, and can select different kernels or graph variants. Draft and target tokenizers or vocabularies can also complicate verification in some systems. Measure accepted-token count by proposal position, full cycle time, verification shape, draft cost, and commit or rollback bytes. Tune k by workload rather than by one mean acceptance number.

Integrate speculation with concurrent serving

Concurrent requests can have different accepted prefix lengths. After one verification batch, some requests advance one token while others advance several. Variable progress changes fairness, next-iteration membership, KV allocation, and the meaning of a sequence budget. A scheduler may need to budget speculative tokens separately from committed progress.

Draft execution can share the target GPU, use another GPU, run on the CPU, or use an integrated draft head. Sharing one GPU adds competing work but avoids transfer and extra hardware. A separate device can overlap some work but adds communication and cost. The best topology depends on target utilization and batch scale.

Speculation can improve single-request latency but lower server throughput if draft work occupies resources that ordinary batching would use more efficiently. Run open-loop load tests with speculation on and off. Compare TTFT, TPOT distribution, output goodput, draft and target utilization, cache memory, and quality-equivalent outputs.