Request Scheduling and KV Cache Management

At 10:00:00, eight requests are generating text on one model server. Seven requests will finish after 32 more tokens, while one request will continue for 512. Two new prompts arrive while the GPU is busy. After every forward pass, the server knows which requests finished, which need another token, how much key-value (KV) state each request owns, and how many prompt tokens are waiting. The server must choose the exact rows of the next GPU batch.

A scheduler decision log can record the iteration number, selected request identifiers, prefill-token count, decode-token count, free KV blocks, preemptions, and queue age. An Nsight Systems timeline can show whether host scheduling leaves a gap before the next GPU launch. A request-level latency log can show whether long prompts interrupt the cadence of active token streams. The three views make scheduling policy observable rather than abstract.

Chapter 18 explains continuous batching, paged KV storage, prefix reuse, and chunked prompt processing as one connected mechanism. Continuous batching changes membership after a model iteration. A paged allocator grows each request's attention state in fixed-size blocks. Prefix caching lets requests reuse identical earlier computation. Chunked prefill divides a long prompt into schedulable units. Every mechanism changes GPU shapes, memory ownership, and request waiting time.

The host central processing unit (CPU) runs the scheduler, while the graphics processing unit (GPU) executes matrix multiplication (GEMM) and attention for the selected batch. Request logs report time to first token (TTFT) and time per output token (TPOT). A scheduling policy must preserve the declared latency limits for those metrics while it assigns KV blocks and GPU work.

Each lesson uses a state table for one iteration: which requests advance, how many tokens they contribute, which KV blocks they own, and which requests remain queued. The state table gives concrete meaning to admission, preemption, capacity, and fairness. A policy is complete only when the table and the resulting latency and throughput measurements can be reproduced.