Inference Execution and Metrics

A user sends a 2048-token prompt to a decoder-only language-model API at 10:00:00.000 and permits up to 256 generated tokens. The model has approximately 7 billion parameters, 32 transformer layers, hidden size 4096, 32 query heads, 8 key-value heads, and head dimension 128. One server GPU has 80 GiB of device memory. The gateway accepts the network connection and authenticates the account. A model server admits the request, converts token identifiers into tensors, and asks a request scheduler to place the prompt into a graphics processing unit (GPU) batch. The GPU processes the complete prompt once, then repeatedly predicts one additional token. A streaming process converts each token identifier into text and sends bytes to the user. Every named stage can add delay or fail.

A server log can record a monotonic timestamp at request arrival, admission, scheduler enqueue, prompt-processing start and finish, each token-generation iteration, the first sampled token, the first streamed byte, and completion. Request and batch identifiers connect the user request to GPU profiler events. The resulting timestamped event record is a request trace; the trace comes from explicit instrumentation in the gateway and model server, not from an unspecified source.

Chapter 17 develops four measurements from that request. A request timeline assigns milliseconds to waiting, prompt processing, repeated token generation, and delivery. A memory calculation accounts for model weights, retained key-value state, workspaces, and reserve. A serving roofline relates each model iteration to bytes moved and arithmetic performed. An open-loop load test sends requests at a declared arrival rate and reveals the point where the completion rate stops keeping pace.

The host central processing unit (CPU) runs gateway, tokenization, and scheduler code, while the graphics processing unit (GPU) performs model arithmetic. Model layers use matrix multiplication (GEMM) and read weights from high-bandwidth memory (HBM). Attention retains key-value (KV) vectors from earlier tokens. Logs report time to first token (TTFT) and time per output token (TPOT), then compare both with a service-level objective (SLO) that names the acceptable request population and latency threshold.

The four measurements answer different questions and must remain connected. A configuration can report high total tokens per second while interactive requests wait several seconds for their first token. Another configuration can have fast GPU kernels but admit few requests because retained attention state exhausts memory. The chapter therefore defines a useful operating point with a workload, latency limits, correctness requirements, and measured completion rate.