Performance objectives and constraints
Describe one request path and define a measurable objective from named timestamps and completed results
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Ability to follow the request narrative in the chapter introduction
A user submits a 2,048-token prompt to a decoder-only language model with about 7 billion learned parameters. The service allows up to 256 generated tokens and runs the model on one graphics processing unit (GPU) with 80 GiB of device memory. The browser records when it sends the request. The server records when it accepts the request, when it admits the request to GPU work, when it receives the first generated token, and when it sends the final response. Browser and server timestamps are the first concrete evidence in the investigation.
The server's central processing unit converts the prompt text into integer token identifiers and places a request object in the scheduler's waiting collection. A model-worker process turns the identifiers into tensors, submits CUDA kernels to the GPU, and retains attention history in a key-value cache. The GPU kernels load learned weights and activation values, perform matrix multiplications and other operations, and write the next-token scores. The model worker then selects a token according to the declared decoding policy. Many production runtimes perform selection with GPU kernels; the teaching implementation can copy the small result needed by CPU-side response code when that choice is stated explicitly.
Every performance term in this lesson attaches to part of that path. Request latency is elapsed time between two declared timestamps. Queue delay is time between server acceptance and scheduler admission. GPU execution time covers declared CUDA events. Throughput counts completed requests or generated tokens during a declared interval. A useful objective combines such a count with quality, latency, reliability, and resource limits so that a faster-looking measurement cannot silently describe a worse product.
Application and scheduler code write the named log timestamps. Nsight Systems records timed CPU calls, CUDA launches, transfers, and GPU kernels while the request runs. Nsight Compute collects selected hardware-counter values from the GPU while profiling one chosen kernel; examples include bytes transferred from high-bandwidth memory and executed instructions.
- Application
A person submits text. The application sends an HTTP request to the server.
Output: request text and generation settingsBrowser event: request_submitted - Server request preparation (CPU)
Server code validates the request, converts text into token identifiers, and creates a queue entry.
Output: token identifiers and queue recordServer log: tokenization_finished - Request scheduler
The scheduler assigns the sequence a batch position after cache space and admission policy permit it to run.
Output: scheduled batch entryScheduler log: request_admitted - Model worker
Framework and runtime code select numeric operations and submit small parallel programs, called kernels, to the GPU.
Output: ordered GPU commandsNsight Systems: CPU calls and CUDA launches - GPU execution
GPU kernels move weights and temporary values through memory, execute arithmetic, and produce scores for the next token.
Output: next-token scoresNsight Systems: kernel intervals; Nsight Compute: selected kernel counters - Response streaming
The model worker selects a token according to the decoding policy. The server converts the identifier into text and streams it to the application.
Output: generated text tokenServer log: token_sent
Follow one request through the service
The browser and server exchange an HTTP request, but the GPU does not read the text directly. Server code on the central processing unit tokenizes the text and creates an ordered list of token identifiers. The request scheduler owns the request while it waits. The scheduler transfers responsibility to the model worker after the key-value cache has free blocks, the next GPU batch has a position for the sequence, and the admission policy permits the request to enter. Server logs can preserve each ownership change as a named timestamp.
The model worker submits CUDA kernels through a framework such as PyTorch. A CUDA kernel is one GPU program executed by many parallel threads over tensor elements. Large kernels multiply activation matrices by stored model-weight matrices. Smaller kernels normalize rows, apply activation functions, update cache blocks, or select an output token. NVIDIA Nsight Systems can produce a timestamped record containing CPU calls, CUDA launches, memory copies, and GPU execution intervals for the request. Its timeline view draws those recorded events in time order.
Prompt processing and output generation repeat different portions of the path. Prompt processing runs the 2,048 input positions through the model and creates the initial key-value cache. Each decode iteration then advances the request by one generated token, reads the retained attention history, and appends one new cache entry per layer. The browser can receive early tokens while later iterations still run, so time to the first token and time to the complete response answer different user questions.
Draw the path before optimizing. Mark the source of every observation: browser timestamps, server logs, scheduler counters, framework profiler records, CUDA events, and GPU-profiler reports. A gap with no GPU work can be CPU preparation or queue delay. A long GPU interval can be one kernel or a chain of kernels. The source and endpoints prevent one measurement from being mistaken for another.
Define completed work
The numerator must count a result that the system is intended to produce. For offline inference, completed samples can be sufficient when every sample has equivalent cost and the benchmark enforces a quality target. For an interactive decoder, generated tokens are incomplete because a token delivered after the latency limit may not satisfy the service requirement. A better unit can be requests that complete correctly while meeting both time-to-first-token and time-per-output-token limits.
Training requires similar care. Tokens processed per second measures input consumption, not learning progress. A numerically unstable run can process tokens quickly while failing to reach the target loss. Time to a declared quality target is often the more complete measure. If a shorter experiment uses tokens per second as a proxy, the report must state the numerical configuration and provide evidence that convergence remains equivalent.
Identify the constrained resource
The denominator represents a resource whose use matters to the system owner. Wall-clock time matters when a training run must finish before a release or when a user is waiting. Accelerator-hours matter when the objective is fleet capacity. Joules matter under a power or energy budget. Dollars can include accelerator rental, CPUs, host memory, networking, storage, and idle reserve.
A lower value in one resource can require more of another. A service can reduce latency by reserving twice as many accelerators, or reduce device memory by recomputing activations and increasing step time. State the primary resource to optimize and specify hard limits for the other resources. This makes the tradeoff explicit instead of hiding it in the implementation.
Specify workload and operating point
A performance objective applies to a workload population, not to an abstract model name. For inference, record prompt lengths, output lengths, arrival pattern, concurrency, decoding policy, and cache state. For training, record batch construction, sequence lengths, precision, parallel configuration, and optimizer. The recorded workload variables change tensor shapes, reuse, communication, and queue behavior.
The operating point states the load and constraints under which the metric was observed. A service that produces 10,000 tokens per second with an unbounded queue is not equivalent to one that produces 8,000 tokens per second while meeting a p99 latency target. Throughput becomes decision-relevant only when the report also states offered load, latency, errors, and quality.
Separate objectives from diagnostics
A performance objective decides whether a system change is valuable. A diagnostic metric explains a mechanism. Occupancy, FLOP/s, HBM bandwidth, cache hit rate, and link utilization are diagnostic metrics. They become relevant after a hypothesis connects them to the objective.
Higher utilization is not automatically better. A device can reach high utilization because a long batch blocks short requests. A kernel can report more FLOP/s because it performs redundant arithmetic. A scheduler can increase token throughput by delaying batch formation. In each case, the diagnostic number improves while the user-visible or economic result can become worse.
Worked example: Defining the objective for an interactive decoder
A service answers coding questions. At the current production arrival distribution, one GPU completes 420 output tokens per second. The p99 time to first token is 1.4 seconds, and the p99 time per output token is 170 milliseconds. The operations team keeps 35% capacity in reserve to absorb bursts.
- Define one completed unit as a response that passes the existing task-quality evaluation, contains no malformed tool call, and finishes without an internal error. Count a retried request once, but include all resources consumed by its failed attempts.
- Set service constraints from product requirements: p99 time to first token below 800 milliseconds and p99 time per output token below 80 milliseconds at the recorded arrival distribution. State prompt- and output-length distributions because both affect the amount of work.
- Choose successful responses per dollar of provisioned accelerator time as the main objective. Provisioned time includes idle replicas and safety headroom because those devices are required to meet the latency objective during bursts.
- Measure candidate schedulers across increasing offered load. For each point, report completed responses, latency percentiles, errors, reserved capacity, and quality. Stop using an operating point when its queue grows continuously or its latency exceeds the stated limits.
- Use batch occupancy, KV-cache allocation, device bandwidth, kernel duration, and scheduler wait time to explain the result. The diagnostic measurements can show whether a scheduler improves matrix efficiency, reduces queue delay, or admits more concurrent sequences.
Conclusion: This specification can prefer a scheduler with lower offline token throughput if it completes more valid responses per dollar while meeting the service limits. The conclusion is limited to the declared workload and operating point.
Common errors
- Using device utilization as the objective. Utilization describes device activity. Device activity does not state whether completed work satisfies quality and latency requirements.
- Comparing token rates without defining token type. Prompt tokens and generated tokens execute different workloads, and different tokenizers can produce different counts for the same text.
- Leaving output quality implicit. Precision changes, approximate kernels, altered decoding settings, and shorter generation limits can increase speed by changing the produced result.
- Excluding provisioned but idle capacity from cost. Latency headroom is part of the deployed system when it is required to satisfy the service objective.
Reference summary
For the recurring 7-billion-parameter inference service, the browser creates a request; the server admits and tokenizes it; the scheduler assigns it to the model worker; CUDA kernels run on one 80 GiB GPU; and the server returns generated tokens. Browser timestamps, server logs, scheduler records, and CUDA events measure different intervals along the same path.
AI systems performance engineering asks how much time, device capacity, memory, network transfer, energy, and engineering work the service requires to produce an accepted result. A rate in floating-point operations per second describes GPU arithmetic. The arithmetic rate alone cannot show whether a request met its latency limit, preserved model quality, or required excessive idle capacity.
performance objective = useful completed work / constrained resource- Quality constraint: specify the evaluation, tolerance, or task-quality limit that an optimization must preserve.
- Latency objective: specify the interval, population, and required percentile. The p99 is the value at or below which 99% of measured observations fall; it describes a different part of the distribution from the median.
- Throughput objective: specify the completed work, the offered load, and the operating constraints under which the rate is valid.
- Reliability constraint: specify which failures count, how retries are counted, and how much recovery time is permitted.
- Cost or energy objective: include provisioned but idle capacity and any additional resources required to meet latency and reliability targets.
Knowledge check
For the request path in this lesson, which timestamps define time to first token, which event qualifies a request as completed work, and which quality condition prevents a faster but worse answer from counting as an improvement?
Show the answer guide
- Client-observed time to first token (TTFT) starts when the browser sends the request and ends when the browser receives the first generated token or first response bytes. A server-internal TTFT can instead start at server acceptance and end when the model worker produces the first token, but the report must not mix the two boundaries.
- A request qualifies as completed work only after the final response is delivered successfully and the request has no internal failure. Retry handling must prevent one user request from being counted twice while still charging resources used by failed attempts.
- The response must pass the declared task-quality evaluation and any structural requirement, such as a valid tool call. Fix the model, decoding policy, and output limit so that a faster but lower-quality or truncated answer does not enter the useful-work numerator.
- The completion count is meaningful only for the declared prompt and output distributions and while the TTFT and time per output token (TPOT) limits remain satisfied.
Annotate one request record
Create a six-event record for one 2,048-token request: browser send, server acceptance, scheduler admission, first model token, first browser byte, and final browser byte. Assign plausible monotonic timestamps, mark whether the response passed a named quality check, and calculate client TTFT, server-internal TTFT, queue delay, and end-to-end latency.
Evidence to keep: Keep the timestamp table, all four interval calculations with units, the completion and quality classification, and a short statement naming which interval represents the user's experience.
A defined objective tells you what improvement means. The next section explains how to investigate which mechanism currently limits that objective.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- The Rust Programming Language↗
Structural inspiration: progressive chapters, concrete examples, and explicit learning flow.
- NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.
- PyTorch Profiler documentation↗
Official reference for recording operator activity, CPU and accelerator events, shapes, stacks, and exported traces.
- MLPerf Inference Rules↗
- MLPerf Training Benchmark↗