Cross-layer performance analysis
Trace one named quantity with stable units from model semantics to an end-to-end service or training metric
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- A reproducible single-device measurement workflow
- Tensor graphs, compiler inspection, and kernel profiling
- Basic distributed ownership or serving-scheduler models
The recurring 7-billion-parameter decoder stores attention history for every active request. Model code describes the history as key and value vectors for each retained token. The serving engine allocates fixed-size blocks in GPU memory, the attention CUDA kernel follows a block table to load the vectors, and the request scheduler decides whether enough free blocks remain to admit another prompt. One choice—the number of tokens stored in each block—changes behavior at all four locations.
A memory log from the serving engine can show unused entries in each request's final block. NVIDIA Nsight Compute can show extra load instructions or changed cache traffic when blocks become smaller. Scheduler logs can show how many requests were admitted and how long they waited. A load generator can finally show whether usable capacity improved while time per output token stayed below the declared limit. No single report contains the complete explanation.
A cross-layer study follows one named quantity through those representations. The worked investigation follows key-value (KV) bytes per retained token from the model architecture, through allocator blocks and kernel addresses, into admitted-request capacity. The study holds model output and request distribution constant, predicts each local effect before measurement, and stops after the quantity reaches the end-to-end service objective.
Select one quantity and one end-to-end objective
Choose a quantity that can be defined at every relevant layer. For attention, possible quantities include score-matrix bytes, KV bytes per token, accepted output tokens, or serial decoding steps. For mixture-of-experts routing, useful quantities include tokens per expert, padding work, expert weight bytes, and all-to-all payload. Write the unit and definition once, then record how each layer transforms or partitions it.
Pair the quantity with an objective. Reducing KV allocation is not sufficient if page translation increases TPOT beyond the service limit. Reducing kernel time is not sufficient if a compiler graph break adds more host delay. The study should end at a metric such as valid training steps per hour, requests per dollar under a p99 latency constraint, or peak supported context at a declared concurrency.
Specify invariants that optimization cannot violate. Required invariants include model outputs or quality, attention masks and position semantics, numerical tolerance, request fairness, failure behavior, and memory safety. Cross-layer changes can accidentally alter semantics because one layer treats a representation choice as an implementation detail while another layer relies on it.
Create a representation table before implementation
For each layer, record the name, owner, shape, layout, datatype, lifetime, and storage location of the selected quantity. At the model layer, a KV cache may be a conceptual sequence of key and value vectors. The framework represents tensors and views. The allocator represents page identifiers and free lists. The kernel sees block tables, strides, and vector loads. The scheduler sees capacity reservations and active requests.
Add conversion interfaces to the table. State when a logical view requires a physical copy, when a datatype conversion occurs, when a tensor crosses devices, and when state changes ownership. Copy, conversion, device-transfer, and ownership interfaces are common sources of traffic and synchronization. The same interfaces are also places where one layer can lose information that another layer needs.
Mark version-sensitive details separately from semantic facts. A current compiler might recognize one layout pattern, while the operation's mathematical layout requirement remains stable. This separation lets the study survive a software update and shows which experiments must be repeated.
Trace cause and effect across layer boundaries
Change one representation or policy and predict its immediate effect at each later layer. A smaller KV page can reduce tail allocation waste. A smaller page also increases block-table entries, can create more address calculations in the attention kernel, and can change scheduler admission capacity. Write every predicted effect before measurement so the study does not stop after observing the first improvement.
Look for missing information. A compiler cannot preserve a layout it does not know. A kernel cannot exploit prefix sharing if requests present independent pointers without a sharing contract. A scheduler cannot avoid a weak GEMM shape if it does not know the kernel's performance regions. The required change may be metadata, a dispatch predicate, or a jointly selected representation rather than a lower-level optimization.
Use controlled ablations to isolate each link. Hold allocator policy fixed while changing kernel page handling. Hold kernel implementation fixed while replaying different scheduler policies. Replace an allocator timing, kernel-transfer measurement, or scheduler decision with a calibrated simulation only when the physical experiment is unavailable, and label the substituted result. The causal account becomes credible when each measured transition agrees with its local prediction and the combined result explains the end-to-end change.
Combine direct measurements with bounded models
A complete production environment is often unavailable. Measure the mechanisms that fit available hardware and model the remaining scale. One GPU can measure page-translation cost and cache traffic. A simulator can replay a timestamped request-event record to evaluate allocator fragmentation over millions of arrivals and completions. A two-node test can validate collective ordering and a bandwidth regime. All three experiments can share one recorded request-event file and parameter table.
Calibrate every modeled parameter from a source or measurement. State whether bandwidth is peak, isolated achieved, or observed under contention. State whether request traces are synthetic or sampled from a deployment. Sweep uncertain parameters and report when the conclusion changes. A model that depends on one unverified large-scale constant should not be presented as a measured production result.
Define the boundary of inference. For example: the experiment verifies that smaller pages reduce allocation tail waste and add a measured amount of kernel overhead on one GPU. The simulation estimates the resulting capacity under a declared workload. The simulation does not verify multi-node congestion or production failure behavior. The scope statement preserves the study's value without overstating its scale.
Stop the study when the causal chain is complete
A cross-layer project expands easily because every boundary exposes another system. Use stop conditions tied to the question. Stop adding layers when the selected quantity reaches the end-to-end objective and remaining layers cannot change the decision within the declared range. Record excluded effects as limitations or follow-up questions.
The final artifact should include the representation table, local models, ablation results, and end-to-end synthesis. Reuse one set of identifiers for shapes and workload cases across code, data, plots, and prose. A reviewer should be able to follow one case from mathematical input through the final service or training metric without translating inconsistent names.
Depth comes from explaining the dependencies among layers. A deep study does not require implementing every layer. A study that uses a mature kernel, a measured allocator, and a calibrated scheduler simulation can be complete when the selected mechanism and inference boundaries are explicit.
Worked example: Following KV cache through the stack
A study asks how cache block size affects useful serving capacity.
- Derive KV bytes per token from model architecture and precision. Define attention correctness and position semantics.
- Create a table for page sizes 8, 16, 32, and 64 tokens. Predict internal tail waste, block-table entries, and the number of cache blocks read for representative context lengths.
- Design the logical-to-physical block table and allocator. Model tail waste and sharing.
- Inspect the attention kernel's address translation, vector loads, and locality for several block sizes.
- Use one declared request record to drive both allocator simulation and kernel shape generation. The record must contain each request's arrival time, prompt length, requested output length, and completion event. Shared input records prevent the allocator and kernel studies from evaluating unrelated length distributions.
- Replay request lengths and scheduling under cache pressure. Report goodput, TPOT, allocator cost, and memory headroom.
- Ablate prefix sharing while keeping page size fixed, then change page size while keeping sharing fixed. The paired runs separate the effects of the two policies.
Conclusion: One parameter—block size—now connects model state, allocator fragmentation, kernel access, and admission policy. The study demonstrates systems reasoning rather than isolated tuning.
Common errors
- Collecting one project per tool. Tool familiarity without a shared mechanism rarely produces transfer.
- Adding layers merely for breadth. Every layer should answer how the chosen quantity or dependency changes.
- Claiming production behavior from a toy simulator. State which mechanisms were validated and which remain modeled.
- Changing the workload distribution between layer experiments. A causal chain cannot be assembled from incomparable inputs.
- Using one name for different quantities, such as allocated KV bytes and transferred KV bytes. Capacity and bandwidth require separate accounting.
Reference summary
For a key-value-cache study, first calculate the logical payload bytes stored per token by the declared model. Next, map the payload into the allocator's fixed-size blocks and measure reserved capacity, including unused space at block tails. Then measure how many bytes the attention kernel reads during one decode iteration and calculate how the remaining allocated capacity changes scheduler admissions. The quantities are related but are not literally the same bytes. A smaller cache block is useful only when admitted-request capacity improves without violating the declared time-per-output-token limit.
- Begin with mathematical work and numerical invariants.
- Map tensor shapes and lifetimes through the framework graph.
- Inspect or implement the critical kernel and its generated instructions.
- Integrate the path under compilation and representative shapes.
- Measure its distributed or serving consequence under load.
Knowledge check
For key-value caching, how would you distinguish bytes allocated by the cache manager, bytes transferred into a CUDA kernel, and bytes reserved but unusable at the end of a block?
Show the answer guide
- Logical KV payload is calculated from the model: retained tokens multiplied by layers, key and value tensors, KV heads, head dimension, and bytes per stored value. The calculation does not include allocator rounding.
- Cache-manager allocation is the number of physical blocks assigned to the sequence multiplied by bytes per block, plus separately reported block-table or allocator metadata. Allocator logs or block tables provide this quantity.
- Kernel-transfer bytes are the actual reads and writes requested or transferred while an attention kernel processes one declared iteration. Nsight Compute memory counters measure this execution quantity; it can differ from both logical payload and allocated capacity.
- Tail waste for one sequence is allocated capacity in its final block minus bytes occupied by valid retained tokens in that block. Sum this value across active sequences to obtain block-tail waste.
- Reserved but unusable memory must be divided further when necessary: free blocks held as safety reserve, fragmented blocks that cannot satisfy an allocation, and allocator or runtime reservations are not active KV payload.
- All values need the same snapshot or workload interval and explicit units before the study relates smaller tail waste to scheduler admissions, TPOT, or goodput.
Compare two KV block sizes
Build a small allocator table for six sequences retaining `[1, 15, 16, 17, 31, 33]` tokens. Assume 1,024 KV bytes per token and compare blocks holding 16 tokens with blocks holding 8. Calculate logical payload, allocated bytes, final-block waste, and block-table entries for every sequence.
Evidence to keep: Keep the per-sequence table, total payload and allocation calculations, a plot of tail waste by sequence, block-table entry totals, and a written causal conclusion describing the capacity benefit and metadata cost of the smaller block.
A cross-layer result is reusable when the report lets another engineer reproduce the evidence and evaluate the causal conclusion.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- The Rust Programming Language↗
Structural inspiration: progressive chapters, concrete examples, and explicit learning flow.
- CUDA C++ Best Practices Guide↗
- Nsight Compute Profiling Guide↗
- MLPerf Inference Rules↗
- torch.compile Programming Model↗
- PagedAttention / vLLM↗