Levels of performance evidence
From one application symptom, select the measurement scope that contains the unexplained time and explain why a narrower tool would be premature
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
The recurring 7-billion-parameter model server processes one 2,048-token prompt on an 80 GiB GPU. A request log reports 121 milliseconds to the first generated token, while device monitoring reports GPU work during only 35 percent of the same request interval. The request may wait in a scheduler queue, central processing unit code may prepare each step, copies may serialize, kernels may be too small, or one long kernel may use a GPU resource inefficiently.
Capture one stable replay with application timestamps and Nsight Systems. The profiler records CUDA API calls on host-thread rows and CUDA copies and kernel executions on GPU-stream rows. The resulting timestamped trace shows when the host submits work and when the GPU executes it. The system trace cannot, by itself, explain which native instructions make one selected kernel slow.
Profile a selected, important kernel with Nsight Compute only after the system trace identifies that launch as a material part of request time. A kernel profile reports memory traffic, scheduler states, execution-pipeline activity, and compiled resources. Disassembly can later show a spill instruction, but the instruction listing cannot establish user-latency impact without the earlier request and kernel measurements.
The analysis therefore moves from the application to progressively smaller scopes. Each measurement should answer a question created by the previous measurement. This procedure limits profiler overhead, avoids detailed analysis of irrelevant kernels, and maintains the connection between a low-level change and the original performance objective.
Establish the application symptom
Record the output-quality requirement, workload distribution, offered load, queueing policy, latency distribution, throughput, error rate, and machine state. The phrase high latency is insufficient. High time to first token under light load and high queueing delay near saturation have different causes and different fixes.
Use a stable replay or a carefully described production interval. Record warmup, batch sizes, sequence lengths, software versions, clock state, and concurrency. Workload and machine conditions form the comparison contract. A later microbenchmark that uses another shape or load does not directly explain the application symptom.
Identify the critical sequence for the objective. In a single request, some operations execute in parallel and do not contribute their full duration to latency. In a throughput test, resource saturation and queueing can matter more than one request's path. State whether the analysis concerns latency, throughput, cost, or another objective before selecting the next tool.
Use a timeline to locate execution and waiting
A system timeline places CPU API calls, runtime activity, copies, kernels, collectives, and synchronization on a common time axis. First mark intervals in which the GPU executes work and intervals in which it does not. Then correlate gaps with host activity and dependency edges.
A repeated 1.8 ms gap before every decode step suggests a host or dependency problem. A continuous sequence of kernels with one family consuming most critical time suggests a kernel or algorithm problem. A long copy followed by an idle compute engine suggests data-placement or overlap work. The timeline observations select the next measurement scope; the observations are not final diagnoses.
Framework traces can explain the CPU portion of a gap. Graph breaks, dynamic shape logic, allocation, logging, or scalar device reads may return execution to Python or force synchronization. A kernel profiler cannot explain time during which no kernel is running.
Use kernel and instruction evidence only when required
Profile a kernel when the timeline establishes that it contributes enough critical time. Begin with duration, operation count, memory traffic, launch geometry, and applicable roofs. Then use scheduler, warp-state, and source-correlated metrics to distinguish insufficient parallelism, data movement, dependencies, barriers, and instruction-pipeline pressure.
Inspect PTX or SASS when a precise compiler question remains. Examples include whether a vector load was emitted, whether matrix instructions are present, whether an array spilled to local memory, or whether tail handling generates a costly branch. Native instructions answer these questions, but they still require timing and profiler context to establish impact.
Each step should produce a falsifiable statement. Replace 'the CPU is slow' with 'a graph break causes 1.8 ms of host dispatch before every decode step.' Replace 'the kernel is memory-bound' with 'three intermediate tensors add 240 MB of device-memory traffic, and measured duration is within 8 percent of the sustainable bandwidth bound.' A specific statement predicts what must change after the fix.
Worked example: Finding an idle-GPU latency problem
Requests have high latency, yet monitoring says the GPU is active only 35% of the time.
- Confirm the application symptom under a stable replay: queue time, time to first token, time per output token, errors, and offered load.
- Capture a system timeline. Observe repeated millisecond gaps between small kernels while a CPU thread performs shape logic and launches operations.
- Correlate the gap with framework traces and discover graph breaks that return execution to Python. Kernel profiles would not explain time when no kernel is running.
- Remove one graph break or capture a repeated region, then verify that gaps shrink, device duty cycle rises, and end-to-end latency improves without changing outputs.
Conclusion: Framework and timeline evidence identified the cause. SASS analysis was unnecessary because native instructions could not explain the host synchronization interval.
Common errors
- Beginning with the most sophisticated tool. Detail is valuable only after scope identifies where detail belongs.
- Collecting every metric at once. Profiler replay and overhead can perturb the workload and bury the handful of counters tied to the hypothesis.
- Omitting end-to-end validation. An improved instruction sequence must also improve the enclosing request or training step.
Reference summary
Each profiling view has a scope. Begin with the view that contains the unexplained time or resource cost. Move to a smaller scope only after the current measurement identifies a specific operation that matters.
- Application: quality, latency distribution, throughput, failures, cost, and workload mix.
- System timeline: CPU launch gaps, copies, synchronization, overlap, collectives, and kernel sequence.
- Kernel profile: memory traffic, pipeline utilization, achieved occupancy, warp states, and roofline position.
- Source correlation: which lines generate expensive instructions or traffic.
- PTX and SASS: instruction selection, address arithmetic, spills, predication, and architecture-specific behavior.
After a local change, repeat the application measurement. The lower-level mechanism explains the change; the application result establishes whether it improved the original objective.
Knowledge check
Which tool level would you use first for low GPU duty cycle with high request latency, and why?
Show the answer guide
- Begin at the request and system-timeline level with server timestamps and Nsight Systems because low GPU duty cycle can result from queueing, CPU launch gaps, copies, synchronization, or absent work rather than a slow kernel.
- A detailed kernel profiler explains only intervals in which the selected kernel runs; it cannot account for time when no kernel is active or when the request waits elsewhere.
- After the timeline identifies a repeated critical-path kernel, use Nsight Compute or an equivalent kernel profiler to test instruction, traffic, readiness, and residency hypotheses.
Move from system timing to kernel evidence
Capture one slow request or training step with application timestamps and Nsight Systems. Select a repeated critical-path kernel only after accounting for CPU, copy, queue, and synchronization intervals, then profile that kernel separately.
Evidence to keep: Annotated request interval, saved system trace, selected kernel and justification, one kernel report, and a statement of what each artifact can and cannot establish.
When application and timeline measurements identify a limiting kernel, a kernel profile can test whether compute, memory, dependencies, or parallelism explain its duration.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.