Chapter 7

GPU Profiling and Instruction Analysis

Select evidence from application metrics to native instructions

Move from an application symptom to timeline, kernel, source, and instruction evidence, and document a reproducible causal result.

A decoder-only model server runs the recurring 7-billion-parameter model on one 80 GiB graphics processing unit (GPU). Under a fixed replay of 2,048-token prompts, time to first token rises from 78 milliseconds to 121 milliseconds while GPU-active time falls to 35 percent. The change could come from queueing, central processing unit (CPU) dispatch, memory copies, synchronization, or slow GPU kernels; one utilization percentage cannot distinguish the causes.

NVIDIA Nsight Systems captures a timestamped event record while the replay runs. The record is a trace. The timeline view places CPU API calls, CUDA memory copies, CUDA kernel executions, and stream synchronization on horizontal rows with start and end times. PyTorch Profiler can add framework operator names and tensor shapes. NVIDIA Nsight Compute separately profiles one selected kernel and reports hardware counters; the detailed collection can replay the kernel and perturb elapsed time.

The investigation narrows only after an observation identifies the next scope. A repeated 1.8-millisecond gap before every decode kernel directs attention to the host thread and dependency graph. A single long kernel that dominates critical time justifies a kernel profile. Parallel Thread Execution (PTX) and native Streaming Assembler (SASS) instructions are inspected only when a concrete compiler question remains.

A causal report names the workload, measurement boundary, mechanism, kernel or subsystem result, end-to-end effect, and limits. Every reported rate states the completed quantity divided by elapsed time. A capacity claim also states the service-level objective (SLO) that qualifies a request as useful.

Prerequisites

Chapters 4–6 hardware, execution, and memory models
Chapter 2 benchmark contract and asynchronous timing
Ability to reproduce one request or training step with fixed shapes and load

Learning objectives

Choose application, system-timeline, kernel-profile, or native-instruction evidence from a specific unanswered question
Interpret throughput, occupancy, and warp-state counters with explicit numerators and denominators
Write a reproducible causal performance report from correctness through end-to-end effect

Lessons

Study this chapter

Chapter projects

Extended exercises

Complete these projects after the lesson-level practice tasks. Each project combines several mechanisms from the chapter.

Check

Timeline diagnosis

Capture a fixed request or training step with Nsight Systems and identify one idle GPU interval or serialization that repeats at least three times.

Evidence: An annotated timestamped profiler record naming the producer, consumer, stream or host dependency, and milliseconds added to the critical path.

Practice

Kernel case study

Select one kernel already shown to matter, then profile its shape, physical work, storage traffic, applicable ceilings, and scheduler readiness.

Evidence: One falsifiable hypothesis supported by at least three mutually consistent measurements and rejected by at least one controlled alternative.

Challenge

Disassembly analysis

Use optimized-build SASS to answer one unresolved question about vector loads, spills, matrix instructions, or tail control flow.

Evidence: Preserved binary and flags, source-correlated before-and-after instructions, predicted counter change, correctness result, and separate CUDA-event timing.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.