Introduction to
AI Systems
Performance Engineering
From accelerator architecture to production inference
A technical textbook about how to use hardware, memory, networks, energy, and time to run AI workloads efficiently and reliably.
Begin reading Lesson 0.1 →One prompt passes through many systems
A person enters a prompt in an application and presses Send. A web server receives the request. A tokenizer converts the text into integer identifiers that the model can process. A request scheduler decides when a model worker can begin. The worker asks one or more graphics processing units (GPUs) to run small parallel programs called kernels. The kernels read model weights and temporary values from GPU memory, perform matrix and vector operations, and produce the numbers used to choose the next token. The service repeats part of this process until the response is complete.
Every stage can add delay or limit capacity. The request can wait in a queue before the GPU has space. A GPU kernel can spend most of its time moving data rather than performing arithmetic. Several GPUs can reach a synchronization point at different times and force the faster GPUs to wait. The model can also produce an incorrect answer after an optimization changes numerical behavior. AI systems performance engineering studies the complete path and determines which specific mechanism explains the observed result.
Engineers learn about the path through measurements. For example, NVIDIA Nsight Systems or the PyTorch Profiler can record a trace while a program runs. A trace is a timestamped record of events such as functions running on the central processing unit (CPU), commands sent through NVIDIA's GPU programming platform called CUDA, GPU kernel executions, and data transfers. A timeline view draws those recorded events as horizontal intervals, which lets an engineer see when work ran, what overlapped, and where nothing was running. Later chapters explain how the software creates those events, what the rows mean, and which conclusions the evidence can and cannot support.
The book follows the same request path from high-level behavior down to hardware and back again. Early chapters introduce measurement, units, tensor operations, and the parts of a GPU. Middle chapters explain CUDA programs, memory traffic, matrix multiplication, attention, compilers, and communication between machines. Later chapters return to the model service and explain request scheduling, cached state, precision, reliability, and cost. Each new term appears inside a concrete workload before the lesson depends on the abstraction.
Prerequisites
The main path assumes that you have written programs, can follow ordinary algebra, and understand basic ideas such as functions, arrays, loops, and processes. The first chapters do not assume that you know CUDA, GPU architecture, machine-learning compilers, distributed training, model serving, or performance-analysis terminology.
Every subsection lists the knowledge that its explanation uses. A prerequisite names an earlier lesson when the required idea is taught in the book. Readers who arrive through a direct link can use those references to recover missing context. Many exercises work on a CPU, a modest GPU, or a paper model; access to a frontier-scale cluster is not required.
Study method
- Follow the concrete system first. Identify the program, request, tensor, machine, and measurement tool in the opening narrative. Draw the path that data and control follow before memorizing terminology.
- Connect evidence to a mechanism. Ask who recorded each event or number, where the value appears, and which physical or software action could produce it. Separate direct observation from an explanation inferred from the observation.
- Build the quantitative model. Write every quantity with units. Predict a lower bound, a direction of change, or a visible signature before examining the final measurement.
- Reproduce and vary the example. Change one condition, collect the named evidence, and explain whether the observation supports the proposed mechanism. Record correctness results as well as performance results.
Table of contents
Introduction
Define the field, its objectives, and its methods.
Foundations
Quantitative models, measurement, numerics, and model work.
- 01Performance Models and MetricsUse units and four small models to predict a GPU program's minimum execution time, how much a faster GPU interval reduces complete request latency, queue growth under load, and the rate permitted by HBM traffic.Read →
- 02Performance Measurement and BenchmarkingDesign benchmarks that represent the target workload, account for asynchronous execution, preserve correctness, and support a specific engineering decision.Read →
- 03Numerical Formats and Transformer WorkloadsBuild a resource model for floating-point arithmetic, transformer layers, training state, and the distinct prefill and decode phases of autoregressive inference.Read →
Single GPU Systems
GPU architecture, CUDA, memory, and profiling.
- 04GPU and Accelerator ArchitectureLearn how GPUs schedule warps, supply execution pipelines, accelerate matrix operations, and move data through the memory hierarchy.Read →
- 05CUDA Execution and SynchronizationMap logical work to CUDA threads, calculate resource-limited residency, establish synchronization invariants, and express dependencies across asynchronous operations.Read →
- 06GPU Memory OptimizationCalculate memory transactions, use shared memory for reuse and rearrangement, control register pressure, and evaluate layouts across operator boundaries.Read →
- 07GPU Profiling and Instruction AnalysisMove from an application symptom to timeline, kernel, source, and instruction evidence, and document a reproducible causal result.Read →
GPU Kernels and Compilers
Parallel primitives, matrix multiplication, attention, integration, and lowering.
- 08Parallel Reduction, Scan, and HistogramsLearn warp and block collectives, hierarchical reductions, scans, histograms, and the cost of irregularity.Read →
- 09Matrix Multiplication (GEMM)Derive blocked matrix multiplication, multi-level tiling, software pipelines, epilogues, and the constraints of matrix engines.Read →
- 10Kernel Fusion, Softmax, and AttentionDerive fusion economics, online softmax, IO-aware attention, and the boundaries where recomputation beats memory traffic.Read →
- 11Triton, CUTLASS, and PyTorch OperatorsChoose the right kernel authoring level, integrate custom operations correctly, and make them composable with autograd and compilation.Read →
- 12ML Compiler Architecture and OptimizationUnderstand tracing, guards, graph breaks, intermediate representations, dialects, lowering, fusion, code generation, and debugging.Read →
Distributed Systems
Networks, collective communication, parallelism, diagnosis, and recovery.
- 13Network Topology and Collective CommunicationTrace communication through the physical cluster, derive latency and byte costs for collective algorithms, and diagnose NCCL execution from rank and topology evidence.Read →
- 14Distributed Model ParallelismDerive the state ownership, local tensor shapes, communication, memory, and scheduling effects of each parallel dimension before composing a physical rank mesh.Read →
- 15Distributed Performance AnalysisReconcile utilization and scaling metrics with wall time, align causality across ranks, measure exposed communication and bubbles, and reduce a large-scale regression to a controlled experiment.Read →
- 16Fault Tolerance and CheckpointingCalculate expected lost progress, design sharded portable checkpoints, distinguish a hang from a slow rank, and allocate resources for long-duration recovery objectives.Read →
Inference systems
Scheduling, memory, precision, speculation, and economics.
- 17Inference Execution and MetricsTrace a request from arrival through streaming, calculate fixed and per-token memory, distinguish prefill from decode, and measure sustainable online capacity.Read →
- 18Request Scheduling and KV Cache ManagementRe-form batches at iteration boundaries, allocate KV state in physical blocks, reuse valid prefixes, and control long-prefill interference with explicit service policies.Read →
- 19Quantization and Speculative DecodingSpecify quantized representations and kernels, evaluate post-training methods under quality constraints, and calculate speculative-decoding progress from complete cycle cost.Read →
- 20Production Inference SystemsDefine useful service outcomes, connect them to causal telemetry, provision constrained goodput with reserve, and keep overload and rollout behavior bounded.Read →
Practice
Laboratory method, capstones, technical writing, and further study.