Study method and evidence
Produce a versioned experiment record with calculations, raw observations, correctness results, and a bounded conclusion
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Ability to write a prediction, identify its evidence, and state what observation would reject it
A learner copies a tiled CUDA matrix-multiplication example and observes a 2.3-fold speedup for a 4,096 × 4,096 matrix. The copied program runs correctly, but the result does not yet explain why the tiling helped, why a 64 × 64 matrix became slower, or why a production library still runs faster. Answering those questions requires a traffic calculation, measurements across several shapes, and inspection of the code that the compiler generated for the GPU.
Every lesson in this book asks you to construct such an explanation. A lesson begins with a named workload and an observable symptom, develops one quantitative model, and uses a worked investigation to connect program behavior with a profiler report, benchmark table, memory record, or service log. The knowledge check tests whether you can reconstruct the causal chain without copying the paragraph above it.
Each exercise creates evidence that another engineer can inspect. A calculation exposes units and assumptions. A benchmark program produces raw timing samples. A profiler records timestamped GPU events or hardware-counter estimates. A correctness test compares the optimized result with a declared reference. Preserving all four artifacts turns a tutorial run into technical knowledge that can survive a new input shape or GPU generation.
Use prerequisites as diagnostic information
A prerequisite names knowledge that the subsection will use without deriving from first principles. Test it actively. If a section assumes matrix multiplication, you should be able to derive the shape of C from A[M,K] and B[K,N] and explain why the arithmetic count is approximately 2MKN. If you cannot, review that topic before continuing.
Do not interpret a missing prerequisite as a judgment about ability. The missing prerequisite identifies the shortest repair path. Systems topics combine architecture, operating systems, numerical analysis, compilers, networking, and machine learning. Every reader will encounter a boundary that requires review.
Alternate explanation and construction
Reading can create recognition without recall. After each subsection, close the page and explain the mechanism in your own words. Name the CPU function, request queue, tensor, CUDA kernel, GPU storage level, or network transfer that performs each step. Then state the direction of data movement, the critical dependency, the limiting resource, and one observation that would reject the explanation. Redraw tensor shapes or timelines from memory when the topic is spatial or sequential.
Then construct something. Reproduce a calculation with different numbers. Modify a benchmark shape. Predict the result of changing a batch size before moving the control in a widget. Construction reveals gaps that smooth prose can conceal.
Combine analytical and empirical work
Analytical exercises force you to state assumptions before hardware produces a complicated profiler timeline. The exercises are available even without an accelerator: count operations, derive lower bounds, draw dependency graphs, and calculate memory lifetimes. Quantitative models teach scale and eliminate impossible explanations.
Empirical exercises test the assumptions. When a model predicts 20 microseconds and the measurement is 70 microseconds, do not report only that the model was inaccurate. Account for the missing 50 microseconds. The difference can contain launch overhead, extra traffic, a different instruction path, synchronization, or queue delay. The unexplained interval becomes the next question.
Maintain an experiment record
Store each substantial experiment under version control. Record the question, workload, environment, prediction, method, raw observations, correctness result, interpretation, and uncertainty. Preserve scripts and machine-readable output beside the prose report. A future hardware or software change can then rerun the experiment instead of reconstructing it from memory.
Include configurations that failed and hypotheses that measurements rejected. A report containing only the fastest result cannot show whether the author understood the search space or selected one favorable observation. Negative evidence establishes which explanations were tested and where the conclusion remains uncertain.
Scale experiments to available hardware
Many core questions can be reduced to an experiment on a CPU or one GPU. A simulated queue can demonstrate saturation. A small CUDA kernel can expose coalescing, reuse, and synchronization. Collective cost can be modeled on paper before it is measured on several devices. Public traces can teach critical-path analysis.
Large clusters become necessary when the phenomenon itself depends on scale, topology, correlated failure, or production traffic. Until then, use the smallest experiment that preserves the causal mechanism. Smaller experiments run more often, cost less, and make instrumentation easier to understand.
Worked example: Turning a tutorial into an experiment
A CUDA tutorial provides a tiled matrix multiplication that is faster than a naive implementation for one square matrix. You want to turn the tutorial into evidence that you understand reuse, boundary handling, and resource tradeoffs.
- Derive the logical arithmetic count. Then estimate global-memory traffic for the naive and tiled access patterns. State the assumed cache behavior and calculate arithmetic intensity under those assumptions.
- Predict where tiling should help. Large matrices with reusable tiles should reduce repeated global loads. Very small matrices may be dominated by launch and setup cost. Awkward dimensions can spend threads on bounds handling and incomplete tiles.
- Build correctness tests for rectangular matrices, dimensions smaller than a tile, dimensions not divisible by a tile, non-contiguous inputs if supported, and extreme values. Compare with a trusted library or wider reference and justify the tolerance.
- Sweep matrix shapes and several tile sizes. Record elapsed distributions, achieved bandwidth or compute rate, occupancy, and resource use. Compare with a mature library so the naive kernel is not the only baseline.
- Write a report that connects the source transformation to the changed traffic and timing. Include one prediction that failed, identify the assumption that caused the failure, and state the range of shapes for which the conclusion applies.
Conclusion: The resulting artifact demonstrates transferable reasoning: it explains why tiling changes physical data movement, when the benefit appears, what it costs in resources and boundary work, and how the evidence supports the conclusion.
Common errors
- Treating a completed reading as mastery. Recognition during reading must be followed by recall, calculation, implementation, or explanation.
- Copying a tutorial benchmark without deriving what it should show. A reproduced speedup demonstrates execution, but it does not establish understanding of the cause.
- Waiting for a large cluster when the relevant mechanism can be isolated on a smaller system. Scale the experiment only when scale is part of the question.
- Publishing only the fastest configuration. A shape sweep, raw observations, and rejected hypotheses show the conditions under which the conclusion holds.
Reference summary
Each subsection identifies the knowledge that it assumes. Review a prerequisite when you cannot explain it without referring to the page. Read the main explanation, reproduce the worked calculation, answer the knowledge check from memory, and complete an exercise that produces inspectable evidence. This sequence tests understanding instead of recognition.
- Chapters 0–3 define objectives, quantitative models, benchmark methods, numerical formats, and transformer workloads.
- Chapters 4–7 explain accelerator architecture, execution, memory behavior, profiling, and generated instructions.
- Chapters 8–12 develop parallel kernel algorithms, GEMM, attention, library integration, and compiler transformations.
- Chapters 13–16 explain networks, collectives, model parallelism, distributed analysis, and failure recovery.
- Chapters 17–20 explain inference execution, scheduling, KV-cache management, quantization, and production operation. Chapter 21 integrates the complete investigation method.
Knowledge check
What must an experiment record preserve so that another engineer can repeat the request, reproduce the measurement, inspect correctness, and evaluate the causal explanation?
Show the answer guide
- Preserve the exact input or request generator, workload parameters, random seeds, and command so another engineer can repeat the same work.
- Preserve the source revision, build configuration, dependency versions, hardware identity, driver and runtime versions, clock or power policy, and relevant environment settings.
- Preserve the warm-up policy, timer endpoints, synchronization method, raw timing samples, sample count, and summary calculation so the measurement can be reproduced rather than copied from one console line.
- Preserve the reference implementation, adversarial inputs, numerical tolerances, test output, and model- or task-quality result used to accept the contender.
- Preserve the prewritten hypothesis, quantitative prediction, controlled intervention, profiler command and report, and observations that would reject the explanation.
- Preserve the conclusion with its supported scope, failed hypotheses, alternative explanations, and unresolved limitations.
Build a portable hash-throughput record
Generate one fixed 16 MiB file, record its checksum, and measure 40 executions of a chosen local hashing program after a declared warm-up. Package the command, source or tool version, environment manifest, raw durations, and an independent verification of the resulting digest.
Evidence to keep: Keep a directory listing containing the manifest, workload checksum, raw CSV, digest-verification output, timing-summary calculation, and a README conclusion that another engineer can follow without access to the original machine.
The remaining chapters use this study method. The next chapter begins with dimensional analysis, which supplies the units and lower bounds for later experiments.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- The Rust Programming Language↗
Structural inspiration: progressive chapters, concrete examples, and explicit learning flow.
- NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.
- PyTorch Profiler documentation↗
Official reference for recording operator activity, CPU and accelerator events, shapes, stacks, and exported traces.
- MLPerf Inference Rules↗
- MLPerf Training Benchmark↗