Accelerator memory hierarchy
Assign one value to registers, block-shared memory, cache, or device memory from its owner, lifetime, reuse, and synchronization requirements
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Chapter 1 definitions of bytes, latency, and completed bytes per second
- Previous section's thread, warp, block, and streaming-multiprocessor model
Return to one multiply-accumulate operation. The arithmetic unit needs two inputs and a current accumulator. If all three values came from off-chip device DRAM for every operation, a matrix engine would spend most of its time waiting for data. The DRAM can be high-bandwidth memory (HBM), graphics double data rate memory, or another device-memory technology. High arithmetic throughput is possible because the same values are retained and reused in smaller storage structures near the execution pipelines.
The CUDA memory hierarchy contains distinct storage spaces with different ownership. Registers normally hold thread-local values. Shared memory holds block-local data that threads manage explicitly. Hardware caches retain global-memory data automatically. Device memory holds the large working set. Host memory is accessible through a separate path and often requires explicit or managed transfers.
The useful mental model is not a simple ranking from fast to slow. Each level has a capacity, scope, access pattern, and allocation cost. Keeping more values in registers may reduce resident warps. Allocating a larger shared tile may reduce resident blocks. Depending on cache retention may produce unstable performance when other blocks or workloads compete for the same cache.
Assign values according to ownership and lifetime
An accumulator used repeatedly by one thread should normally remain in registers until its output is complete. An input tile needed by many threads in one block can be loaded once into shared memory. Read-only data used by many blocks may benefit from L2 or another cache, but blocks cannot assume that an ordinary cache line remains present. A large model state that exceeds on-chip capacity must reside in device memory and be transferred as needed.
For each value, ask four questions. Which threads use the value? How many times do the threads use the value? How long must the value remain live? Which synchronization is required before another thread can read the value? Ownership, lifetime, reuse, and synchronization determine whether private registers, cooperative shared memory, automatic cache behavior, or global storage is appropriate.
Calculate traffic at each boundary
Count transfers separately at the device-DRAM-to-cache, cache-to-SM, shared-memory, and register boundaries when those levels matter. A high L2 hit rate can reduce off-chip traffic while leaving L2 bandwidth as the limiting resource. A tiled kernel can minimize device-memory reads but issue excessive shared-memory loads. One traffic number cannot represent the complete hierarchy.
Reuse is the number of useful operations or downstream uses obtained from one transfer across a selected boundary. Tiling increases reuse by applying a loaded input tile to several outputs. Fusion keeps an intermediate near its consumer instead of writing it to device memory. Recomputation replaces a stored intermediate with new arithmetic. Each method changes both traffic and another resource, so record bytes, instructions, and storage capacity together.
Cache-hit percentage requires similar care. Ninety percent hits can still mean substantial device-memory traffic when the request count is enormous. A lower hit percentage can be acceptable when total requests are few. Always calculate actual sectors or bytes and compare their service time with the measured kernel duration.
Capacity creates resource tradeoffs
Larger register and shared-memory tiles usually increase reuse, but their storage is reserved for each resident thread or block. At a capacity threshold, a small increase can remove an entire resident block. The resulting loss of ready warps may expose memory or execution latency that the extra reuse was intended to hide.
Register demand has an additional failure mode. When values cannot remain in registers, the compiler can place them in local memory. Local is private in the programming model, but it is backed by the device memory hierarchy. A design intended to keep more data near arithmetic can therefore create new load and store instructions. Compiler reports and profiler traffic must confirm that the planned placement occurred.
The selected placement is valid only for a stated shape and architecture. Cache sizes, shared-memory capacity, register allocation granularity, and instruction support change across devices. Preserve the logical algorithm separately from the tile parameters, and measure the resource balance on every supported target.
Worked example: Placing data for a tiled matrix multiply
A block computes one output tile C. The block needs repeated panels of A and B across the K dimension.
- Assign C accumulators to registers so partial sums never round-trip to device memory during the K loop.
- Load panels of A and B cooperatively from global memory into shared memory. Choose global access patterns so lanes fetch adjacent aligned words.
- Have threads or warps consume shared panels repeatedly in matrix instructions. Count shared-memory reads and inspect bank mapping; device-memory traffic may be ideal while shared service becomes the limiter.
- Increase tile or pipeline stages cautiously. Recalculate shared bytes, registers, and resident blocks, then measure whether greater reuse or overlap compensates for lower occupancy.
Conclusion: The algorithm did not change, but the traffic path did: device memory supplies panels once, shared memory distributes them, registers retain accumulators, and only final output returns to device memory.
Common errors
- Believing the fastest memory level is always best. Its limited capacity can create spills or reduce the parallelism needed to hide latency.
- Reasoning about off-chip memory alone. On-chip bandwidth, bank conflicts, register dependencies, or cache behavior may set the active roof.
- Assuming cache hits are free. They still consume cache bandwidth, queues, and instruction issue, and may interfere with other data.
Reference summary
Place a value according to the threads that use it and the length of its useful lifetime. A thread-local accumulator can remain in registers. A tile reused by many threads in one block can reside in shared memory. Large model state normally remains in device memory and passes through hardware caches when requested.
Every closer storage level has limited capacity. A larger register tile can reduce resident warps. A larger shared-memory tile can reduce resident blocks. Register pressure can also create local-memory spills. The selected placement must reduce traffic without removing too much schedulable work.
reuse = operations performed / values fetched from the next slower level- Registers: private values, accumulators, addresses, and temporary state; excess demand can spill to device memory.
- Shared memory: programmer-controlled tiles and cross-thread exchange within a block; bank conflicts increase the number of serialized bank transactions required for one warp request.
- L2 and other caches: automatically managed reuse with capacity and contention that depend on the active schedule.
- Device DRAM and host links: large-capacity paths whose transferred bytes and attainable bandwidth must be measured. Device DRAM can be HBM, GDDR, or another technology.
Knowledge check
Give one reason that using more registers can speed a kernel and one reason it can slow the same kernel.
Show the answer guide
- More registers can retain reused values and independent partial results close to the execution pipelines, reducing reloads and increasing instruction-level parallelism.
- The same register increase can reduce resident blocks or warps because each streaming multiprocessor has finite register storage.
- If demand exceeds the allocation strategy, spills can add local-memory instructions and device-memory traffic, so timing, residency, and spill evidence must be considered together.
Measure a register tradeoff
Compile one CUDA kernel with at least three tile or unroll settings that change registers per thread. Measure execution time and record registers, spills, and achieved residency for the same inputs.
Evidence to keep: Compiler resource reports, raw timings, profiler residency and local-memory traffic, plus a conclusion that identifies the winning mechanism rather than the largest occupancy value.
Architecture gives the resources and grouping. CUDA execution determines how software maps work onto them and expresses the dependencies the scheduler must honor.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.