Coalescing and useful bytes
List every active lane address for one memory instruction and estimate requested bytes, transferred bytes, and useful-byte fraction
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Chapter 4 definition of a 32-lane warp
- Ability to calculate aligned byte ranges for contiguous and strided arrays
Consider one warp in which every lane loads one FP32 value. The program requests 32 × 4 = 128 useful bytes. Device memory does not service 32 abstract scalar requests independently. The hardware combines lane addresses into aligned sectors and transactions. The number of transferred bytes depends on where those 32 addresses fall.
If lane 0 reads x[0], lane 1 reads x[1], and so on, the addresses occupy a short contiguous range. If lane 0 reads x[0], lane 1 reads x[32], and each later lane advances by another 128 bytes, the warp still requests 128 useful bytes but touches many separated regions. The memory system must transfer aligned units that contain mostly unused data.
Coalescing describes the addresses generated by one warp instruction. Coalescing is not a permanent label attached to an allocation or tensor. A row-major matrix supports efficient row access when lanes move across columns, but the same matrix can produce scattered access when lanes move down rows. Alignment, predicates, and element width can change the transaction set again.
Calculate requested and transferred bytes
First list the active lanes and the bytes each lane requests. Their sum is requested data. Next map every address to the aligned sectors or segments used by the target memory level. Count each serviced unit once. Their total is transferred data. Requested bytes divided by transferred bytes provides an intuitive efficiency, although profiler metrics may define the level and transaction units more precisely.
For a contiguous aligned FP32 load by 32 lanes, the useful range is 128 bytes. On current NVIDIA architectures, global transactions are discussed in terms of smaller sectors within cache lines, and exact behavior depends on architecture and cache path. Do not memorize one transaction size as a universal law. The stable method is to map addresses to the documented units for the target and confirm sector traffic in the profiler.
A cache can make overfetch useful later, but later accesses must consume the retained data before eviction. Later cache reuse does not erase the first transfer. When analyzing an isolated instruction, calculate the immediate waste. When analyzing a loop, also model whether adjacent iterations reuse the same sectors.
Alignment and vector width
A contiguous access can still cross an additional aligned boundary. Device allocations provide strong base alignment, but an offset into the allocation can change the first address of each warp. Write the address of lane 0 for every important warp, especially when rows have padded or unusual strides.
Vector loads such as a four-element load can reduce instruction count and express contiguous data movement. Vector loads require suitable type and address alignment, and the compiler must emit the intended operation. A cast does not repair misaligned storage. A misaligned or out-of-range vector access can create undefined behavior or force a slower instruction sequence. Handle tails separately when a vector width does not divide the logical size.
Vectorization does not reduce the number of bytes required by a streaming algorithm. If device-memory bandwidth is already saturated with coalesced scalar operations, fewer load instructions may not improve elapsed time. Verify both generated instructions and the active resource.
Choose a layout for the access pattern
An array of structures stores all fields of one record together. Record-oriented storage can be appropriate when each thread consumes a complete record. A structure of arrays stores one field for many records contiguously. Field-oriented storage can be appropriate when each warp loads the same field from many records. Neither organization is universally better.
A transpose or shared-memory tile can convert one access direction. Threads first perform coalesced global loads along the contiguous dimension, rearrange values on chip, and then consume or write them in another logical order. The transformation adds shared-memory traffic and synchronization. Tiling is beneficial when the removed global transactions cost more than the added on-chip work.
Worked example: A warp reading a strided column
Thirty-two lanes each read one FP32 value from consecutive rows of a matrix whose row stride is 128 bytes.
- Requested data is 32 lanes × 4 bytes = 128 bytes.
- Addresses are separated by 128 bytes, so they fall in 32 distinct serviced regions rather than one short contiguous range. On a target where the documented and measured service unit is a 32-byte sector, the instruction transfers 32 × 32 = 1,024 bytes. Useful-byte efficiency is then 128 / 1,024 = 12.5 percent. Recalculate this number for the service unit and cache path of the actual target.
- Remap a block to load a two-dimensional tile with lanes moving across contiguous columns, then transpose or reinterpret the tile in shared memory for column-oriented computation.
- Measure global requested-versus-transferred bytes and total time. Include shared-memory traffic and synchronization introduced by the tiled path.
Conclusion: Under the stated 32-byte-sector model, the original warp uses only 12.5 percent of the bytes transferred for that instruction. The tiled version adds on-chip operations and converts scattered global accesses into dense transactions. The tiled version is faster when the removed device-memory traffic costs more than the staging operations.
Common errors
- Calling a pointer aligned without considering each warp's offset. A well-aligned allocation can still produce misaligned instruction addresses.
- Vectorizing blindly. Wider types help only when alignment, layout, tails, and generated instructions support them.
- Stopping after coalescing is perfect. A streaming kernel can remain at the device-memory bandwidth roof; further progress requires less traffic or more reuse.
Reference summary
For one warp instruction, list the bytes requested by every active lane. Then map those addresses to the aligned sectors or transactions used by the target architecture. Requested bytes measure logical demand. Transferred bytes measure service by the memory system.
Adjacent, aligned addresses usually occupy fewer serviced units. Strided or scattered addresses can require many units that contain data no lane requested. Cache reuse can make some overfetched data useful later, so analyze both the instruction and the enclosing loop.
global load efficiency = requested bytes / transferred bytes- Align base pointers and map lane id across the contiguous dimension.
- Vectorized loads help only when alignment, layout, and compiler code generation support them.
- Array-of-structures can be convenient while structure-of-arrays often improves field-wise coalescing.
- A transpose can turn one phase’s coalesced access into another phase’s strided access; tile the transformation.
Knowledge check
A warp’s lanes load 32 floats separated by 128 bytes. Why is requested bandwidth small relative to transferred bandwidth?
Show the answer guide
- A 128-byte stride between adjacent lanes places the 32 requested floats in separate widely spaced sectors instead of a compact contiguous region.
- Each lane requests only four useful bytes, but the memory system must fetch the containing sector or transaction unit, so transferred bytes greatly exceed the warp's 128 useful bytes.
- Requested bandwidth can therefore remain small while transferred bandwidth approaches the device limit; sector counts and useful-to-transferred-byte efficiency reveal the waste.
Measure coalescing efficiency
Benchmark a warp-oriented load kernel with contiguous addresses and with 32-, 64-, and 128-byte lane strides. Use the same number of useful float results and prevent the compiler from removing the loads.
Evidence to keep: Address table for one warp, raw timings, requested bytes, measured sectors or transferred bytes, and useful payload divided by physical traffic for every stride.
When global access is efficient but data must be reused or rearranged, shared memory provides an explicitly managed on-chip exchange.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.