Matrix instructions, fragments, and layouts

Trace operands and accumulators through a cooperative matrix-instruction contract

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

Tensor-core concept
Multi-level tiling
Floating-point accumulation

Compile an FP16 matrix-multiplication kernel for a recent NVIDIA GPU and open the Source page in Nsight Compute. The page can display SASS, the native machine-instruction listing generated for that NVIDIA GPU architecture. Each SASS line is an instruction executed by the device rather than a CUDA source statement. A successful tensor-core path contains cooperative matrix multiply-accumulate instructions rather than a long sequence of independent scalar multiply-add instructions. Changing K from an aligned value such as 4096 to an unsupported tail shape, changing pointer alignment, or compiling for another architecture can select a different instruction sequence and produce a visible timing change.

One cooperative instruction consumes small pieces of the A and B matrices and updates pieces of the output accumulator held across several threads. CUDA documentation and CUTLASS describe each piece as a fragment. The fragment is distributed: no single thread owns the complete little matrix shown in a diagram. The lane mapping matters when data moves between shared memory, registers, and the instruction operands.

Matrix instructions are cooperative operations. A warp or warp group supplies fragments of A and B and collectively owns an accumulator fragment. No single lane possesses the whole miniature matrix. The API may abstract that mapping, but layout, alignment, instruction shape, and accumulator type still govern whether the fast path is legal and efficient.

Matrix instructions impose hardware requirements on the tile hierarchy. A block tile must decompose into warp tiles, and warp tiles must decompose into supported instruction tiles. Shared-memory and register layouts must deliver each fragment without conflicts or excessive rearrangement. Tail dimensions need padding, predicates, or alternate instructions.

Do not begin by memorizing lane-to-element tables for every architecture. Begin with the contract: a group of threads supplies operand fragments of a supported shape and datatype, and the instruction updates a distributed accumulator fragment. Libraries encode the current mapping. You need to understand the mapping well enough to satisfy alignment, layout, and lifetime requirements and to inspect generated instructions.

The matrix-instruction contract

A matrix multiply-accumulate instruction computes a small operation of the form D = A×B + C. Its instruction shape defines how many rows, columns, and K elements participate. Its datatype contract defines operand formats and accumulator format. Its thread-group contract defines whether a warp or a larger group must execute the instruction cooperatively.

A supported input datatype is necessary but not sufficient. Pointer alignment, leading dimensions, target architecture, fragment layout, and K divisibility can determine whether the compiler or library selects the matrix path. Confirm the native instruction in generated code. If the fast instruction is absent, inspect eligibility before tuning occupancy or pipeline depth.

Decomposing a warp tile into instruction tiles

Suppose a warp owns a 64×64 output tile and uses an illustrative m16n8k16 instruction. The names m, n, and k give the instruction's row, column, and reduction dimensions: one instruction contributes to a 16-row by 8-column region while consuming 16 elements of the K dimension. The warp therefore issues a grid of instruction operations across its 64×64 output region for every 16-element K fragment. The distributed accumulator fragments remain live across all K iterations. This concrete instruction grid determines how many accumulator groups are live and how many registers each lane requires. Other GPU architectures and datatypes can provide different instruction shapes, so inspect the instruction selected for the declared target.

The fragment is distributed. Lane 0 does not own the complete top-left submatrix, and lane order does not imply row-major element order. Use documented fragment access or a library layout abstraction for transformations. Direct lane mapping is architecture-specific and should be introduced only when the required operation cannot be expressed through the supported interfaces.

Moving operands into fragments

Global-memory layout describes where logical elements reside in HBM. Shared-memory layout can be different because lanes must load fragments concurrently without severe bank conflicts. Swizzles alter the physical shared-memory address while preserving a known logical coordinate mapping. Matrix-load instructions may also impose transfer shapes and alignment rules.

Follow one coordinate through the complete path: logical A[row,k], global byte address, shared-memory position, lane and register fragment, then matrix instruction. Repeat for B. The coordinate path reveals whether an apparent transpose is only a layout interpretation or an actual data movement. The path also explains why copying a conventional row-major shared tile can be correct yet slow.

Accumulator output and the epilogue

The accumulator layout is selected for matrix computation, not for coalesced global stores. The epilogue reorganizes distributed accumulator elements into an output access pattern. The epilogue may use shared memory as an exchange. The same phase can apply scaling, add a bias or residual, execute an activation, convert datatype, and clamp before the final store.

For large K, the main loop performs enough matrix work to amortize this output phase. For small K, epilogue instructions, conversions, and shared-memory exchange can occupy a substantial fraction of total time. A fused epilogue still saves an extra tensor round trip, but it can increase register lifetime. Profile the main loop and epilogue separately when the nominal matrix throughput does not explain total kernel time.

Worked example: Explaining poor tensor-core utilization

A GEMM uses a supported FP16 datatype but achieves only a small fraction of expected tensor throughput.

Verify native instructions. The compiler or library may have selected scalar or SIMT arithmetic because dimensions, alignment, or target architecture failed eligibility.
Write down the expected matrix instruction shape and accumulation type. Compare these with the generated instruction rather than checking only for any tensor operation.
Inspect shape fit. Narrow M or N can provide too few warp tiles, while ragged K can require cleanup and padding that lower useful-lane efficiency.
Examine supply: shared-memory bank conflicts, fragment-load stalls, or shallow pipelining can starve matrix instructions even when they are present.
Measure epilogue and conversion share. For small K, output processing and launch overhead may dominate the nominally fast multiply.
Change one cause at a time: align the operands, pad K, select a compatible layout, or isolate the epilogue. Reprofile after each change so the explanation remains causal.

Conclusion: Datatype enables a matrix path; it does not guarantee that shape, layout, pipeline, and surrounding work can sustain it.

Common errors

Searching only for a matrix opcode and declaring success. Frequency, eligibility, operand supply, and useful work determine realized throughput.
Assuming logical row-major or column-major fully describes fragments. Cooperative lane mapping and swizzles are part of the layout.
Ignoring accumulators and epilogue. Register pressure and output transformation can be the limiting costs.
Copying an architecture-specific fragment map into general code without an explicit target guard. A later architecture can change the legal mapping or instruction family.

Reference summary

Matrix instructions distribute operand fragments and accumulators across a cooperating warp or warpgroup. The mapping is architecture- and instruction-specific. Libraries such as CUTLASS and its CuTe abstractions encode layouts, copies, MMA operations, and pipelines compositionally. The learning goal is not to memorize every fragment map; it is to understand the contracts well enough to inspect, choose, and adapt them.

Choose an instruction supported by the target architecture and datatype.
Map global and shared layouts so fragment loads are aligned and conflict-aware.
Keep enough independent accumulator fragments to feed the matrix pipeline.
Budget accumulator registers; output tiles can dominate register pressure.
Handle edge tiles through predicates, padding, or specialized kernels.

Knowledge check

Why can an epilogue that looks cheap in FLOPs materially limit a tensor-core GEMM?

Show the answer guide

Tensor-core matrix instructions can make the matrix multiply body extremely fast, so the epilogue becomes a larger fraction of total kernel time even when its FLOP count is small.
The epilogue must read accumulator fragments, apply scaling, bias, activation, quantization, or reduction logic, and write the final tensor. Its cost is often data movement, conversion throughput, dependency latency, or synchronization rather than arithmetic count.
Extra epilogue values can increase register lifetime and register count. Lower occupancy or spills can slow the matrix body as well as the epilogue.
An output layout that does not match accumulator lane layout may require shuffles or shared-memory staging. The epilogue can therefore limit total GEMM throughput through bandwidth, resource pressure, and layout conversion despite few nominal FLOPs.

Practice

Measure an epilogue that limits tensor-core GEMM

Benchmark a tensor-core GEMM with M=N=K=4096 in three forms: plain output conversion, fused bias plus ReLU, and fused bias plus GELU plus per-row maximum. Use the same matrix schedule when possible and validate every result against a high-precision framework reference.

Evidence to keep: Submit the kernel or library configuration, numerical tests with tolerances, a latency table, and profiler reports with register count, spills, occupancy, memory traffic, and instruction categories. Quantify the epilogue's share of the added time and identify whether bandwidth, conversion, layout movement, or resource pressure explains it.

A custom schedule demonstrates the implementation mechanism. Production systems also require libraries and dispatch policies for many tensor-shape families.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.