Tensor cores and matrix engines
Explain why instruction-tile shape, operand layout, input precision, accumulator precision, and surrounding data movement limit measured matrix-kernel throughput
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Ability to multiply an M×K matrix by a K×N matrix
- Chapter 3 introduction to floating-point formats
- Previous section's thread, warp, and block model
General matrix multiplication (GEMM) computes many dot products. In C = A × B, each element of C reuses a row of A and a column of B, and nearby outputs reuse much of the same input data. The regular reuse makes matrix multiplication suitable for specialized hardware that performs a fixed tile of multiply-accumulate operations per instruction.
On NVIDIA GPUs, tensor-core instructions are issued cooperatively by groups of threads. Other accelerators use different names and dataflows, including matrix-core instructions and systolic arrays. Tensor cores, matrix cores, and systolic arrays do not share one programming or scheduling interface. The limited common model is structured matrix arithmetic supplied by tiled operands. Each target reaches high throughput only when software satisfies its own datatype, shape, layout, and parallelism requirements.
A peak tensor-throughput number assumes that the matrix array remains supplied and that every operation in its fixed instruction shapes is useful. Real kernels must load operands, arrange them, keep accumulators, handle boundary tiles, and store results. Operand loading, fragment movement, boundary handling, and output storage often explain the difference between peak throughput and measured application speed.
Three levels of matrix tiling
The first level is the instruction tile. A matrix instruction accepts fixed logical dimensions and datatypes defined by the architecture. Threads contribute operand fragments and receive accumulator fragments. The programmer or library cannot ask that instruction to compute an arbitrary ragged rectangle; unsupported edges need padding, predicates, or another instruction path.
The second level is the warp or warp-group tile. Several matrix instructions cover a larger output region and advance through the K dimension. The third level is the thread-block tile. Threads in a block cooperatively load a larger region of A and B, reuse the inputs across instructions, and retain output accumulators in registers. Instruction, warp, and block tiles should not be confused because each level has different ownership, synchronization, and capacity limits.
Layout is part of the instruction contract. Row-major versus column-major storage is only the first choice. Operand fragments must also map to lane registers and shared-memory addresses in the pattern expected by the instruction. A logically correct matrix can require a transpose, swizzle, or packing step before the specialized instruction can consume it efficiently.
Supplying the matrix pipeline
A matrix instruction can finish quickly only if its operands are ready. A typical loop loads a future global-memory tile, stages or rearranges it in shared memory, moves fragments into registers, issues matrix operations on the current tile, and advances accumulators. If these stages execute strictly one after another, the matrix pipeline waits during every load.
Software pipelining overlaps stages from different K iterations. While matrix instructions consume tile k, asynchronous or ordinary load operations prepare tile k+1. Two stages need storage for two tile states; more stages need still more shared memory and registers. Additional stages can hide memory latency, but they can also reduce the number of resident blocks. The correct stage count depends on load latency, tile size, and resource limits.
Small or narrow matrices create a second supply problem. Even a perfectly tiled kernel cannot occupy a large device if the matrix contains only a few output tiles. Batched operations can provide parallelism across matrices. Shape specialization can also select a tile that produces more independent blocks, even if its per-block reuse is lower.
Accumulation and the epilogue
Matrix inputs may use a narrow format while accumulators use a wider format. This choice affects throughput, register use, overflow, and numerical error. Low-precision inputs also require a scaling contract. A hardware path is useful only if the resulting model quality remains within the required tolerance.
After the K loop, the accumulator tile must become output. Bias addition, activation, scaling, clipping, and quantization can often run while values remain in registers. A fused epilogue avoids writing an intermediate tensor to device memory and reading it again. However, the added operations can lengthen live ranges, increase register pressure, complicate tail handling, or reduce store efficiency. Measure the complete GEMM operation, including preparation and epilogue.
Worked example: Effect of padding on projection performance
A projection has K = 4090, while the preferred matrix path consumes K in aligned chunks. A scalar cleanup handles the final elements.
- Estimate useful work for K = 4090 and padded work for K = 4096. Padding adds only about 0.15% arithmetic.
- Compare execution paths. The padded shape may use uniform vector loads and matrix instructions throughout, while the ragged shape introduces predicates, a cleanup loop, or a slower library kernel.
- Account for memory. If weights can be stored padded once and reused many times, the small storage cost is amortized. Padding activations on every request may add a conversion whose cost must be measured.
- Verify numerics by filling padded values with zeros and ensuring the output contract ignores padded channels. Benchmark the full producer-consumer chain, not only the matrix call.
Conclusion: Padding adds a small amount of arithmetic but can remove a cleanup path and permit uniform matrix instructions. Padding is beneficial only when the saved execution and control cost exceeds the padding and conversion cost for the complete producer-consumer path.
Common errors
- Dividing peak tensor FLOP/s by operation count and calling the result predicted latency. The feed path, shape fit, and epilogue may set lower ceilings.
- Assuming smaller datatypes are automatically faster. Conversions, scaling, unsupported shapes, and accumulator requirements can dominate.
- Optimizing only the instruction tile. Block-level reuse and end-to-end layout determine whether the instruction receives data efficiently.
Reference summary
Matrix engines implement fixed, structured multiply-accumulate operations. On NVIDIA GPUs, a warp or warp group supplies operand fragments and retains accumulator fragments, depending on the instruction family and GPU generation. Other accelerators can expose a different execution interface. Libraries combine the supported operations into larger tiles.
The matrix instruction is only one stage. Global loads, shared-memory staging, fragment movement, synchronization, and the output epilogue must supply and drain it. A slow surrounding stage lowers complete-kernel throughput even when the matrix instructions themselves execute at high rate.
- The datatype, instruction shape, input layout, and alignment must be supported by the target path.
- Narrow or ragged dimensions can leave tile positions unused or require a cleanup path.
- Accumulator width, scaling, and output conversion must satisfy the numerical contract.
- The grid must contain enough independent tiles to occupy the device.
Knowledge check
Why is advertising peak tensor TFLOP/s insufficient to predict end-to-end transformer speed?
Show the answer guide
- Peak tensor TFLOP/s applies only to supported datatypes, instruction shapes, accumulator rules, and hardware paths; unsupported or awkward shapes can use another instruction path.
- A complete kernel must load and stage operands, issue matrix instructions, manage accumulator fragments, synchronize, and store or transform the output.
- End-to-end transformer time also contains non-matrix operations, communication, launch and framework work, so a matrix-engine ceiling cannot predict the complete workload by itself.
Test matrix-engine eligibility
Benchmark one matrix multiplication while varying one dimension through aligned and ragged values at a fixed datatype. Record which instruction path executes and compare achieved rate with the applicable documented ceiling.
Evidence to keep: Shape table, datatype and accumulator format, raw timings, profiler or disassembly evidence for the instruction path, and achieved FLOP/s for every shape.
Matrix engines require operands at a sufficient rate. The memory hierarchy determines how often software can reuse each value before it must access a slower memory level.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.