Latency processors and throughput processors
Given one memory-dependent workload, predict whether CPU latency reduction or GPU latency hiding is likely to matter and name the measurement that tests the prediction
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Ability to distinguish an instruction from the data that the instruction reads
- Introductory understanding that caches store recently used data closer to a processor than off-chip DRAM
Consider a loop that reads one value from memory, multiplies it, and writes a result. When a load misses the nearby caches, the requesting instruction cannot use the value until memory returns it. A processor must either reduce that delay, find independent instructions from the same thread, or run work from another thread. CPU and GPU designs use all three methods, but they allocate hardware to them in different proportions.
A high-performance central processing unit (CPU) core tries to make one or a few instruction streams finish quickly. The CPU predicts branches so that instruction fetching can continue. The CPU examines a window of instructions and issues independent operations out of program order. Large CPU caches reduce the number of requests that reach dynamic random-access memory (DRAM). Branch prediction, out-of-order issue, and large caches consume considerable chip area and power, but the mechanisms help programs with irregular control flow and limited parallelism.
A graphics processing unit (GPU) assumes that the program can provide many independent threads. The GPU keeps the state of many warps resident on each streaming multiprocessor. If one warp issues a load and must wait for its operands, the warp scheduler can select another eligible warp. The original load still has the same latency. The GPU maintains throughput by performing other useful work during that latency.
The many-warp approach works well only when the workload supplies enough ready work. Large matrix operations, convolutions, and elementwise tensor operations often contain thousands of similar output elements. A small graph traversal may contain too few ready nodes. A parser may send neighboring lanes through different branches. A sequence of tiny dependent kernels may leave the GPU waiting for the host. A floating-point operation (FLOP) rate in operations per second describes one hardware limit, not the workload's ability to approach it.
Latency, concurrency, and throughput
Latency is the time between starting an operation and receiving its result. Concurrency is the number of independent operations that can be in progress. Throughput is the number of operations completed per unit time. The three quantities describe different properties. A GPU can tolerate a long-latency load when enough other warps can issue independent instructions. The GPU cannot tolerate the same load when every resident warp depends on the missing value.
Suppose, for illustration, one warp scheduler can select an instruction from a ready warp each device cycle, and a memory dependency takes roughly 400 cycles. One cycle is one tick of the relevant device clock; it is not a fixed amount of wall time across clock states. A warp with a strictly dependent load-use sequence can issue its load and then remain unable to proceed for most of that interval. Other warps or independent instructions from the same warp can occupy some of those issue opportunities. Do not infer that exactly 400 warps are required: real SMs contain several scheduler partitions, and each warp can have more than one independent operation in flight.
More concurrency does not increase the capacity of a saturated resource. If off-chip device memory can sustain a fixed number of bytes per second, additional warps can create enough outstanding requests to approach that rate. After the rate is reached, still more requests wait in queues. Parallel work is needed to reach a throughput limit, but it cannot raise the limit.
The host and device execute one combined schedule
A typical framework operation may require CPU shape calculations, runtime dispatch, a kernel launch, device execution, and a later synchronization. For a large kernel, launch time may be negligible. For a sequence of tiny kernels, repeated host work and idle gaps can dominate. A GPU can therefore be underused even when each individual kernel uses its assigned hardware efficiently.
Data placement is part of the same schedule. A tensor already in device memory can feed several kernels without crossing PCIe or another host link. An input produced on the CPU may require allocation, transfer, and synchronization before the first device instruction executes. The useful comparison between CPU and GPU implementations includes this complete path, not only the body of the GPU kernel.
Interpret utilization as a resource measurement
The word utilization is incomplete without a resource and a denominator. GPU-active time asks whether at least one kernel was present. Compute utilization compares achieved arithmetic rate with an applicable peak. Memory utilization compares traffic with an attainable bandwidth. Scheduler metrics ask whether warps were eligible and whether instructions issued. The measurements can disagree without contradiction because their numerators and denominators differ. A device can be active for the entire interval while performing little useful work.
Use the architectural model to form a prediction before profiling. For a small branch-heavy operation, predict significant fixed overhead, limited block count, and inactive lanes. For a large streaming operation, predict high memory traffic and enough parallelism to approach the memory-bandwidth limit. For a large matrix multiply, predict pressure on matrix pipelines and on the memory paths that supply operands. Select a measurement that can reject or support the prediction.
Worked example: CPU performance for a small irregular workload
A branch-heavy operator processes 2,000 variable-length records. Each record requires about 40 arithmetic operations and follows one of several short control paths. The input already resides in CPU memory. A GPU offers much higher peak arithmetic throughput, yet the tuned CPU implementation finishes sooner.
- Count the useful arithmetic: about 80,000 operations. Even a modest processor can execute this amount quickly, so dividing by the GPU's peak FLOP/s predicts a time that is smaller than launch and scheduling overhead. The peak number is not informative at this scale.
- Examine available parallelism. Two thousand records create 62 full warps and one partial warp if one thread handles one record: ceil(2,000 / 32) = 63 scheduled warps. With 256-thread blocks, the grid contains only ceil(2,000 / 256) = 8 blocks, so no more than eight SMs can receive a block from this launch. Variable record lengths also cause some lanes to finish while others continue.
- Account for data placement and fixed work. Moving inputs to the device, launching a kernel, and waiting for a result occur once per operation. Transfer, launch, and synchronization costs do not become smaller because the kernel contains little work.
- Explain the CPU result. The records already reside in the CPU memory hierarchy, the CPU's branch machinery handles irregular paths, and one function call begins useful work without a device transfer. The CPU's lower peak arithmetic rate is not the limiting property.
- Change the scenario. If one million records arrive in a batch and remain on the device for several later operators, transfer and launch costs are amortized and many more warps become available. Under those conditions, the GPU can become faster without any change to its peak specification.
Conclusion: The CPU wins because this instance provides little regular work and pays a large fixed cost to cross the device boundary. The conclusion is conditional on batch size, data location, and control-flow distribution; it is not a general statement that the operator belongs on the CPU.
Common errors
- Explaining every result with core count. Execution width, instruction mix, memory behavior, and schedulable parallelism determine whether those cores can contribute.
- Treating latency hiding as latency reduction. The individual dependency still takes time and appears when independent work runs out.
- Ignoring orchestration. At small granularity, the path into the accelerator can dominate the work performed there.
Reference summary
A CPU and a GPU both need ready instructions while memory operations are pending. A high-performance CPU searches for independent instructions within a small number of instruction streams and uses large caches and prediction hardware to reduce delay. A GPU keeps state for many warps and selects another eligible warp when one warp must wait.
The waiting operation does not finish sooner on the GPU. Other instructions execute during the wait. This method requires enough independent warps, and it stops helping after the limiting memory or execution pipeline reaches its throughput capacity.
- CPUs are often effective for serial sections, irregular branches, operating-system work, and workloads with modest parallelism.
- GPUs are often effective when a workload exposes many similar operations, regular memory access, and enough work to occupy the device.
- End-to-end time includes host preparation, allocation, data transfer, launch, device execution, and synchronization.
Knowledge check
Why can a CPU be faster than a GPU with higher peak FLOP/s for a small, branch-heavy workload?
Show the answer guide
- A small workload may not expose enough independent operations to fill a GPU, so launch, transfer, and scheduling costs can dominate its useful arithmetic.
- Branch-heavy control flow can make lanes in a GPU warp execute different paths, while a CPU core can use prediction and out-of-order execution to reduce the latency of one instruction stream.
- Peak FLOP/s describes a ceiling for a compatible arithmetic path; the comparison must also measure end-to-end latency, active parallel work, data movement, and the executed instruction mix.
Compare latency and throughput processors
Run one small branch-heavy array operation and one large regular array operation on a CPU and an available GPU or GPU simulator. Predict the crossover before measurement and include transfer and launch time in one GPU result.
Evidence to keep: Source, input sizes, raw latency samples, declared timing boundaries, and a short explanation of why the faster processor changes or does not change with workload size.
The throughput model depends on groups of ready work. The next section explains exactly how CUDA threads are grouped into warps, how blocks occupy streaming multiprocessors, and what happens when lanes in one warp follow different paths.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.