Production and upstream integration
Specify an optimized input region and provide reproducible correctness, performance, and maintenance evidence
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Benchmark report
- Custom-op contract
- Version control and code review
A proposed PyTorch or vLLM optimization reduces one 4096 × 4096 benchmark from 120 to 105 microseconds. Reviewers still need to know what happens for M=1 decode, odd dimensions, transposed inputs, BF16 values, unsupported GPUs, graph capture, and backward propagation. A benchmark script, operator tests, and continuous-integration results turn those questions into inspectable evidence instead of assumptions.
The public operation may support more inputs than the new kernel. An eligibility predicate checks architecture, shape, alignment, datatype, layout, and execution mode before selecting the fast path. Boundary tests run inputs immediately inside and outside each condition and compare both routes with a semantic reference. Production and upstream integration depend on this routing contract as much as on the optimized instruction sequence.
A private benchmark proves that code ran quickly once. An upstream-quality change must remain correct across users, shapes, devices, datatypes, and future modifications. The change must make its performance claim reproducible, preserve or improve maintainability, and fit the project's architectural conventions. The standard is demanding because widely shared systems amplify both gains and mistakes.
Define the scope of a specialization before implementation. State exactly where it applies and provide a safe fallback elsewhere. Test the boundaries of that region. Benchmark representative shape families against mature baselines. Explain the mechanism and limitations without requiring reviewers to reconstruct them from the patch.
A useful contribution separates three claims. The semantic claim states that results remain correct. The mechanism claim states why resource use or execution changes. The performance claim states where that change improves the selected objective. Each claim requires different evidence.
Specify the supported region
Define the new path with a compact predicate over device, architecture, datatype, shape, stride, alignment, and mode. The predicate belongs near dispatch code and in the design description. A reviewer should be able to identify which users receive the new behavior and which users retain the previous implementation.
Test the boundaries of this predicate. If width must be divisible by 16, include 15, 16, and 17. If a path starts at a row-count threshold, test immediately below and above it. If only contiguous tensors are supported, add a transposed or sliced case and confirm correct fallback behavior.
Preserve the public operator contract
The public operator contract can be wider than one optimized implementation. An existing operator may support several devices, layouts, datatypes, or mutation modes even when the new kernel supports only one subset. Dispatch should select the new path inside its eligibility region and preserve an established implementation elsewhere. Do not turn a fast-path limitation into a new user-visible error unless the project explicitly approves an API change.
Compatibility includes more than accepted shapes. Preserve documented aliasing, output layout, numerical tolerance, determinism, exception behavior, autocast, gradients, and compilation behavior. When the optimization intentionally changes one of these properties, review it as a semantic change with separate justification and migration evidence. A performance table cannot authorize an unstated contract change.
Build the correctness evidence
Compare against a trusted reference across devices, datatypes, shapes, strides, and numerical extremes. Add deterministic cases for known edge behavior and randomized cases for broader coverage. Test forward, backward, fake execution, compilation, and autocast when the operator promises those modes.
Numerical comparison does not detect every memory error. Use sanitizer or race tools for out-of-bounds access and synchronization defects where available. Keep continuous-integration tests small and deterministic. Preserve larger sweeps in a reproducible benchmark or scheduled validation job.
Build the performance evidence
Measure a representative shape matrix and include the appropriate library or existing implementation. Record warm-up, synchronization, timing statistic, software versions, target device, and raw results. Show neutral and slower cases as well as improvements. The neutral and slower cases justify the dispatch boundary and prevent reviewers from extrapolating beyond the data.
Connect the result to a mechanism. Report removed bytes, fewer launches, improved instruction selection, lower spills, or better load balance as applicable. Then run an enclosing model or workload replay. A fast microbenchmark has limited value when conversions, compilation, or a low call frequency remove its end-to-end effect.
Make maintenance cost reviewable
Explain why an existing primitive, compiler change, or library configuration does not solve the problem. State the architecture and datatype maintenance surface. If code generation or templates increase compilation time or binary size, measure that change. Include a fallback and an owner for future architecture updates.
Keep semantic changes separate from mechanical refactoring when practical. A focused patch helps reviewers associate each test and benchmark with the code that caused it. Documentation should explain the schedule and supported range without requiring a reader to reconstruct them from template parameters or profiler screenshots.
Worked example: Preparing a kernel optimization for review
A new Triton path reduces time by 18% for a common normalization shape but increases time for very small rows.
- Define an eligibility threshold backed by a shape sweep and leave the existing path for smaller rows. Test dimensions immediately around the threshold.
- State whether the threshold is expected to vary by device. If it is architecture-specific, keep it behind the corresponding device selection rather than presenting it as universal.
- Add numerical tests for supported dtypes, odd widths, non-finite inputs according to semantics, and compiled execution. Include backward if the operator promises it.
- Publish benchmark scripts and raw results across at least the important shape families and target devices. Explain the traffic or instruction change that causes the gain.
- Run an enclosing model or workload replay to show weighted value. Document compilation impact, fallback behavior, and unsupported architectures.
- Include the small-row regression in the published table. The regression is the evidence for retaining the old path below the threshold.
Conclusion: The change is reviewable because its faster input region, causal mechanism, correctness range, and maintenance boundaries are explicit.
Common errors
- Submitting only the best number. Reviewers need the shape surface, regressions, and reproducible method.
- Overfitting to one device. Architecture-specific code is valid when scoped, but dispatch and fallback must be deliberate.
- Treating readability as unrelated to performance. Unmaintainable tuning decays quickly as compilers, hardware, and workloads change.
- Combining a broad refactor with a new fast path and one benchmark. Reviewers cannot isolate the semantic and performance effects.
Reference summary
The public operator contract and the optimized path's eligibility predicate are different boundaries. A fast path can support only selected architectures, datatypes, layouts, or shapes while the operator continues to support a wider input set through existing implementations. A performance change must not silently narrow established behavior.
- Isolate a minimal feature or regression with a reproducible benchmark.
- Add correctness tests before or with the optimized path.
- Document supported and fallback shapes, devices, dtypes, and layouts.
- Show results across a representative matrix, including losses or neutral cases.
- Explain compilation, binary-size, and maintenance impact.
- Respond to review by strengthening evidence, not defending sunk work.
Knowledge check
Why should a performance pull request include shapes where the new path is not faster?
Show the answer guide
- A performance pull request changes dispatch behavior for an input domain, not only for the shapes on which the new kernel wins. Users need to know the complete boundary of that change.
- Shapes where the new path is slower reveal regressions, crossover points, alignment effects, or missing eligibility conditions. They let reviewers require a fallback or narrower dispatch predicate before merging.
- Reporting only wins creates selection bias and can move real workloads onto a slower path. Representative neutral and losing cases make the claimed aggregate benefit reproducible.
- The complete table also becomes a baseline for future devices and compiler versions, where the crossover can move even if correctness stays unchanged.
Prepare a reviewable performance change
Optimize one custom operator and declare its intended input domain before benchmarking. Select at least 12 cases spanning small and large sizes, aligned and odd dimensions, contiguous and supported strided layouts, and at least two cases expected not to improve. Implement an eligibility predicate and retain a correct fallback.
Evidence to keep: Submit the patch, correctness and fallback tests, a benchmark matrix containing every predeclared case with baseline and new latency, environment metadata, and a dispatch-boundary plot or table. Write a concise PR note that identifies wins, neutral cases, losses, and any scope restriction caused by the data.
Framework integration exposes the program to graph compilers. Understanding capture and lowering shows when a custom kernel is needed and when compiler optimization can create it automatically.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.