What are fused model kernels, and why are they important?

Fused model kernels are GPU programs that perform multiple consecutive neural-network operations in one kernel launch, rather than running a separate kernel for every operation. During LLM inference, this can reduce GPU memory transfers, temporary tensors, and kernel-launch overhead.

A kernel is a function executed on the GPU. Fusion means combining compatible operations while preserving the intended computation (subject to ordinary floating-point differences).

How does kernel fusion work?

Imagine an LLM layer performing three operations:

Unfused:
  Kernel 1: add residual
       ↓ write intermediate tensor to GPU memory
  Kernel 2: add bias
       ↓ write another intermediate tensor
  Kernel 3: apply GELU activation
       ↓ write final result

Fused:
  One kernel: load inputs → add residual → add bias → GELU
       ↓ write final result

When feasible, intermediate values can stay in registers or on-chip memory instead of being written to and read from slower GPU global memory.

A numerical example

Suppose two intermediate tensors each contain 10 million FP16 elements.

• One intermediate tensor occupies approximately 20 MB.
• Writing and then rereading it transfers roughly 40 MB.
• Avoiding that round trip for two intermediates can eliminate roughly 80 MB of global-memory traffic.

This is an illustrative upper-level estimate: caches, compiler decisions, actual tensor layouts, and hardware behavior affect real traffic and speedups.

Where are fused kernels used in LLMs?

• Residual addition + normalization: Some implementations combine compatible elementwise and normalization work.
• Bias + activation: For example, adding a bias and applying GELU or SiLU in one kernel.
• Gated MLP operations: Some kernels combine activation and elementwise multiplication, as used in SwiGLU-style feed-forward blocks.
• Fused attention: FlashAttention is an important example of combining attention sub-operations through tiling and online softmax, reducing intermediate attention-memory traffic.

Not every operation can be fused efficiently. Large matrix multiplications, reductions, and attention have different computational and synchronization requirements, so their best fusion strategies vary.

Why does fusion improve LLM performance?

1. Fewer launches: A single GPU kernel can replace several launches, reducing CPU-side dispatch and GPU scheduling overhead.
2. Less memory traffic: Intermediate tensors may never need to reach GPU global memory.
3. Less temporary storage: Fewer large intermediate buffers may reduce peak memory.
4. Potentially lower inference latency: These savings can matter especially when processing small batches or short decode steps.

How can you use fused kernels?

You do not always need to write GPU code manually. Supported operations can be fused by compilers such as PyTorch `torch.compile` / TorchInductor, or implemented directly using Triton or CUDA.

A minimal PyTorch example:

PYTHON
import torch
import torch.nn.functional as F

def block(x, residual, bias):
    return F.gelu(x + residual + bias)

optimized_block = torch.compile(block)

The compiler may fuse compatible operations for supported devices and shapes. It does not guarantee one kernel or a speedup; compilation itself has an initial cost.

When can fusion hurt performance?

An oversized fused kernel may use too many registers, lower GPU occupancy, require extra synchronization, or prevent reuse of intermediates needed elsewhere. Fusion may also be limited by dynamic shapes, unsupported operators, or numerical requirements.

Measure before and after: Compare warm-run TTFT, TPOT, throughput, GPU memory, and profiler traces under the same workload. A faster microbenchmark does not necessarily mean an entire LLM service becomes equally faster.

For a measurement workflow, see “How do you benchmark and profile LLM inference performance?”

In short: Fused kernels run compatible model operations together, primarily saving GPU launch and memory-movement costs. Their value depends on the operation, model workload, and hardware.

References: Triton: Fused Softmax, PyTorch: Why Is PyTorch Compile So Fast—Kernel Fusion, and NVIDIA: Kernel Fusion in CUDA.