Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
How does LLM distillation work?
LLM distillation works by training a student model to learn from the outputs or probability distributions of a stronger teacher model. The teacher provides supervision; the student updates its own parameters to reproduce useful teacher behavior. Step by step workflow 1. Choose a teacher and student. Select a capable teacher and a student small enough to meet the intended memory, latency, or deployment budget. 2. Prepare representative prompts. Include examples resembling the real tasks the student must handle. Keep separate validation and test sets. 3. Collect teacher supervision. The teacher may generate answers, labels, explanations, or next token probability distributions. Which signals are accessible depends on the teacher and API. 4. Train the student. Compare student predictions with teacher targets, calculate a loss, and update the student's weights using backpropagation. The teacher normally remains frozen. 5. Evaluate and refine. Measure task accuracy, generalization, safety, latency, and cost on held out examples. Improve data or training if necessary. 6. Deploy the student. If its quality is sufficient, serve the student without running the teacher for every user request. What loss does the student minimize? There are two widely used forms of supervision: Response level distillation: Train on the teacher's generated answers, often with standard cross entropy loss over the answer tokens. This works even when the teacher exposes only text. Logit level distillation: Use the teacher's next token probabilities as soft targets , often with a KL divergence based objective. This preserves information about alternative tokens that a single generated answer omits. Some methods combine these losses. When teacher and student use different tokenizers, directly aligning their token probabilities may require special handling. Simple example Suppose a teacher classifies a support message: A distribution matching loss encourages the student to move toward the teacher's distribution. The teacher communicates not only that Billing is the most likely answer, but also that Account is a plausible alternative. For generative LLMs, this idea applies to next token distributions , not just three classification labels. Offline vs. on policy distillation Offline (off policy): Train the student from a fixed dataset of teacher generated responses or teacher scores. It is comparatively simple, but the student may encounter different continuations when generating on its own. On policy: Let the student generate completions , then have the teacher evaluate those continuations. This can help the student learn from its own mistakes and reduce training–inference mismatch. For example, Hugging Face TRL's DistillationTrainer implements on policy training that matches student and teacher token distributions. What are the main limitations? Distillation requires teacher inference and student training, may transfer teacher errors, and does not guarantee the student matches every teacher capability. Good results on a narrow task do not imply equivalent general intelligence. See “What is model distillation for LLMs?” for the conceptual overview. In short: collect teacher supervision, optimize the student's predictions against it, and test the smaller model on real workloads before deploying it independently. References: Hugging Face DistillationTrainer, Distilling Step by Step!, and On Policy Distillation of Language Models.
What is speculative decoding, and how does it accelerate LLM inference?
Speculative decoding is an LLM inference technique that lets a fast draft model propose several output tokens, then uses the larger target model to verify them together. When enough draft tokens are accepted, the target model needs fewer sequential decoding passes, reducing response latency. The key idea is: make cheap predictions first; verify several of them with the expensive model in parallel. How does it work? 1. Draft: A small, fast model proposes a short sequence of candidate tokens. 2. Verify: The target LLM evaluates those candidates in one forward pass. Because all proposed tokens are available as input, its causal attention can evaluate several positions together. 3. Accept or correct: The algorithm accepts a valid consecutive prefix of the draft. At the first rejected token, the target model supplies a corrected continuation using the appropriate verification rule. 4. Repeat: The process continues until generation ends. A practical example Suppose a draft model predicts the next four tokens as: The target model may accept the first three but reject "London" and generate "Paris" instead. The final output follows the target model's verified decision, not the draft model's guess. For greedy decoding , a candidate is accepted when it matches the target's required next token. For sampling , standard speculative decoding uses an acceptance/rejection and correction rule , rather than merely comparing the most likely tokens. Why can this be faster? Ordinary autoregressive decoding typically needs another large model forward pass before each next token can be chosen. Speculative decoding can validate several proposed positions in one target model pass . That is useful because short sequence decoding on modern accelerators is often limited by memory movement and sequential execution, while checking a batch of proposed positions can use the hardware more efficiently. The speedup comes from fewer serial target model calls , minus the extra cost of drafting and verification. It is not a guaranteed 2× improvement . Does speculative decoding reduce accuracy? Standard, correctly implemented speculative sampling can preserve the target model's exact output probability distribution (subject to normal numerical and implementation constraints). The draft is only a proposal mechanism; the target's distribution remains authoritative. This differs from using a smaller model alone or changing the target with quantization. It does not mean independent random runs must produce the same exact words. When does it work well—and when not? It works best when: The draft model is much cheaper than the target. The target accepts many proposed tokens . The workload benefits from lower decode latency . It can disappoint when the draft is slow or inaccurate, proposals are frequently rejected, verification has high overhead, or the serving system already uses highly efficient large batch decoding. Some implementations also place constraints on batched inputs and tokenizer compatibility. Related techniques KV caching saves previously computed attention keys and values; it tackles repeated work, not the same serial token bottleneck. Self speculative decoding can draft with a model's own early layers. Prompt lookup decoding can propose tokens copied from matching prompt segments. Distillation may create a suitable small draft model, but speculative decoding itself does not require training the target. In short: speculative decoding trades a little cheap drafting for fewer expensive, sequential target model passes—potentially making LLM generation faster without changing the target model's sampling distribution . References: Hugging Face: Assisted decoding, Fast Inference from Transformers via Speculative Decoding, and Accelerating LLM Decoding with Speculative Sampling.
What is continuous batching in LLM inference?
Continuous batching (also called iteration level batching ) is an LLM serving technique that dynamically groups requests at each generation step , rather than keeping the same requests in a fixed batch until all of them finish. The main benefit is higher GPU utilization and serving throughput when many users send requests of different lengths. Why is ordinary batching inefficient? Suppose three requests arrive together: A: Generate 10 tokens. B: Generate 100 tokens. C: Generate 30 tokens. In a static batch , the serving system may wait for the whole group to finish before admitting another batch. Once A and C finish, their slots may be wasted while B continues. With continuous batching , A leaves the running batch as soon as it finishes, and a new request, D , can join while B and C continue. This is a simplified illustration: real schedulers also manage prompt processing, token budgets, and memory availability. How does continuous batching work? 1. Queue requests: Incoming prompts wait for capacity. 2. Schedule a step: The server chooses active requests for the next model iteration. 3. Run the model: Each decoding request typically advances by one token in that iteration. 4. Update state: Completed requests leave; unfinished requests retain their own KV cache state . 5. Admit new work: Waiting requests join when capacity permits, and the next iteration begins. The scheduler can therefore change the active batch without restarting unfinished generations . Why does it improve inference performance? Less idle GPU capacity: A short request finishing does not force the system to wait for longer requests. Higher aggregate throughput: More requests or output tokens may be processed over time. Better handling of variable length outputs: Requests with different generation lengths can share hardware efficiently. Potentially lower queueing delay: New requests need not wait for the slowest member of a previous static batch. These are serving level improvements ; continuous batching does not automatically reduce the compute required for a single token or guarantee lower latency for every individual request. How does it relate to KV cache and prefill? KV cache: Each active sequence retains attention keys and values so it can resume decoding across iterations. Prefill: Newly admitted prompts must first be processed to build their KV caches. Chunked prefill: A separate but complementary technique that divides long prompt processing work into smaller pieces, allowing the scheduler to interleave prefill and decoding rather than blocking decoding with one very long prompt. What are the trade offs? Continuous batching needs a scheduler that respects GPU memory and compute limits. Too many active sequences can exhaust KV cache capacity , increase per token latency, or create unfair waiting times. Systems use limits on active sequences, batched tokens, queue depth, and admission to balance throughput, responsiveness, and memory. Serving engines such as vLLM and Hugging Face Transformers serving use dynamic request scheduling, with implementation specific policies. What should you monitor in production? Track requests per second , output tokens per second , time to first token (TTFT) , time per output token (TPOT) , queueing delay, and GPU/KV cache utilization. Increasing throughput is useful only if user facing latency remains acceptable. In short: continuous batching keeps a changing group of LLM requests running together, adding and removing sequences at each generation step to use accelerator capacity more efficiently. References: Hugging Face: Continuous batching, ORCA: A Distributed Serving System for Transformer Based Generative Models, and vLLM scheduler configuration.
What is chunked prefill in LLM inference?
Chunked prefill is an LLM serving technique that processes a long input prompt in smaller groups of tokens across multiple scheduler iterations , rather than running the entire prompt in one large prefill operation. Its purpose is to let the serving system interleave prompt processing with ongoing token generation , helping balance responsiveness and GPU utilization when multiple requests are active. Why is it needed? LLM inference typically has two phases: Prefill: Process the prompt and build its attention KV cache . Long prompts can require substantial computation. Decode: Generate output tokens one at a time using the stored KV cache. If a long prompt occupies an entire scheduling step, users whose requests are already decoding may have to wait longer for their next token. Chunked prefill makes the long prefill work easier to schedule alongside those decodes. Practical example Suppose a new request contains 6,000 input tokens , and the system allows at most 2,048 prefill tokens per chunk when sufficient scheduling capacity is available: Actual chunk sizes depend on the scheduler's available token budget and concurrent workload. The example is illustrative, not a fixed vLLM configuration. The model processes the chunks in original token order and reuses previously computed attention state. This is not splitting a document into independent summaries or changing the prompt's meaning. How does the scheduler handle chunked prefill? 1. A long prompt enters the prefill queue. 2. The scheduler chooses a chunk that fits the current token budget. 3. The model processes that chunk and preserves its KV cache state. 4. Other requests can continue decoding during interleaved scheduling steps. 5. After the entire prompt is processed, the new request begins output generation. No ordinary answer tokens are generated for that new request until its prompt has been fully processed. How does it improve performance? Reduces long prefill interruptions: Existing decode requests can receive more regular opportunities to generate tokens. Balances GPU work: Compute heavy prefill and memory intensive decode workloads can share scheduled batches. Supports throughput/latency trade offs: A scheduler can control how much prefill work to include per iteration. However, chunking does not eliminate prompt computation, automatically reduce total FLOPs, or guarantee lower time to first token (TTFT) for the long prompt request. What are the trade offs? Smaller chunks: May improve inter token latency (ITL) for active decodes, but increase scheduling or kernel launch overhead and delay completion of long prefills. Larger chunks: May process prompts more efficiently, but can interrupt decoding for longer. Capacity limits: Chunked requests still require KV cache memory and fair scheduling; poorly tuned policies can increase queueing delays. For example, vLLM exposes a max num batched tokens budget that influences how many tokens can be processed per iteration. Chunked prefill vs. continuous batching Continuous batching: Dynamically admits and removes requests as generation proceeds. Chunked prefill: Divides a request's prompt processing phase across iterations. They are complementary: continuous batching decides which requests share an iteration; chunked prefill helps decide how much prompt work can share it with decoding. In short: chunked prefill breaks long prompt computation into schedulable pieces, helping LLM servers balance throughput, existing users' decode latency, and new requests' TTFT. References: vLLM scheduler configuration, SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills, and Sarathi Serve.
What is Paged KV Cache?
Paged KV cache is a memory management approach for LLM inference that stores attention keys (K) and values (V) in small, fixed size blocks instead of requiring one large contiguous allocation for every sequence. The idea comes from operating system virtual memory paging and is central to PagedAttention , introduced by the vLLM research paper. Why is ordinary KV cache allocation challenging? Prompt and generated response lengths vary between requests. The cache grows as more tokens are generated. Reserving a large contiguous buffer can waste unused space; repeatedly allocating and freeing differently sized buffers can lead to fragmentation. This wasted memory limits the number of requests a GPU can serve simultaneously. A paged cache allocates blocks as needed , so a sequence can grow without relocating a large contiguous cache buffer. How does it work? 1. Divide the logical sequence into blocks. Each block holds K/V data for a fixed number of token positions. 2. Allocate physical blocks on demand. The blocks do not have to sit next to one another in GPU memory. 3. Track a block table. This maps the sequence's logical block positions to actual physical KV cache blocks. 4. Read K/V through the mapping. A compatible attention kernel gathers the relevant keys and values during inference. 5. Grow or release blocks. As tokens arrive, the system allocates additional blocks; completed requests can release their allocations. Some implementations also support sharing identical cached prefixes. Simple example Assume each KV cache block holds 16 tokens . A request with 35 cached tokens requires: The three physical blocks are not contiguous . Only 13 positions in the final block are unused; the earlier two blocks are full. This illustrates a key benefit: fixed size block allocation limits much of the waste to the partially filled final block , although real implementations still have block table and allocator overhead. What are the advantages? Less fragmentation and over reservation: Cache memory can be allocated incrementally in reusable blocks. Higher potential concurrency: Better cache utilization can let the server admit more sequences before reaching its memory limit. Flexible growth: Requests can generate different numbers of tokens without requiring one pre sized contiguous region. Possible prefix sharing: Under supported conditions, requests can point to shared cached blocks rather than duplicating identical K/V data. Writable shared blocks require appropriate protection, such as copy on write. What does it not do? Paging is not KV cache quantization. It does not inherently reduce the number of K/V elements stored for a given attention context or change their precision. Nor does it eliminate attention calculations or guarantee faster inference for every workload. It improves how KV cache memory is allocated, addressed, and reused . Compatible kernels and metadata lookups introduce engineering overhead, so actual throughput gains must be benchmarked. How is it different from PagedAttention? Paged KV cache: The storage organization —K/V tensors in physical blocks with logical to physical mappings. PagedAttention: An attention computation approach that accesses K/V using those block mappings. The two work together in serving systems such as vLLM . For prerequisites, see “What is KV cache in an LLM?” and “Why does KV cache consume significant memory?” . A separate follow up will cover paged attention implementations. In short: Paged KV caching makes an LLM's growing attention memory more flexible by using reusable, non contiguous blocks and a block table, reducing allocation waste and supporting more concurrent requests. References: PagedAttention / vLLM paper, vLLM: Paged Attention kernel, and vLLM: Block table.
What is Direct Paged Attention?
Direct Paged Attention describes an implementation of attention in which a GPU kernel reads keys and values directly from a paged KV cache using a block table , instead of first gathering the entire cache into one contiguous tensor. Terminology note: "Direct Paged Attention" is not a universally standardized name for a separate Transformer attention algorithm. It is a useful description of the direct block table access approach used by paged attention kernels. Why is direct access useful? A paged KV cache stores past attention keys and values in fixed size physical blocks that may be scattered across GPU memory . The tokens still have a logical sequence order. A simpler implementation might: 1. Follow the block table. 2. Gather/copy all K/V blocks into a contiguous buffer. 3. Run a conventional attention kernel on that buffer. Direct paged attention combines reading the paged cache and computing attention in the kernel, avoiding that extra full cache materialization. How does the kernel find the correct token? Suppose each KV block holds 16 token positions and the sequence's block table is: To read logical token position 20 (zero based indexing): The kernel repeats these mapped reads for the needed token positions while calculating attention for the current query. What computation happens? For the new token's query (Q) , a direct paged attention kernel: Looks up physical block locations through the block table . Loads the relevant cached K/V tiles . Calculates query–key scores and the softmax weighted value aggregation. Produces the attention output without requiring a separate dense copy of the full KV cache . Efficient kernels may use tiled loads, parallel reductions, and numerically stable online softmax . The exact GPU implementation varies. Direct paged attention vs. related concepts Paged KV Cache: The memory layout —K/V stored in reusable physical blocks. PagedAttention: Attention that accesses those blocks through logical to physical mappings. Direct paged attention kernel: An implementation that uses those mappings inside the attention computation , rather than gathering a contiguous KV tensor first. Paged FlashAttention: Refers to combining paged KV access with FlashAttention style tiled/IO aware attention ; the exact implementation depends on the library. Does direct paged attention always run faster? No. It can avoid intermediate allocations and extra KV data copies, which is valuable for long contexts and memory bandwidth sensitive decoding. But indirection, irregular memory access, block table lookups, and kernel overhead can reduce or outweigh the benefit, especially for small or single request workloads. For example, a published from scratch MoE inference engine project reports a direct paged attention Triton kernel that performed better than its earlier gather based path in one benchmark, but still underperformed its contiguous cache path . Those figures are workload specific, not general speedup guarantees. Correctness requirement: When the same tokens are attended to, direct and gather based implementations should produce equivalent attention outputs within floating point tolerances . A wrong block table mapping can silently produce incorrect results. For prerequisites, see "What is Paged KV Cache?" and "What is KV cache in an LLM?" In short: Direct Paged Attention removes an unnecessary intermediate gather by making the attention kernel read the non contiguous KV cache through its block table . It optimizes memory access and data movement, not the underlying attention formula. References: vLLM Paged Attention kernel documentation, PagedAttention paper, and From Scratch MoE Inference Engine implementation notes.
What is Paged FlashAttention?
Paged FlashAttention refers to using a FlashAttention style, IO aware attention kernel directly with a paged KV cache . The keys and values for previous tokens live in fixed size memory pages, while the kernel uses a block table to find them and computes attention in tiles. Terminology: "Paged FlashAttention" is a descriptive term, not a universally standardized, separate attention algorithm. The official FlashAttention implementation supports paged KV cache access, including a block table argument in flash attn with kvcache. Why combine paging with FlashAttention? These techniques address different sources of memory inefficiency: Paged KV cache: Stores keys and values in reusable, potentially non contiguous physical blocks, reducing wasted capacity from large per request reservations. FlashAttention: Uses tiling and online softmax to compute exact attention without materializing a large attention score matrix in GPU high bandwidth memory. Together: An appropriate kernel follows the KV block mapping while performing tiled, IO aware attention, avoiding a separate full cache gather into a contiguous tensor. How does it work? 1. Locate the KV pages. The request has a block table mapping logical token blocks to physical cache blocks. 2. Load small K/V tiles. The kernel fetches the required key and value data from those physical blocks. 3. Compute attention incrementally. It forms query–key scores for each tile and maintains the normalization using an online, numerically stable softmax . 4. Accumulate the output. It combines the partial results without storing a full attention score matrix or reconstructing a full contiguous KV cache first. Practical example Suppose a cache page holds 256 tokens . A sequence has 600 tokens , with this illustrative mapping: The attention kernel reads the relevant K/V data from physical page 1, offset 44 , and processes it as part of a tile. The physical pages do not need to sit next to one another. Note: The 256 token page size is illustrative and is also a supported page granularity in FlashAttention 2's documented paged KV API; page size constraints vary by kernel and version. How is it different from Direct Paged Attention? Direct Paged Attention: Emphasizes reading paged K/V inside the attention kernel , without a separate gather. Paged FlashAttention: Adds the specific FlashAttention style tiled, IO aware computation and online softmax strategy . The two descriptions can refer to overlapping implementations. They are not necessarily competing products or mutually exclusive algorithms. Does it change the attention result or complexity? The intention is to compute the same exact attention operation up to floating point differences . For standard full attention, FlashAttention mainly reduces memory traffic and temporary storage ; it does not turn quadratic prefill attention arithmetic into linear time computation. In single token decoding , a full quadratic score matrix is not the main issue. Efficient reads from the growing KV cache, kernel overhead, and GPU utilization are often more important. Does it always improve inference speed? No. Performance depends on GPU architecture, kernel/backend, context length, page layout, batch size, and competing overheads. Block table indirection and irregular accesses can cost time, so benchmark time to first token, time per output token, throughput, and peak memory . See "What is Paged KV Cache?" for the memory organization and "What is Direct Paged Attention?" for direct mapped reads. In short: Paged FlashAttention combines paged KV cache access with FlashAttention's tiled, IO aware attention computation , reducing avoidable data movement while preserving the underlying attention operation. References: Official FlashAttention implementation, FlashAttention paper, and PagedAttention paper.
What is Mixture of Experts (MoE) in Large Language Models?
Mixture of Experts (MoE) is a Transformer architecture that contains multiple smaller expert networks but activates only a subset for each token. A learned router chooses which experts process the token, allowing a model to have many total parameters without using all of them for every token. In most modern MoE LLMs, experts replace some feed forward (MLP/FFN) sublayers . Attention and other parts of the Transformer are still shared. How does MoE work? 1. Token representation: A token reaches an MoE layer after earlier Transformer operations. 2. Routing: A small routing network scores the available experts. 3. Top k selection: The router activates the best scoring one or few experts for that token. 4. Expert computation: Only those selected expert FFNs process the token. 5. Combine: Their outputs are weighted by the router scores and passed to the next layer. Different tokens can select different experts , even within the same prompt. An expert is a learned subnetwork—not necessarily a predefined "math expert" or "coding expert." Real world example: Mixtral 8x7B Mixtral 8x7B has eight feed forward experts in each MoE layer. Its router chooses two experts per token per layer . The published architecture has approximately 47 billion total parameters , but roughly 13 billion active parameters per token , including shared components. This example shows why total parameter count and active parameter count are different . The exact compute and speed also depend on attention, batch size, hardware, and how experts are executed. Why use MoE instead of a dense LLM? Sparse computation: Only a few expert FFNs run for each token, potentially increasing model capacity without proportional per token computation. Scaling: Model designers can add expert capacity while keeping a limited number active. Serving trade off: Inference may be efficient relative to a dense model with the same total parameter count , but MoE is not automatically faster than a smaller dense model . What makes MoE challenging? Expert memory: All experts—or their distributed partitions—must be available for inference. Sparse activation does not mean unused expert weights consume no memory. Load imbalance: A router may repeatedly favor a few experts, leaving others underused or overloaded. Training often uses load balancing techniques. Communication: When experts run on different GPUs, routing tokens between devices introduces overhead. Implementation complexity: Expert batching, routing, capacity management, and numerical stability require careful engineering. MoE versus multi agent systems MoE experts are neural network components selected inside the model , not independent AI agents holding separate conversations or calling external tools. For background, review "What are RMSNorm and MLP in Transformer architectures?" when it is available, or earlier MachinoAI Transformer architecture and attention questions. In short: MoE increases a model's total capacity by routing each token through only a few expert networks. It saves active computation compared with using every expert, but still brings memory, routing, and communication costs. References: Hugging Face: Mixture of Experts in Transformers, Mixtral of Experts, and Switch Transformers.
How do you benchmark and profile LLM inference performance?
Benchmarking LLM inference means measuring how quickly a model serving system handles a controlled, representative workload . Profiling goes one step further: it identifies where the time and GPU resources are spent, so you can explain slow results and choose an optimization. A useful benchmark answers "How fast and for how many users?" A profiler answers "What is making it slow?" 1. Choose the metrics Time to First Token (TTFT): Time from sending a streaming request until the first output token is received. It includes effects such as queuing and prompt prefill. Time Per Output Token (TPOT) / Inter Token Latency (ITL): Time between output tokens after the first token. Tool definitions vary, so keep the same measurement method when comparing systems. End to end latency: Total time from sending the request until its response finishes. Throughput: Successful requests per second ( RPS ) and generated output tokens per second ( tokens/s ). GPU memory and utilization: Peak allocated/reserved memory, KV cache occupancy, compute utilization, and memory bandwidth where measurable. Tail latency and reliability: Report p50, p95, and p99 , errors, timeouts, and rejected requests—not just the mean. 2. Design a fair workload Fix and record the model version, quantization, GPU, number of GPUs, inference engine, precision, batching settings, and maximum context . Use prompts representative of production and record their input and output token lengths . Test several cases, for example: Short prompt / short response: Chat responsiveness. Long prompt / short response: Prefill and TTFT pressure. Short prompt / long response: Decode and KV cache pressure. Mixed prompts with concurrent users: Realistic serving behavior. Run a warm up before collecting measurements, repeat tests, and ensure the benchmark client and network do not become the bottleneck. Compare systems using the same dataset, tokenizer aware token counts, decoding settings, and load pattern . 3. Run a load benchmark A vLLM server can be benchmarked with the vllm bench serve command. For example, after starting a matching vLLM API server: This is an illustrative load test configuration , not a performance claim. The model name and endpoint must match the running service, and the server must have sufficient resources. For more realistic traffic, replay anonymized production request lengths and arrival patterns. NVIDIA AIPerf also supports concurrency sweeps, request rate tests, and latency/throughput reporting. 4. Interpret the results Imagine a hypothetical 20 second test completes 120 requests , each producing exactly 128 output tokens : Suppose the same test shows p95 TTFT = 1.8 s and p95 TPOT = 75 ms . Throughput alone might look acceptable, but the tail delays may be poor for interactive chat. Report both metrics, not just tokens/s. Increase request rate or concurrency gradually and observe where throughput stops improving while queuing, TTFT, or TPOT worsens. The best production capacity is often the highest throughput that still meets latency SLOs (sometimes called goodput ). 5. Profile the bottleneck Once a benchmark reveals a regression, collect a small representative trace: High TTFT: Inspect request queues, tokenization, long prompt prefill, and prefill scheduling. High TPOT / ITL: Inspect decode kernels, KV cache reads, memory bandwidth, batching, and synchronization. Low throughput with idle GPU: Check scheduling gaps, small batches, CPU overhead, and data transfers. Out of memory or falling concurrency: Inspect model weight memory, KV cache capacity, fragmentation, and allocation policies. Use PyTorch Profiler for operator and GPU kernel traces, and hardware profilers or GPU telemetry for detailed device behavior. Be careful: profiling itself adds overhead ; use unprofiled runs for final benchmark numbers. Common benchmarking mistakes Measuring one request and claiming production throughput. Mixing input and output tokens in one tokens/s figure without labeling it. Comparing average latency while ignoring p95/p99. Comparing models with different prompt lengths, output limits, or hardware. Benchmarking the service with an overloaded load generator. In short: Benchmark across realistic input lengths and concurrency levels, report TTFT, TPOT/ITL, latency percentiles, throughput and memory, then profile the slowest stage before optimizing. References: vLLM benchmarking CLI, NVIDIA AIPerf CLI, NVIDIA inference metrics definitions, and PyTorch Profiler.
What are fused model kernels, and why are they important?
Fused model kernels are GPU programs that perform multiple consecutive neural network operations in one kernel launch , rather than running a separate kernel for every operation. During LLM inference, this can reduce GPU memory transfers , temporary tensors , and kernel launch overhead . A kernel is a function executed on the GPU. Fusion means combining compatible operations while preserving the intended computation (subject to ordinary floating point differences). How does kernel fusion work? Imagine an LLM layer performing three operations: When feasible, intermediate values can stay in registers or on chip memory instead of being written to and read from slower GPU global memory. A numerical example Suppose two intermediate tensors each contain 10 million FP16 elements . One intermediate tensor occupies approximately 20 MB . Writing and then rereading it transfers roughly 40 MB . Avoiding that round trip for two intermediates can eliminate roughly 80 MB of global memory traffic . This is an illustrative upper level estimate: caches, compiler decisions, actual tensor layouts, and hardware behavior affect real traffic and speedups. Where are fused kernels used in LLMs? Residual addition + normalization: Some implementations combine compatible elementwise and normalization work. Bias + activation: For example, adding a bias and applying GELU or SiLU in one kernel. Gated MLP operations: Some kernels combine activation and elementwise multiplication, as used in SwiGLU style feed forward blocks. Fused attention: FlashAttention is an important example of combining attention sub operations through tiling and online softmax, reducing intermediate attention memory traffic. Not every operation can be fused efficiently. Large matrix multiplications, reductions, and attention have different computational and synchronization requirements, so their best fusion strategies vary. Why does fusion improve LLM performance? 1. Fewer launches: A single GPU kernel can replace several launches, reducing CPU side dispatch and GPU scheduling overhead. 2. Less memory traffic: Intermediate tensors may never need to reach GPU global memory. 3. Less temporary storage: Fewer large intermediate buffers may reduce peak memory. 4. Potentially lower inference latency: These savings can matter especially when processing small batches or short decode steps. How can you use fused kernels? You do not always need to write GPU code manually. Supported operations can be fused by compilers such as PyTorch torch.compile / TorchInductor , or implemented directly using Triton or CUDA. A minimal PyTorch example: The compiler may fuse compatible operations for supported devices and shapes. It does not guarantee one kernel or a speedup; compilation itself has an initial cost. When can fusion hurt performance? An oversized fused kernel may use too many registers , lower GPU occupancy , require extra synchronization, or prevent reuse of intermediates needed elsewhere. Fusion may also be limited by dynamic shapes, unsupported operators, or numerical requirements. Measure before and after: Compare warm run TTFT, TPOT, throughput, GPU memory, and profiler traces under the same workload . A faster microbenchmark does not necessarily mean an entire LLM service becomes equally faster. For a measurement workflow, see “How do you benchmark and profile LLM inference performance?” In short: Fused kernels run compatible model operations together, primarily saving GPU launch and memory movement costs. Their value depends on the operation, model workload, and hardware. References: Triton: Fused Softmax, PyTorch: Why Is PyTorch Compile So Fast—Kernel Fusion, and NVIDIA: Kernel Fusion in CUDA.