Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
What is time to first token (TTFT)?
Time to First Token (TTFT) measures how long a user waits from submitting an LLM inference request until the first generated output token is received. It is a key indicator of how responsive a streaming chatbot feels. How is TTFT measured? For example: After that first token, the model may continue generating the rest of its answer for several seconds. What happens before the first token? A typical hosted inference request passes through these stages: 1. Transport and request handling: Network communication, authentication, routing, and prompt preparation. 2. Queueing: The request may wait for available GPU resources or a scheduling slot. 3. Prefill: The model processes the prompt to produce next token scores; a serving engine often builds its initial KV cache . 4. First token selection and delivery: The decoder selects a token and the server sends the first nonempty generated output to the client. What increases TTFT? Long prompts: More input tokens generally require more prefill work. Server congestion: High concurrency can increase queueing. Model and hardware: Larger models, memory pressure, or slower accelerators may delay processing. Cold starts and network overhead: Loading an inactive model or reaching a distant server can add delay. How can you reduce TTFT? Trim unnecessary prompt content without removing information required for a correct answer. Optimize prefill and serving: Use suitable hardware, efficient batching/scheduling, and prefix caching when supported and useful. Reduce queueing: Monitor request volume, concurrency, capacity, and admission limits. Stream responses: Streaming makes the first generated output visible sooner than waiting for a complete response; it does not necessarily speed up the model's first token computation. TTFT vs. other latency metrics TTFT: Time until the first generated output is received. Time per output token (TPOT): Average time between later output tokens, under a specified measurement convention. End to end latency: Time until the final response has been received. A system may have a fast TTFT but still take a long time to finish a lengthy answer. Measure TTFT and total latency together . Important measurement detail TTFT definitions can differ across benchmarking tools. Client observed TTFT includes network and queueing delays, while some server side measurements start later and exclude them. Ignore empty streaming events or headers that contain no generated content. Non streaming APIs generally cannot expose client observed TTFT without separate server instrumentation. For related concepts, see “What is inference latency in LLMs?” and “What is LLM throughput?” In short: TTFT measures how quickly an LLM begins responding, not how quickly it finishes. Lower TTFT usually makes interactive applications feel more responsive. References: NVIDIA LLM inference metrics and vLLM benchmark metric definitions.
What is time per output token in LLM inference?
Time Per Output Token (TPOT) measures the average time required to produce each additional output token after the first during an LLM inference request. It describes the speed of the decoding phase and is usually reported in milliseconds per token (ms/token) . How is TPOT calculated? A widely used client side definition, also used in vLLM serving benchmarks, is: TTFT (Time to First Token): Time until the first generated token arrives. End to end latency: Time until the final generated token arrives. Output tokens: Total number of generated tokens. The first token is excluded because the time spent waiting for it is already counted in TTFT, which includes work such as queueing and prefill . Practical example Suppose a request has: TTFT: 400 ms End to end latency: 2,000 ms Generated output: 41 tokens So, after the first token, this request delivers output at an average rate of about 25 tokens per second (1,000 / 40). What affects TPOT? Model size and hardware: GPU compute, memory bandwidth, and model architecture affect decoding speed. Active context and KV cache: Attention work and memory use can change as the sequence grows. Concurrent requests: Scheduling and contention can increase per request time between tokens. Serving optimizations: Efficient batching, quantization, and specialized decoding strategies may improve TPOT, depending on workload. TPOT vs. TTFT vs. throughput TTFT: How long the user waits for the first token. TPOT: How quickly subsequent tokens arrive on average for one request. Throughput: Total requests or tokens served per second across the system. Low TTFT makes the response start quickly ; low TPOT makes the response continue smoothly . A server can improve aggregate throughput through batching while individual requests experience higher TPOT. Is TPOT the same as inter token latency (ITL)? Not always. ITL often measures gaps between consecutive streamed output events . In ordinary one token per event streaming, ITL and TPOT are similar. With speculative decoding or chunked output, one event may contain several tokens, so their measured values can differ. Measurement notes: TPOT is undefined under the formula above for responses containing zero or one output token; benchmarking tools may omit those requests or record a placeholder. Compare benchmark definitions before comparing systems. For related concepts, see “What is time to first token (TTFT)?” , “What is inference latency in LLMs?” , and “What is LLM throughput?” In short: TPOT is the average time per generated token after the first , helping measure the responsiveness of ongoing LLM output. References: NVIDIA NIM inference metrics and vLLM benchmarking metric definitions.
What is KV cache in an LLM?
KV cache (Key–Value cache) is temporary inference time memory that stores the key (K) and value (V) tensors computed for previously processed tokens in a Transformer's attention layers. In an autoregressive LLM, each new output token must attend to earlier tokens. Instead of recalculating the earlier keys and values on every step, the model can reuse them from the cache. What exactly is stored? In self attention, a token produces query (Q) , key (K) , and value (V) representations. The KV cache retains K and V for past tokens, separately for each relevant attention layer . The current token's query can attend to those cached representations. A cache is not a copy of the original text, the model's trained weights, or a permanent memory of a conversation. Simple example Suppose a prompt has three tokens, and the model is generating a fourth: The model reuses the stored attention information from A, B, and C while processing D, and adds D's key/value tensors for later steps. When is the cache created? 1. Prefill: The model processes the prompt and stores its K/V tensors. 2. Decode: Newly processed tokens contribute additional K/V entries. 3. Request completion: Request specific cache memory can be released or reused by the serving system. Some serving systems also support prefix caching , letting requests reuse the computed K/V state of an identical shared prompt prefix when safe and supported. Why is KV cache important? Less repeated computation: Earlier K/V projections do not need to be recomputed for each new token. Faster autoregressive decoding: Token generation becomes more efficient. Memory cost: Storing K/V tensors for many layers, tokens, and concurrent requests can consume substantial accelerator memory. For a standard full attention cache, memory generally grows with the number of cached tokens. The exact amount depends on model layers, KV heads, head dimension, numerical precision, and the number of active sequences. Architectures with grouped query attention (GQA) or multi query attention (MQA) can use fewer KV heads; sliding window caches may retain only part of the context. What it does not do KV caching does not change what the model has learned, eliminate all attention computation, or make unlimited context free. It trades additional memory for reduced repeated work during inference. For related topics, see “How does LLM inference work?” , “How does KV caching speed up LLM inference?” , and “Why does KV cache consume significant memory?” In short: the KV cache is a reusable, per layer store of past attention keys and values that makes token by token LLM inference more efficient. References: Hugging Face: How caching works and Hugging Face: Cache strategies.
How does KV caching speed up LLM inference?
KV caching speeds up LLM inference by avoiding repeated computation for tokens the model has already processed. During autoregressive generation, each new token needs information from previous tokens. A key–value (KV) cache lets the model reuse their stored attention keys and values instead of processing the entire prefix again. How does it save work? In a causal Transformer, future tokens do not change the attention representations computed for earlier tokens. Once their keys and values are calculated, those tensors can be reused during later decoding steps. Without KV caching: Each generation step can require running the entire growing sequence through the model again. With KV caching: The model processes the current token, calculates its new query/key/value representations, attends to cached keys and values from earlier tokens, and adds the new key/value pair to the cache. Why does this improve performance? Avoids redundant projections and hidden state computation: Previously processed tokens need not be rerun through all Transformer layers at every generation step. Reduces per step attention work: In standard full causal attention, recomputing the whole prefix involves attention over many earlier query positions. Cached decoding instead computes attention for the new query against earlier keys, so the attention work per step grows roughly linearly , rather than quadratically, with context length. Improves token by token responsiveness: The savings are especially important during the decode phase , when output is produced sequentially. These are algorithmic differences, not a guaranteed fixed speedup percentage . Actual latency depends on model architecture, hardware, sequence length, batching, and cache implementation. Does KV caching also speed up prefill? Within a single request, the initial prefill still processes the input prompt and builds the cache. Standard KV caching mainly avoids repeated work during subsequent decoding . A separate optimization, prefix caching , can reuse suitable cached prompt prefix blocks across requests , reducing prefill work when the same prefix occurs again. What is the trade off? KV caching uses memory to save computation. The cache grows with retained token count and must store keys and values across applicable attention layers. Long contexts and many concurrent requests can put pressure on GPU memory; specialized cache allocation, quantization, or offloading can help, sometimes with their own performance costs. For the underlying concept, see “What is KV cache in an LLM?” . For the memory implications, see “Why does KV cache consume significant memory?” In short: KV caching accelerates autoregressive LLM inference because previous attention keys and values are reused instead of recalculated at each output step, trading extra memory for faster decoding. References: Hugging Face caching explanation and vLLM automatic prefix caching.
Why does KV cache consume significant memory?
KV cache consumes significant memory because an autoregressive LLM stores two attention tensors—keys (K) and values (V)—for every retained token across its attention layers. These tensors are kept during inference so subsequent tokens can reuse earlier attention computations instead of recalculating them. Why does it grow so large? Two tensors per token: Both K and V must be retained. Many Transformer layers: Each layer maintains its own cache. Long prompts and answers: In standard full attention, every additional retained token adds cache entries. Concurrent users: Each active request needs cache state; unless a serving system shares identical prefixes, total memory increases with simultaneous sequences. Numerical precision: FP16/BF16 tensors use 2 bytes per element; lower precision KV formats may use less. How do we estimate KV cache memory? For a conventional full attention decoder with uniform layer dimensions: The leading 2 represents the separate key and value tensors. Use the number of KV heads , which can be smaller than the number of query heads in grouped query attention (GQA). Practical example Consider an illustrative model with: 32 attention layers 8 KV heads per layer, each of dimension 128 8,192 cached tokens in one request FP16 KV tensors (2 bytes per element) That is approximately 1 GiB of raw KV cache data for one sequence , excluding model weights, temporary tensors, allocator overhead, and any reserved but unused capacity. Eight independent requests of that length would require roughly 8 GiB for their raw KV data alone if no cache entries are shared. How can systems reduce the memory cost? GQA or MQA: Models designed with fewer KV heads store fewer key/value elements. KV cache quantization: Stores cache tensors with fewer bits, potentially adding accuracy or speed trade offs. Sliding window attention: For models supporting it, retains only a bounded recent context in applicable layers. Paged KV cache allocation: Manages cache in blocks to reduce wasted reserved memory and fragmentation; it does not remove the underlying need to store K/V data. Cache offloading: Moves some cache state to CPU memory, saving GPU capacity but possibly increasing latency. Shorter contexts and careful concurrency limits: Reduce the number of live cached token positions. Important distinction Model weights are persistent learned parameters; KV cache is temporary request specific attention state. Even after model weight quantization, KV cache may remain a major GPU memory cost during long context, high concurrency serving. See “What is KV cache in an LLM?” and “How does KV caching speed up LLM inference?” for the mechanism. In short: KV cache memory grows with retained tokens, KV heads, layers, precision, and concurrent sequences. Caching buys faster decoding at the cost of considerable working memory. References: Hugging Face: How caching works, Hugging Face: Cache strategies, and vLLM: KV cache configuration.
What is LLM quantization, and how does it work?
LLM quantization reduces the numerical precision used to store—and sometimes compute with—a model's weights or activations. A common example is representing many model weights using 8 bit or 4 bit values instead of 16 bit floating point numbers. The goal is to reduce memory use and, on suitable hardware, make inference more efficient . Quantization is usually approximate, so it can introduce some error. How does quantization work? A simplified quantizer maps floating point values to a smaller set of numbers using a scale and sometimes a zero point : For example, with scale = 0.1 and zero point = 0, a weight of 0.73 becomes integer 7 ; reconstructing it gives approximately 0.7 . The 0.03 difference is quantization error. Real LLM quantizers often use per group scales, special 4 bit formats, or calibration data to reduce this error. Why does it matter for LLM memory? Imagine 7 billion parameters , stored without other overhead: FP16 (16 bit): about 14 GB of raw weights. INT8 (8 bit): about 7 GB of raw weights. 4 bit: about 3.5 GB of raw weights. These are theoretical weight storage estimates, not total GPU memory requirements. Scales, metadata, unquantized layers, activations, and KV cache also consume memory. Common approaches Weight only quantization: Compresses model weights while activations or computation may remain in higher precision. Weight and activation quantization: Uses lower precision for more of the inference computation when the hardware and kernels support it. Post training quantization (PTQ): Converts a pretrained model without full retraining; methods such as GPTQ or AWQ may use calibration data. Quantization aware training (QAT): Accounts for quantization effects during training or fine tuning. Mixed precision: Keeps particularly sensitive operations at higher precision to preserve quality. Practical Python example Hugging Face Transformers supports 4 bit weight loading with bitsandbytes: This is an illustrative example; support depends on installed libraries and hardware. It loads supported model layers with quantized weights without changing the original pretrained model. Does quantization always make inference faster? No. Lower precision generally reduces weight storage and memory transfer, but speed depends on supported kernels, accelerator hardware, dequantization overhead, batch size, and workload. Aggressive quantization can also reduce accuracy, especially on sensitive tasks. Best practice: Compare model quality, GPU memory, latency , and throughput on representative prompts before deployment. See “Why does KV cache consume significant memory?” for a separate inference memory cost: quantizing model weights does not automatically shrink the KV cache. In short: quantization trades numerical precision for smaller model storage and potentially faster serving. The right bit width and method depend on acceptable quality loss and actual hardware performance. References: Hugging Face: bitsandbytes, GPTQ paper, and AWQ paper.
What is 8-bit LLM quantization?
8 bit LLM quantization stores supported model weights using 8 bit values—typically INT8—instead of 16 bit floating point values. This can roughly halve raw weight memory compared with FP16 or BF16, helping large language models fit onto smaller GPUs. Unlike general quantization, this question focuses on how 8 bit inference works and why implementations sometimes keep sensitive calculations at higher precision. How does INT8 quantization work? A floating point weight is mapped to a nearby 8 bit integer using a scale factor: Because 0.70 differs from 0.73, quantization introduces rounding error . Practical implementations use scales for groups or vectors of weights to reduce that error. What is special about LLM.int8()? The LLM.int8() method uses mixed precision matrix multiplication: Most values are processed using 8 bit operations . Sensitive outlier features are handled separately in 16 bit precision . This helps preserve output quality, rather than forcing every computation into INT8. Other 8 bit quantization methods may use different calibration, scaling, or computational approaches. 8 bit weight storage does not imply that every activation or operation is INT8. How much memory can it save? For an illustrative 7 billion parameter model: These are raw weight storage estimates . Real GPU usage is higher because some layers remain unquantized and the runtime also needs scales, activations, workspace memory, and KV cache . Practical example using Transformers This loads supported layers with bitsandbytes 8 bit quantization when the installed software and hardware support it. It is an inference example , not an instruction to retrain the model. Does INT8 always improve speed? No. Smaller weights reduce memory traffic, but measured latency depends on GPU support, optimized kernels, batch size, the extra work for outlier handling, and the decoding workload. Benchmark quality, latency, throughput, and total memory before choosing a production configuration. 8 bit vs. 4 bit: 4 bit weights can use roughly half the raw storage of 8 bit weights, but may introduce greater quantization error and different kernel overhead. Use the least aggressive precision that meets the application's hardware and quality requirements. See “What is LLM quantization, and how does it work?” for the overview, and “Why does KV cache consume significant memory?” for a memory cost that weight only quantization does not automatically shrink. In short: 8 bit quantization cuts model weight storage by approximately 50% relative to FP16 while aiming to preserve output quality; its real inference benefit depends on the implementation and hardware. References: Hugging Face bitsandbytes documentation, LLM.int8() paper (arXiv:2208.07339), and MachinoAI LLM.int8() research.
What is 4-bit LLM quantization?
4 bit LLM quantization represents supported model weights using only 4 bits per stored value instead of 16 bit floating point precision. This compresses the model's weights and can make a large language model fit on hardware with less memory. It is a form of lossy compression : the quantized representation approximates the original weights rather than preserving every number exactly. How does it work? 1. Group the weights. Quantizers often process weights in blocks so each group can use an appropriate scale. 2. Map values to 4 bit codes. Four bits represent 16 possible codes . Each code stands for an approximate weight value. 3. Store scaling metadata. Scales and any other metadata are kept so the quantized values can be interpreted during computation. 4. Run inference. Specialized kernels either operate on the compact representation or dequantize values as needed, often computing in a higher precision. How much memory can it save? For a model with 7 billion parameters , the theoretical raw weight memory is: So 4 bit weights use about 75% less raw storage than FP16 , or half as much as INT8 . Important: Actual GPU memory is higher because scales, some higher precision layers, activations, temporary buffers, and the KV cache still consume memory. What are FP4 and NF4? FP4: A low bit floating point style format supported by some quantization libraries. NF4 (NormalFloat 4): A 4 bit format introduced with QLoRA , designed for normally distributed model weights. Nested/double quantization: Further compresses some quantization metadata , such as scaling constants, to save additional memory. The choice of format affects how well the original weight distribution is approximated. 4 bit storage does not mean every computation runs in 4 bit precision. Practical example with Hugging Face This example uses bitsandbytes to load supported linear layer weights in a 4 bit format. Hardware, backend, and compatible library versions are required. When should you use 4 bit quantization? It is especially useful when model weights do not fit into available accelerator memory , or when reducing model storage and memory traffic matters. However, consider these trade offs: Accuracy: Some tasks and models are more sensitive to 4 bit rounding and reconstruction errors than others. Speed: Lower weight memory does not guarantee faster inference; dequantization and kernel efficiency can become bottlenecks. Fine tuning: Methods such as QLoRA keep a quantized base model frozen and train lightweight adapters; 4 bit inference alone is not the same as fine tuning. Benchmark output quality, latency, throughput, and peak memory using representative inputs before deployment. For prerequisites, see “What is LLM quantization, and how does it work?” and “What is 8 bit LLM quantization?” . See “What is QLoRA?” for adapter based fine tuning of quantized models. In short: 4 bit quantization reduces model weight storage by approximating weights with 16 possible codes per value, trading precision for a smaller memory footprint. References: Hugging Face bitsandbytes quantization and QLoRA: Efficient Finetuning of Quantized LLMs.
How does quantization affect LLM quality?
Quantization can affect LLM quality because reducing numerical precision changes how accurately the model's weights—and sometimes activations or KV cache values—are represented. These small numerical differences can alter token probabilities and occasionally change the model's final answer. However, quantization does not automatically make an LLM noticeably worse . Quality loss depends on the model, method, bit width, and task. Why can quality change? Rounding and reconstruction error: A weight such as 0.73 may be approximated as 0.70. Across many layers, such changes can affect intermediate calculations. Sensitive weights and outliers: Some channels or layers are more important to model behavior than others. Treating every value with the same precision can hurt accuracy. Low bit widths: Fewer representable values usually create more approximation error. A well quantized 4 bit model can remain useful, but aggressive 2 bit quantization is typically more challenging. Quantization design: Weight only quantization, weight and activation quantization, and KV cache quantization alter different parts of inference and can have different quality effects. Is 8 bit always better than 4 bit? Not necessarily on every benchmark. With the same model and a comparable method, 8 bit usually has more numerical precision and often stays close to the FP16/BF16 baseline. Carefully calibrated 4 bit methods , such as GPTQ or AWQ, can also preserve strong quality while using less memory. The method matters as much as the nominal bit width. One poorly configured 8 bit approach can perform worse on a specific task than a well tuned 4 bit approach. A simple example Consider a multiple choice question where the unquantized model narrowly prefers answer A over B . Quantization slightly changes the model's scores: The preferred answer flips. This is an illustration , not a universal prediction of quantization behavior; many prompts remain unchanged. How can quality loss be reduced? Use suitable scales and group sizes: Per channel or group wise quantization can better fit weight distributions than a single global scale. Protect sensitive values: Mixed precision schemes can retain higher precision for outliers or important layers. Use calibration data: Methods such as GPTQ and AWQ can account for representative activations when determining how to quantize weights. Avoid unnecessary precision reduction: Choose a bit width that meets actual memory and quality requirements. How should you evaluate a quantized LLM? 1. Keep the baseline fixed: Compare the same model before and after quantization, using the same prompts and decoding settings. 2. Check task quality: Evaluate factual Q&A, reasoning, coding, structured extraction, or domain specific tasks relevant to your application. 3. Measure next token behavior: Perplexity can help identify language modeling changes, but it cannot substitute for application level evaluation. 4. Inspect failures: Review incorrect answers, formatting errors, and difficult or long context inputs instead of relying only on one average score. 5. Measure the benefit: Record peak GPU memory, latency, and throughput alongside quality metrics. A model with nearly unchanged benchmark accuracy may still fail on a small but important subset of production examples. A quality versus memory decision is workload specific. See “What is LLM quantization, and how does it work?” , “What is 8 bit LLM quantization?” , and “What is 4 bit LLM quantization?” for the underlying techniques. In short: quantization introduces numerical approximation that may reduce LLM quality, especially at very low precision. Good quantization methods can retain much of the original performance, but production suitability must be established by evaluation. References: Hugging Face: Selecting a quantization method, GPTQ, and AWQ.
What is model distillation for LLMs?
Model distillation (knowledge distillation) for LLMs is a training technique in which a student model learns to imitate useful behavior from a more capable teacher model . The student is often smaller, making it potentially cheaper and faster to deploy. The goal is not to copy the teacher's parameters. It is to transfer some of the teacher's learned behavior into a separately trained model. Simple example Imagine a company uses a large LLM to classify customer support tickets into billing , technical , or account issues. The teacher LLM generates high quality classifications and explanations for representative tickets. A smaller student is trained on those examples. After evaluation, the student may handle this narrower task without calling the expensive teacher for every request. This can reduce serving cost and latency if the student meets the required quality standard . What does the student learn from? Different distillation methods use different teacher signals: Generated answers or labels: The teacher produces example responses that become supervised training targets. Soft token probabilities (logits): When available, the student learns from the teacher's distribution over possible next tokens, not only the chosen token. Explanations or rationales: Carefully generated intermediate explanations can provide additional training signals for some tasks. The teacher and student need not share the same architecture. However, token probability distillation can require additional alignment work when their vocabularies or tokenizers differ. Why distill an LLM? Smaller deployed model: Lower parameter count can reduce weight memory requirements. Potentially faster responses: A smaller student may improve latency or throughput on suitable hardware. Lower serving cost: Repeated requests may become cheaper once the student is trained. Task specialization: The student can learn strong behavior for a narrower, well defined workload. Distillation vs. quantization vs. fine tuning Distillation: Trains a student model using signals from a teacher. Quantization: Stores or computes using lower numerical precision; it does not inherently train a new student. Ordinary supervised fine tuning: Updates a model using labeled examples. Distillation may use fine tuning internally, but the key difference is that the teacher supplies the learning signal . These approaches can be combined; for example, a distilled student may later be quantized. Does the student become as good as the teacher? Not automatically. A student can inherit teacher mistakes, lose abilities outside its training distribution, and struggle on tasks that exceed its capacity. Preparing teacher generated training data also takes compute, time, and appropriate usage rights. Evaluate the student on held out, representative tasks and measure quality, latency, memory, and cost before deployment. For the training procedure in more detail, see “How does LLM distillation work?” In short: LLM distillation trains a usually smaller student to reproduce selected behaviors of a stronger teacher, trading training effort for a potentially more efficient model. References: Hugging Face: Distillation Trainer, Distilling Step by Step!, and Distilling the Knowledge in a Neural Network.