Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
What is top-k sampling?
Top k sampling is a decoding strategy where an LLM limits the next token choice to the k most probable tokens and then samples from that reduced set. Instead of sampling from the entire vocabulary, the model ignores all tokens outside the top k candidates. How does top k sampling work? Suppose the model predicts: If: only these remain eligible: The probabilities of those remaining tokens are renormalized, and the next token is sampled from them. Why use top k sampling? An LLM vocabulary can contain tens of thousands of tokens. Many of those tokens may have extremely small probabilities for the current context. Sampling from the full vocabulary can occasionally select an unlikely token and produce strange output. Top k reduces this risk by restricting sampling to the most likely candidates. What happens with a small k? A small value makes generation more conservative. For example: means only the single most probable token is allowed. That effectively behaves like greedy decoding. A slightly larger value allows some variation while remaining tightly controlled. What happens with a large k? A larger k allows more candidate tokens into the sampling pool. That usually increases: Diversity Creativity Variation between responses But if k is too large, lower quality or less relevant tokens may also become eligible. Is top k itself random? Top k filtering is not the random part. It first decides which tokens are allowed . Sampling then chooses among those allowed tokens according to their probabilities. Conceptually: How does top k interact with temperature? Temperature modifies the relative probabilities of the candidate tokens. Top k controls how many tokens are allowed. They can be used together: The exact order can vary by implementation, but both influence the final token selection. How is top k different from top p? Top k always keeps a fixed number of tokens . Top p keeps a variable number of tokens based on cumulative probability. For example: Top k = 50 always considers up to 50 tokens. Top p = 0.9 considers however many tokens are needed to cover roughly 90% of the probability mass. This makes top p adapt to how confident the model is at each step. When is top k useful? Top k can be useful when you want: More diversity than greedy decoding Less randomness than unrestricted sampling Protection against extremely unlikely token choices A simple way to control the candidate set In short: top k sampling keeps only the k most likely next tokens and samples from that smaller group. It provides a simple balance between deterministic generation and unrestricted randomness.
How does top-k sampling work?
Top k sampling works by keeping only the k highest probability next tokens, renormalizing their probabilities, and randomly selecting one of them. The process repeats for every generated token. Step by step example Suppose an LLM predicts the following next token probabilities: Step 1: Choose k. Set top k = 3. Step 2: Keep the top three. Tokens A, B, and C stay; D and E are excluded from selection. Step 3: Renormalize. The retained probabilities sum to 0.40 + 0.30 + 0.15 = 0.85. Divide each retained probability by 0.85: The new probabilities sum to approximately 1. Step 4: Sample. Draw one token using these probabilities. A is most likely, but B and C can also be chosen. Step 5: Repeat. Append the chosen token to the context, obtain the next token probabilities, select the new top k candidates, and sample again. What changes when k changes? k = 1: Only the highest probability token remains, effectively giving greedy decoding. Small k: Fewer choices, generally less variety. Large k: More choices, generally more variety and a greater chance of weak continuations. k covering the vocabulary: No tokens are excluded by top k filtering. Important limitation Top k uses a fixed number of candidate tokens , even when the model is extremely confident or highly uncertain. For example, k = 50 keeps up to 50 candidates at each step regardless of how their probability mass is distributed. This differs from top p sampling , which adjusts the number of candidates based on cumulative probability. Top k can also be combined with temperature to adjust probabilities before sampling. In short: top k sampling filters out all but the k most likely tokens, rescales the remaining probabilities, and samples one token at each generation step.
What is top-p sampling?
Top p sampling , also called nucleus sampling , is an LLM decoding strategy that samples the next token from the smallest set of high probability tokens whose cumulative probability is at least p . Unlike top k sampling, which keeps a fixed number of tokens, top p adapts the candidate set to the model's confidence at each generation step. Simple example Suppose the LLM predicts these next token probabilities, sorted from highest to lowest: With top p = 0.90, the model keeps A, B, C, and D . Their cumulative probability is 0.95, which is the first point where the total reaches or exceeds 0.90. Token E is excluded. The probabilities of the retained tokens are renormalized, and one is sampled. Why is top p useful? The model's next token distribution changes with the context: Confident prediction: A few tokens may already cover most of the probability mass, so top p keeps a small candidate set. Uncertain prediction: Probability is spread across many plausible tokens, so top p can keep more candidates. This adaptive filtering helps balance natural variation with protection against very unlikely token choices. What happens when p changes? Lower p: Keeps a smaller, more conservative group of candidates, generally reducing variety. Higher p: Allows more candidates and usually increases diversity. p = 1: Does not filter tokens by cumulative probability. Top p is a sampling filter , not a guarantee of factual accuracy. Raising p can allow weaker continuations, while lowering p can still produce incorrect answers. Top p vs. top k Top k: Keep exactly the k most probable tokens (when at least k are available). Top p: Keep however many highest probability tokens are needed to meet the probability threshold. Top p is commonly combined with temperature , which reshapes probabilities before sampling. The exact behavior also depends on the inference implementation. For the fixed size alternative, see “What is top k sampling?” In short: top p sampling dynamically selects a nucleus of likely next tokens whose combined probability reaches a specified threshold, then samples from that set.
How does nucleus sampling work?
Nucleus sampling (also called top p sampling ) works by dynamically choosing a set of likely next tokens whose combined probability reaches a specified threshold, then randomly sampling one token from that set. Unlike top k, the number of eligible tokens can change at every generation step. Step 1: Predict next token probabilities For the current context, imagine the LLM produces: The tokens are sorted from highest to lowest probability. Step 2: Apply the top p threshold Suppose top p = 0.80. Add the probabilities in descending order until the total first reaches or exceeds 0.80 : The nucleus is A, B, and C . Tokens D and E are excluded. Notice that the retained probability mass is 0.85 , not exactly 0.80: the threshold determines when to stop adding complete tokens. Step 3: Renormalize and sample The retained probabilities sum to 0.85. Divide each by 0.85 so that the allowed candidates sum to 1: Sample one token from this new distribution. A is most likely, but B or C can still be selected. Step 4: Repeat for the next token Append the sampled token to the generated sequence. The LLM predicts a fresh next token distribution, and the nucleus is calculated again. Why does the nucleus size change? Confident prediction: One or two tokens may already exceed the threshold, leaving a small nucleus. Uncertain prediction: Probabilities may be spread across many tokens, so more candidates are needed. This is the key advantage over top k, which uses a fixed candidate count. How does temperature fit in? Temperature can reshape the probabilities before nucleus filtering. Higher temperature typically makes the distribution flatter, which may increase the number of tokens needed to reach the same top p threshold. In short: nucleus sampling sorts tokens by probability, retains the smallest group whose cumulative probability meets the top p threshold, renormalizes that group, samples a token, and repeats.
What is the difference between top-k and top-p sampling?
Top k and top p sampling differ in how they restrict the candidate tokens before sampling. Top k keeps a fixed number of likely tokens; top p (nucleus sampling) keeps the smallest set whose cumulative probability meets a threshold. Top k: fixed candidate count With top k = 3, only the three highest probability tokens remain eligible, regardless of how confident the model is. Top p: adaptive candidate count With top p = 0.90, the model keeps adding tokens from most to least likely until their probabilities add up to at least 90%. The number of retained tokens may change at each generation step. Example Suppose the probabilities are: Top k = 3 retains A, B and C. Top p = 0.90 retains A, B, C and D: their cumulative probability is 0.95, the first total to reach 0.90. Both methods renormalize the retained probabilities before randomly selecting a token. Why does the distinction matter? When the model is very confident, one token might have probability 0.93. Top p = 0.90 would keep only that token, whereas top k = 3 still keeps three. When many tokens have similar probabilities, top p may retain more options. Use top k when you want a consistent maximum number of candidates. Use top p when you want the number of candidates to adapt to confidence. Some inference systems support combining the two filters with temperature. Neither strategy guarantees factual correctness. See “What is top k sampling?” and “What is top p sampling?” for the individual mechanisms. In short: top k controls the number of eligible tokens; top p controls their cumulative probability .
What is beam search in language models?
Beam search is a text decoding method that keeps several promising partial responses at once rather than selecting only one next token. At each generation step, the LLM predicts possible next tokens for every retained candidate. Beam search scores the extended sequences and keeps only the best few. What does beam width mean? Beam width (often written as B) is the number of candidate sequences retained. B = 1 behaves like greedy decoding. B = 3 keeps three promising paths. Larger B explores more alternatives but uses more compute and memory. Simple illustration With beam width 2: The algorithm typically ranks sequences using accumulated token log probabilities, sometimes adjusted for length. How is beam search different? Greedy decoding: Keeps only the highest scoring immediate continuation. Beam search: Keeps several candidate sequences and compares their accumulated scores. Sampling: Randomly chooses tokens according to probabilities to encourage variety. When is beam search useful? It can be useful for sequence generation tasks with relatively constrained outputs, such as machine translation. Open ended chat often uses sampling instead. What are its limitations? Beam search costs more than greedy decoding. It cannot guarantee the globally best sequence because discarded candidates are not revisited. High scoring sequences can still be repetitive, generic, or factually wrong. Increasing beam width does not automatically improve human perceived quality. For more background, see “What is greedy decoding?” and “What are decoding strategies in LLMs?” In short: beam search explores multiple high scoring continuations in parallel and keeps the strongest candidates at each step, balancing wider search against computational cost.
How does beam search work?
Beam search works by expanding several candidate sequences at each generation step and retaining only the highest scoring ones. The number of candidates kept is called the beam width . Step 1: Choose a beam width Suppose beam width = 2. The decoder keeps two candidate sequences instead of committing to a single next token. Step 2: Generate initial candidates Imagine the model assigns the first token probabilities: Both A and B remain because the beam width is two. Step 3: Expand every retained candidate The model predicts another token separately for each candidate: These numbers are illustrative. Actual implementations normally add log probabilities rather than multiply probabilities directly. Step 4: Keep the best two sequences The decoder compares all expanded candidates and retains: Notice that B was less likely than A initially, but its continuation B → Z now scores highest. This is why beam search can outperform choosing the best immediate token. Step 5: Repeat until stopping Each retained sequence is expanded again. The process continues until an end token or configured length/stopping condition is reached. Finished candidates are tracked, and the final output is selected from completed sequences using the decoding score. How are sequences scored fairly? Raw sequence probabilities shrink as text gets longer. Implementations often apply length penalties or normalization so that very short answers do not always win. Important trade offs Increasing beam width explores more candidate paths but uses more memory and computation. Beam search is not guaranteed to find the globally best sequence: discarded paths cannot be recovered. High model probability does not guarantee factual accuracy or engaging wording. For the overall concept, see “What is beam search in language models?” In short: beam search repeatedly expands candidate responses, scores the resulting sequences, and keeps only the best few until generation finishes.
What is a repetition penalty in LLM generation?
A repetition penalty is a decoding control that discourages an LLM from selecting tokens it has already used, helping reduce repetitive words, phrases, and loops during text generation. It adjusts the next token scores during inference ; it does not retrain the model or change its weights. Why is it needed? An LLM generates text one token at a time. Sometimes it repeatedly favors the same continuation: A repetition penalty makes previously seen tokens less attractive, increasing the chance of a different continuation. How does it work? In a common implementation, the decoder: 1. Identifies tokens appearing in a configured history (often the prompt and generated output, depending on the system). 2. Adjusts their logits (next token scores) to discourage reusing them. 3. Selects the next token using the chosen decoding strategy, such as greedy decoding or sampling. 4. Repeats this process at each generation step. For a common multiplicative repetition penalty convention: 1.0: No repetition penalty. Greater than 1.0: Discourages previously seen tokens more strongly. Less than 1.0: Can favor repetition, where supported. Exact parameter behavior varies between inference engines and APIs. What are the trade offs? A modest penalty may improve diversity and reduce loops. Too much penalty can damage fluency because some repetition is necessary for coherent writing. For example, an explanation may need to repeat a technical term, a person's name, or a key variable. An excessive penalty may encourage awkward substitutions instead. Is it the same as frequency or presence penalties? Not necessarily. A repetition penalty typically modifies scores for tokens that have appeared before. A presence penalty commonly applies a penalty once a token has appeared, regardless of its count. A frequency penalty commonly increases the penalty with the number of prior occurrences. These are different mechanisms, and providers do not always implement or name them identically. Does it guarantee no repetition? No. It changes token selection preferences rather than imposing an absolute ban. A model may still repeat content when that continuation remains likely or necessary. For background, see “What are decoding strategies in LLMs?” In short: a repetition penalty discourages reuse of previously seen tokens during decoding. Used moderately, it can reduce repetitive output; set too high, it can make responses unnatural or less coherent.
Why do LLMs sometimes repeat themselves?
LLMs sometimes repeat themselves because they generate text one token at a time, using their own previous output as part of the context for the next prediction. A repeated phrase can make that same phrase increasingly likely to appear again, creating a feedback loop. What causes repetition? Self reinforcing predictions: Once a phrase repeats, the growing context may make another repetition seem like a probable continuation. Decoding strategy: Greedy decoding and beam search favor high probability continuations. For open ended writing, these can get stuck in repetitive loops rather than explore alternatives. Repetitive context: A prompt containing the same wording or examples many times may encourage the model to continue that pattern. Long generation or weak stopping cues: Without a clear endpoint, a model may circle back to earlier ideas instead of finishing. Simple example After the phrase repeats once, the repeated wording becomes part of the context, potentially making another copy more likely. How can repetition be reduced? Use suitable decoding settings: Top p sampling can introduce alternative continuations instead of always choosing the most likely token. Temperature also affects variation. Apply a modest repetition penalty: This lowers the preference for tokens already seen. See “What is a repetition penalty in LLM generation?” Use repetition constraints when appropriate: Some inference systems support no repeat ngram size, which blocks repeated token sequences of a specified length. Give focused instructions and stopping conditions: Request a specific length or structure and avoid unnecessarily long output limits. Can repetition ever be useful? Yes. Repeating a technical term, variable name, or key fact can improve clarity. Excessive penalties may make language unnatural or interfere with legitimate repetition. Research on neural text degeneration has shown that decoding choices can strongly affect repetitive generation, even when the underlying language model stays unchanged. In short: repetition is often a feedback effect of autoregressive generation, amplified by certain decoding settings or repetitive context. Better decoding and clear stopping instructions can reduce it, though they cannot guarantee its elimination.
What is a stop sequence in LLM generation?
A stop sequence is a predefined text pattern that tells an LLM serving system to stop generating when that pattern appears in the newly generated output. It acts as a stopping rule during inference —not as a change to the model's learned weights. Simple example Imagine an application asks the model to answer a question and uses this stop sequence: The model starts generating: When the system detects User:, it stops generation. Many APIs omit the matched stop sequence from the response returned to the application. How does it work? 1. The application configures one or more stop sequences. 2. The LLM generates tokens as usual. 3. The serving system checks the generated text for a matching sequence. 4. When a match occurs, generation stops instead of continuing. A stop sequence can contain multiple characters and may span multiple tokens; it does not have to match a single tokenizer token. Why use stop sequences? Control response boundaries: Prevent a model from continuing into another dialogue turn or section. Separate structured sections: Stop before a known delimiter in generated text. Avoid unnecessary output: End generation once a particular marker appears. What if the model never generates the sequence? Then that stop rule does not trigger. Generation may instead end because of a model end of sequence token, an output token limit, or another application stop condition. Stop sequence vs. end of sequence token Stop sequence: Usually a custom text pattern configured by the application or inference API. End of sequence (EOS) token: A special token used by the model/tokenizer to signal a sequence ending. Maximum output tokens: A length limit that stops generation even when no delimiter or EOS appears. Common mistake Avoid choosing a delimiter that appears frequently in ordinary answers. For example, using a period (.) as a stop sequence could cut off a response unexpectedly. Prefer a distinctive marker. Exact matching and whether the marker is included in the returned text depend on the API or inference engine. In short: a stop sequence is an application defined output delimiter that terminates LLM generation when detected, helping control where a response ends.