Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
How does an LLM generate a response?
An LLM generates a response one token at a time . It reads the input, predicts probabilities for the next possible token, selects one using a decoding strategy, appends it to the sequence, and repeats the process. 1. The prompt is converted into tokens The user's text is first processed by the model's tokenizer. For example: These token IDs are what the model actually processes. 2. The model processes the context The tokens pass through the Transformer layers. Using attention and the model's learned parameters, the LLM builds internal representations of the current context. The context can include: System instructions User messages Previous conversation Tool or retrieved content Tokens already generated in the response 3. The model predicts the next token At the final position, the model produces a score for every token in its vocabulary. These scores are converted into probabilities. Conceptually: The model does not directly generate the whole sentence at once. 4. A decoding strategy chooses the next token The next token can be chosen in different ways. For example: Greedy decoding selects the highest probability token. Sampling chooses among tokens according to their probabilities. Temperature controls how concentrated or varied those probabilities are. Top k and top p restrict which tokens can be sampled. These choices affect how deterministic or creative the response becomes. 5. The chosen token is added to the context Suppose the model chooses: The sequence becomes: The model then predicts the next token again. 6. The process repeats Generation is autoregressive: This continues until the response is complete. 7. What makes the model stop? Generation can stop when: An end of sequence token is produced A configured stop sequence appears The maximum output token limit is reached The application stops generation for another reason Does the model plan the entire answer first? Not in the basic autoregressive generation process. Each next token is generated based on the context available at that moment. However, the context itself can encode patterns that produce structured, multi step responses. Why can the same prompt produce different responses? If sampling is enabled, more than one token may have a meaningful chance of being selected at each step. Small differences early in generation can lead to very different later responses. This is why settings such as temperature and top p matter. How is generation made efficient? During inference, implementations commonly reuse previously computed attention information through a KV cache instead of recomputing everything from scratch for every new token. That makes token by token generation much faster. In short: an LLM generates a response by tokenizing the input, processing the current context, predicting a probability distribution over the next token, selecting one token, appending it, and repeating until a stopping condition is reached.
What is decoding in an LLM?
Decoding in an LLM is the process of choosing which token to generate next from the probability distribution produced by the model. The model predicts probabilities for many possible next tokens. Decoding decides how one of those tokens is selected. Where does decoding happen? Suppose the current text is: The model may assign probabilities such as: Decoding determines which token is actually chosen. Why is decoding needed? The model does not directly output a finished sentence. At every generation step it produces scores for tokens in its vocabulary. Those scores are converted into probabilities, and a decoding strategy chooses the next token. The process is: What are common decoding strategies? Common approaches include: Greedy decoding — choose the highest probability token. Sampling — randomly choose according to the probability distribution. Top k sampling — sample only from the k most likely tokens. Top p sampling — sample from the smallest group of tokens whose cumulative probability reaches a threshold. Beam search — keep multiple promising token sequences and compare them as generation continues. Each strategy creates different output behavior. How does decoding affect the response? Decoding influences properties such as: Determinism Creativity Diversity Repetition Fluency Response stability For example, greedy decoding tends to be more deterministic, while sampling can produce more varied responses. What role does temperature play? Temperature modifies the token probability distribution before sampling. A lower temperature makes high probability tokens more dominant. A higher temperature spreads probability across more possibilities. So temperature is not itself the model's knowledge. It changes how aggressively decoding favors likely tokens. Is decoding the same as next token prediction? No. Next token prediction produces the probability distribution. Decoding determines how a token is chosen from that distribution. Conceptually: Does decoding happen only once? No. It happens repeatedly during autoregressive generation. After each selected token is appended to the sequence, the model produces a new probability distribution and the decoding process runs again. Can decoding parameters change the answer without changing the model? Yes. The same LLM and prompt can produce different responses when settings such as: Temperature Top k Top p Repetition penalties are changed. The model parameters remain the same; only the generation policy changes. For the different approaches in more detail, see “What are decoding strategies in LLMs?” In short: decoding is the step that converts an LLM's next token probability distribution into an actual token choice. The decoding strategy strongly affects how deterministic, diverse, or creative the generated response becomes.
What are decoding strategies in LLMs?
Decoding strategies are the methods an LLM uses to choose the next token from the probability distribution it predicts during text generation. Different strategies can make the same model behave more deterministically, creatively, or conservatively. Why are decoding strategies needed? At every generation step, the model produces probabilities for many possible next tokens. For example: A decoding strategy decides how one token is selected from these possibilities. 1. Greedy decoding Greedy decoding always chooses the token with the highest probability. It is simple and deterministic, but it can sometimes produce repetitive or overly predictable text. 2. Sampling Sampling selects the next token probabilistically instead of always taking the highest probability option. A token with a higher probability is more likely to be chosen, but lower probability tokens can still appear. This increases output diversity. 3. Temperature sampling Temperature changes how sharp or spread out the probability distribution is before sampling. Lower temperature makes high probability tokens more dominant. Higher temperature gives lower probability tokens more chance. So temperature controls randomness rather than changing what the model has learned. 4. Top k sampling Top k keeps only the k most likely tokens and samples from that smaller set. For example: This prevents very unlikely tokens from being selected. 5. Top p sampling Top p, also called nucleus sampling , keeps the smallest set of tokens whose cumulative probability reaches a chosen threshold. For example, with: the model samples only from tokens that together account for roughly 90% of the probability mass. The number of allowed tokens can change at every generation step. 6. Beam search Beam search keeps multiple promising sequences instead of committing to only one token path. Conceptually: The algorithm expands and compares these candidates as generation continues. Beam search is common in some sequence generation tasks, although open ended chat generation often uses sampling based strategies instead. Can strategies be combined? Yes. Generation often combines settings such as: Temperature Top k Top p Repetition penalties Stop conditions For example, a model might use both temperature and top p sampling at the same time. Which decoding strategy is best? There is no single best strategy. The right choice depends on the task. For example: Deterministic extraction may benefit from low randomness. Creative writing may benefit from sampling. Conversational generation often needs a balance between stability and diversity. Do decoding strategies change the model itself? No. They change how the model's predicted probabilities are converted into output tokens . The model parameters remain unchanged. In short: decoding strategies control how an LLM selects each next token. Greedy decoding favors certainty, sampling adds randomness, top k and top p restrict the candidate set, temperature controls randomness, and beam search explores multiple possible sequences.
What is greedy decoding?
Greedy decoding is a text generation strategy where an LLM always chooses the highest probability token at each generation step. It is one of the simplest decoding methods. How does greedy decoding choose a token? Suppose the model predicts: Greedy decoding selects: because it has the highest probability. The selected token is appended to the sequence, and the model repeats the same process for the next token. Why is it called “greedy”? It is called greedy because the algorithm always takes the locally best option available at that moment. It does not explore alternative token choices that might lead to a better overall sequence later. Conceptually: Is greedy decoding deterministic? Usually, yes. If the model, prompt, and inference conditions are the same, greedy decoding normally produces the same sequence because there is no random sampling step. This makes it useful when stable output is important. What are the advantages of greedy decoding? Greedy decoding is: Simple Fast Deterministic Easy to reproduce Computationally inexpensive compared with strategies that track multiple candidate sequences It can work well for tasks where the model strongly prefers one clear continuation. What are the limitations? The highest probability token at one step does not guarantee the best full response. A locally optimal choice can lead to a weaker sequence later. Greedy decoding can also make outputs: Repetitive Predictable Less diverse Less creative More likely to get stuck in repetitive phrasing How is greedy decoding different from sampling? With greedy decoding: With sampling: Sampling allows lower probability tokens to occasionally be selected, which can create more varied outputs. Is greedy decoding the same as temperature = 0? They are often treated similarly in practical APIs because both are used to obtain highly deterministic generation. However, the exact behavior depends on the serving implementation. Greedy decoding specifically means selecting the highest probability token directly at each step. When is greedy decoding useful? It can be useful when you want: Reproducible output Low randomness Simple generation Short deterministic completions A baseline for comparing other decoding strategies For open ended conversation or creative writing, sampling based strategies often produce more natural variation. In short: greedy decoding generates text by selecting the highest probability next token at every step. It is simple, fast, and deterministic, but it can sacrifice diversity and may choose a locally best token that does not lead to the best overall response.
How does greedy decoding work?
Greedy decoding works by selecting the single highest probability token at every generation step. The model does not sample randomly or keep multiple candidate paths. Step 1: Predict next token probabilities Suppose the current context is: The model may predict: Greedy decoding chooses: because it has the highest probability. Step 2: Append the selected token The sequence becomes: That updated sequence is passed back to the model. Step 3: Predict again The model now calculates probabilities for the next token. For example: Greedy decoding again chooses the highest probability option. Step 4: Repeat until generation stops The full process is: Generation stops when an end token, stop sequence, or output length limit is reached. Why is greedy decoding deterministic? There is no random sampling step. If the same model and context produce the same probability distribution, greedy decoding always chooses the same highest probability token. Why can greedy decoding still produce a weak answer? The best token right now is not always part of the best complete sequence. For example, one token may have the highest immediate probability but lead to a repetitive or less coherent continuation later. Greedy decoding does not reconsider previous choices. How is it different from beam search? Greedy decoding keeps only one path: Beam search keeps several promising candidate sequences and compares them as generation continues. How is it different from sampling? Sampling allows multiple tokens to have a chance of being selected. Greedy decoding always chooses: so it favors reproducibility over diversity. For the broader concept, see “What is greedy decoding?” In short: greedy decoding repeatedly selects the highest probability next token, appends it to the context, and continues until a stopping condition is reached. It is simple and deterministic, but its locally optimal choices do not always produce the best overall response.
What is sampling in LLM text generation?
Sampling in LLM text generation is a decoding method where the next token is chosen probabilistically from the model's predicted token distribution instead of always selecting the single most likely token. This introduces controlled randomness into generation. How does sampling work? Suppose the model predicts: With sampling, sunny is the most likely choice, but the model can still select warm, cloudy, or rainy. Higher probability tokens are simply more likely to be selected. How is this different from greedy decoding? Greedy decoding always chooses: Sampling instead does: So sampling allows the same prompt to produce different outputs across runs. Why is sampling useful? If an LLM always chose the most likely token, its responses could become: Repetitive Predictable Less creative Too similar across requests Sampling gives the model room to explore other plausible continuations. This is especially useful for: Conversation Brainstorming Creative writing Story generation Generating multiple alternatives Does sampling mean choosing tokens completely at random? No. Sampling is still guided by the model's probabilities. A token with a 40% probability is much more likely to be selected than one with a 0.1% probability. So generation remains model guided rather than uniformly random. What controls how random sampling is? Several decoding parameters can shape the distribution before sampling. Common ones include: Temperature Top k Top p For example, a lower temperature makes the distribution sharper, so high probability tokens dominate more strongly. A higher temperature spreads probability across more alternatives. Why use top k or top p with sampling? The full vocabulary may contain many extremely unlikely tokens. Sampling from all of them can occasionally produce strange outputs. Top k and top p reduce the candidate set before sampling. For example: This keeps some diversity while reducing unlikely choices. What happens after a token is sampled? The selected token is appended to the sequence. The model then recomputes the next token distribution using the updated context and sampling happens again. So generation remains autoregressive: Can sampling reduce factual reliability? It can. Increasing randomness makes lower probability continuations more likely. For factual or structured tasks, aggressive sampling may increase the chance of unusual or incorrect outputs. That is why decoding settings should match the task. Is sampling always better than greedy decoding? No. Sampling is useful when diversity matters. Greedy or low randomness decoding may be preferable when you want: Consistency Reproducibility Structured extraction Predictable outputs In short: sampling chooses each next token probabilistically from the LLM's predicted distribution. It creates more diverse and varied responses than greedy decoding, while still favoring tokens the model considers more likely.
What is temperature in LLM generation?
Temperature in LLM generation is a decoding parameter that controls how strongly the model favors high probability next tokens during sampling. It changes the shape of the token probability distribution before the next token is selected. How does temperature work? Suppose the model predicts: Temperature can make this distribution either sharper or flatter. Lower temperature A lower temperature makes the most likely tokens more dominant. Conceptually: This usually makes generation: More deterministic More conservative Less varied More focused on likely continuations Higher temperature A higher temperature spreads probability more evenly across candidate tokens. This gives lower probability tokens a better chance of being selected. Conceptually: This can make responses: More diverse More creative Less predictable More likely to include unusual continuations Does temperature change the model's knowledge? No. Temperature does not retrain the model and does not change its parameters. It only changes how the model's predicted token scores are converted into probabilities during generation. So the same model can produce very different styles of output depending on the temperature setting. What happens mathematically? Before sampling, the model produces token scores called logits . Temperature rescales those logits before softmax: A lower temperature increases the relative difference between logits. A higher temperature reduces that difference. The resulting softmax probabilities are then used for sampling. Does temperature matter with greedy decoding? Not in the same way. Greedy decoding simply selects the highest probability token. Temperature is most useful when sampling is enabled because it changes the probabilities used in that random selection. What temperature should be used? There is no universal best value. It depends on the task. For example: Factual or structured tasks usually benefit from lower randomness. Brainstorming and creative writing may benefit from more variation. Conversational applications often use a middle ground. The useful range also depends on the specific model and serving system. Can high temperature reduce answer quality? Yes. If temperature is too high, lower probability tokens become easier to select. That can increase: Incoherent phrasing Topic drift Factual mistakes Unusual word choices So more randomness does not automatically mean better generation. Is temperature the only control over randomness? No. It is often combined with other decoding controls such as: Top k Top p Repetition penalties Stop conditions These settings work together to shape the final output. In short: temperature controls how strongly an LLM favors high probability tokens during sampling. Lower temperature makes outputs more predictable, while higher temperature increases diversity by giving less likely tokens a greater chance of being selected.
How does temperature affect LLM responses?
Temperature affects an LLM response by changing how strongly generation favors high probability tokens over lower probability alternatives. It does not change what the model has learned. It changes how the next token is selected during decoding. Lower temperature A lower temperature makes the token distribution sharper. That means the model is more likely to choose its highest probability continuations. Typical effects include: More predictable responses Less variation between runs More conservative wording Lower creativity Greater consistency for structured tasks For example, if the model strongly prefers one answer, a low temperature makes that preference even stronger. Higher temperature A higher temperature flattens the token probability distribution. This gives lower probability tokens a greater chance of being selected. Typical effects include: More varied responses More creative wording Greater diversity Less reproducibility Higher chance of unusual or incorrect continuations So increasing temperature adds randomness, not intelligence. Simple example Suppose the model predicts: At a lower temperature, excellent becomes even more dominant. At a higher temperature, alternatives such as good or useful become more competitive. Does low temperature guarantee a correct answer? No. A low temperature only makes the model more likely to follow its highest probability path. If the model's most likely continuation is wrong, lowering temperature can make that wrong answer more consistent. Temperature controls randomness, not factual accuracy. Does high temperature cause hallucinations? It can increase the risk. At higher temperatures, less likely tokens receive more probability, so the model may produce: Unexpected claims Topic drift Contradictions Less coherent wording But hallucination is not caused only by temperature. It can also result from limitations in the model, prompt, context, or available knowledge. What happens near temperature 0? Very low temperature makes generation approach highly deterministic token selection. In many serving systems, temperature 0 is implemented as greedy or effectively greedy decoding. The exact behavior can depend on the API or inference framework. How should temperature be chosen? It depends on the task. For example: Extraction and structured output usually benefit from low randomness. Factual Q&A often uses lower temperature. Brainstorming may use a higher temperature. Creative writing can benefit from greater variation. The useful setting also depends on the model. Does temperature work alone? Usually not. It is often combined with: Top k Top p Repetition penalties Stop sequences Together, these settings control how the model explores possible continuations. For the underlying mechanism, see “What is temperature in LLM generation?” In short: lower temperature makes LLM responses more predictable and focused on high probability tokens, while higher temperature increases diversity and randomness. Temperature changes generation behavior, not the model's underlying knowledge.
What happens when LLM temperature is set to zero?
When an LLM's temperature is set to zero , generation becomes as deterministic as the serving system allows. In practice, the model usually behaves like it is choosing the highest probability next token at each step. What changes at temperature 0? Normally, temperature reshapes the probability distribution before sampling. As temperature becomes very small: High probability tokens dominate more strongly. Lower probability alternatives become much less likely. Output variation decreases. Repeated runs tend to produce the same response. Conceptually: Is temperature 0 the same as greedy decoding? Often, yes in practical APIs. Many inference systems interpret temperature 0 as effectively using greedy decoding: However, the exact implementation depends on the model serving framework or API. Does temperature 0 guarantee exactly the same response every time? Not always. It makes decoding highly deterministic, but other sources of variation may still exist, such as: Backend implementation details Floating point behavior Model updates Different system prompts or hidden context Parallel inference behavior So temperature 0 reduces randomness in decoding, but it does not guarantee absolute reproducibility across every environment. Does temperature 0 make answers more accurate? No. It makes the model more consistent, not necessarily more correct. If the model's highest probability continuation contains an error, temperature 0 can make that same incorrect answer appear consistently. For example: When is temperature 0 useful? It is useful when you want: Stable output Reproducible behavior Structured extraction Classification like responses Consistent formatting Less creative variation For brainstorming or creative writing, some randomness is usually more useful. What happens to creativity? Creativity and diversity usually decrease because the model has less freedom to explore lower probability alternatives. Responses may become: More predictable More repetitive More conservative That can be desirable for precise tasks but limiting for open ended generation. For the broader mechanism, see “What is temperature in LLM generation?” and “How does temperature affect LLM responses?” In short: setting temperature to zero makes an LLM strongly favor its highest probability next token, usually producing near greedy and highly deterministic output. It improves consistency, but it does not guarantee correctness.
What happens when LLM temperature is high?
When an LLM's temperature is high , the model gives lower probability tokens a greater chance of being selected during sampling. This makes generation more diverse and less predictable. What changes at high temperature? Temperature reshapes the token probability distribution before sampling. At a higher temperature: Probability differences between tokens become smaller. More candidate tokens remain competitive. Lower probability tokens are selected more often. Responses vary more across repeated runs. Conceptually: Simple example Suppose the model predicts: With a low temperature, great may dominate strongly. With a higher temperature, the other choices become relatively more likely, so the model explores a wider range of continuations. How does this affect responses? Higher temperature can make responses: More creative More varied Less repetitive Less deterministic More surprising This can be useful for brainstorming, storytelling, or generating alternatives. What are the risks? If temperature becomes too high, the model may give too much probability to weak token choices. That can lead to: Less coherent sentences Topic drift Contradictions Unusual wording More factual mistakes Greater hallucination risk So increasing temperature trades some predictability for diversity. Does high temperature make the model smarter? No. Temperature does not add knowledge, reasoning ability, or new information. It only changes how the model samples from the probabilities it already produced. A lower probability answer is not necessarily more creative or more correct. Does high temperature always produce bad output? No. Moderately higher temperature can be useful when multiple valid answers exist. For example: Brainstorming product names Generating story ideas Writing different versions of a sentence Exploring alternative approaches The problem appears when randomness becomes too strong for the task. How does high temperature interact with top p or top k? Temperature is often combined with filtering methods such as top p or top k. For example: These controls can preserve diversity while preventing extremely unlikely tokens from being selected. Should factual tasks use high temperature? Usually, high randomness is less desirable when the goal is: Precise factual answers Structured output Classification Data extraction Reproducible results A lower temperature is generally more appropriate for those tasks. For the underlying concept, see “What is temperature in LLM generation?” and “How does temperature affect LLM responses?” In short: high temperature flattens an LLM's token probability distribution, allowing more lower probability tokens to be sampled. This increases creativity and diversity, but excessive temperature can reduce consistency, coherence, and factual reliability.