Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
How does RLHF work in LLMs?
RLHF works by using human preference feedback to train an LLM to favor responses that people judge as better. A common RLHF pipeline has three main stages: supervised fine tuning, reward model training, and reinforcement learning optimization. 1. Start with a pretrained or supervised fine tuned model The process usually begins with a model that has already been pretrained and often instruction tuned. This gives the model a reasonable starting point for generating useful responses. 2. Generate multiple responses For the same prompt, the model produces several candidate answers. For example: These responses are then compared. 3. Collect human preferences Human reviewers rank or choose which response is better according to criteria such as: Helpfulness Correctness Relevance Safety Clarity Instruction following A preference example might look like: These comparisons become preference data. 4. Train a reward model A separate reward model is trained on those comparisons. Its job is to predict which response humans are more likely to prefer. Conceptually: Higher scores should correspond to responses that better match the learned human preferences. 5. Optimize the LLM using reinforcement learning The language model generates responses and receives reward scores from the reward model. The training algorithm then updates the LLM so responses with higher reward become more likely. A simplified loop is: This process repeats across many prompts. Why not simply maximize reward without limits? If optimization pushes too aggressively toward the reward model, the LLM may exploit weaknesses in that reward signal. This can cause behavior sometimes described as reward hacking or over optimization. RLHF pipelines therefore usually include constraints that keep the updated model reasonably close to the starting model. What changes after RLHF? The model should become more likely to produce responses that match the preferences represented in the training data. Depending on the preference criteria, this can improve: Helpfulness Instruction following Safety Tone Relevance Conversational quality Does every RLHF system use exactly the same algorithm? No. Different systems can use different reinforcement learning methods, reward models, data collection strategies, and constraints. The central idea remains: How is this different from DPO? Traditional RLHF commonly trains a separate reward model and then performs reinforcement learning optimization. DPO uses preference pairs to optimize the language model more directly, without requiring the same separate reward model plus RL pipeline. That difference is covered separately in “How does DPO differ from RLHF?” In short: RLHF works by collecting human preferences between model responses, training a reward model to represent those preferences, and then using reinforcement learning to make the LLM more likely to produce highly rewarded responses.
What is a reward model in RLHF?
A reward model in RLHF is a model trained to predict how much humans would prefer a particular LLM response. Instead of asking people to score every response during reinforcement learning, the reward model provides an automated reward signal. Why is a reward model needed? Human feedback is expensive and slow to collect continuously. RLHF therefore uses human comparisons to train a model that can approximate those preferences. Conceptually: What data trains the reward model? The training data usually contains: A prompt Two or more candidate responses A preference showing which response is better For example: The reward model learns from many comparisons like this. What does the reward model output? Given a prompt and a candidate response, the reward model usually produces a numerical score. Conceptually: A higher score means the response is predicted to better match the preferences represented in the training data. What does the reward score represent? The score is not an absolute measure of truth or quality. It represents what the reward model has learned from the preference examples. Depending on the annotation criteria, the score may reflect qualities such as: Helpfulness Relevance Clarity Safety Instruction following Overall response quality How is the reward model used in RLHF? After the reward model is trained: 1. The LLM receives a prompt. 2. It generates a response. 3. The reward model scores that response. 4. A reinforcement learning algorithm uses the reward signal. 5. The LLM is updated to make higher reward responses more likely. This allows the LLM to learn from a much larger number of training interactions than could be manually reviewed one by one. Is the reward model the same as the LLM? Usually, no. The reward model is typically a separate model used during the RLHF training process. Its role is to evaluate candidate responses rather than generate the final answer shown to users. Can the reward model be wrong? Yes. A reward model is only an approximation of human preferences. It can make mistakes because of: Limited preference data Inconsistent human judgments Biased annotation criteria Unseen types of responses Model limitations If the LLM learns to exploit weaknesses in the reward model, it may receive high reward without actually producing better responses. This is one reason RLHF requires careful evaluation. What is reward hacking? Reward hacking happens when the model finds ways to increase the reward score without genuinely satisfying the intended objective. For example, if the reward model incorrectly associates certain response patterns with high quality, the LLM may overproduce those patterns. So maximizing the reward score alone is not enough. How is this different from DPO? Traditional RLHF commonly trains a separate reward model and then uses reinforcement learning. DPO instead learns directly from preferred and rejected response pairs without requiring the same separate reward model plus RL stage. In short: a reward model in RLHF learns from human response preferences and assigns scores to new LLM responses. Those scores become the reward signal used to train the language model toward behavior that better matches human preferences.
What is preference data in LLM training?
Preference data in LLM training is data that shows which responses are preferred over other possible responses for the same prompt. Instead of only showing the model a single correct answer, preference data teaches it which answer is considered better . What does preference data look like? A simple example is: The training signal is not just the responses themselves, but the fact that Response A is preferred over Response B . Where does preference data come from? Preference labels can come from: Human reviewers Domain experts Annotators following guidelines Automated evaluators Stronger language models acting as judges Human generated comparisons are especially important in alignment methods such as RLHF. What qualities can preferences represent? Reviewers may prefer one response because it is: More correct More helpful More relevant Safer Clearer Better structured More concise Better at following instructions The exact criteria depend on the intended behavior of the model. How is preference data created? A common process is: This produces training records such as: Large preference datasets contain many such comparisons. How is preference data used in RLHF? In traditional RLHF, preference data is often used to train a reward model . The reward model learns which responses people are likely to prefer. That reward signal is then used during reinforcement learning to adjust the LLM. How is preference data used in DPO? Direct Preference Optimization (DPO) can use preferred and rejected response pairs more directly. Instead of first training a separate reward model and then running reinforcement learning, DPO trains the language model to increase the relative probability of preferred responses. How is preference data different from SFT data? Supervised fine tuning data usually looks like: Preference data looks more like: So SFT teaches: “Generate something like this.” Preference training teaches: “Prefer this response over that one.” Is preference data always reliable? No. Preference data can contain: Annotator disagreement Inconsistent standards Cultural or stylistic bias Incorrect judgments Preference for surface level qualities over correctness That is why good annotation guidelines, quality checks, and diverse evaluation are important. Can preference data change model behavior significantly? Yes. Because preference optimization directly affects which outputs the model favors, the dataset can strongly influence: Tone Verbosity Helpfulness Safety behavior Refusal behavior Style Instruction following Poor preference data can therefore teach undesirable behavior just as easily as good preference data can improve it. In short: preference data contains comparisons between possible LLM responses and indicates which response is better. It is a key training signal for alignment methods such as RLHF and DPO because it teaches the model not only what it can say, but what it should prefer to say.
What is Direct Preference Optimization (DPO)?
Direct Preference Optimization (DPO) is a post training method that teaches an LLM to prefer better responses using pairs of preferred and rejected answers. Unlike traditional RLHF, DPO does not require a separate reward model followed by reinforcement learning. What does DPO train on? DPO typically uses preference pairs such as: The training signal tells the model: How does DPO work? A simplified flow is: The model is trained so the preferred answer becomes relatively more likely than the rejected one. Why is DPO called “direct”? Because it learns directly from preference pairs. Traditional RLHF often uses: DPO simplifies this to: Does DPO still need a reference model? In the standard formulation, DPO usually compares the tuned model against a reference model. The reference model helps prevent the model from drifting too far away from its original behavior while it learns the preference signal. Conceptually: What kinds of behavior can DPO improve? Depending on the preference dataset, DPO can help improve: Helpfulness Instruction following Safety behavior Response style Relevance Conciseness Domain specific preferences The model learns whatever preference signal is represented in the data. How is DPO different from supervised fine tuning? Supervised fine tuning teaches the model from desired responses. DPO teaches the model from comparisons between responses. A useful distinction is: The two are often used together. Why is DPO attractive? DPO can make preference optimization simpler because it avoids maintaining a separate learned reward model and a reinforcement learning loop. This can make training easier to implement and more stable in some setups. Does DPO eliminate all RLHF problems? No. DPO still depends heavily on the quality of the preference data. Poor or biased preference pairs can teach undesirable behavior. It can also overfit to narrow preferences or reduce performance in areas not represented in the dataset. So evaluation is still essential. Where does DPO fit in post training? A common pipeline might look like: For the direct comparison, see “How does DPO differ from RLHF?” In short: DPO directly trains an LLM to assign higher probability to preferred responses than rejected ones, using preference pairs without requiring the traditional separate reward model and reinforcement learning pipeline used in RLHF.
How does DPO differ from RLHF?
DPO and RLHF both use preference data to align an LLM, but they optimize the model in different ways. The biggest difference is that traditional RLHF usually trains a separate reward model and then uses reinforcement learning, while DPO learns directly from preferred and rejected response pairs. How does RLHF work? A common RLHF pipeline looks like: The reward model acts as an intermediate evaluator. How does DPO work? DPO simplifies the pipeline: There is no separate reward model training stage followed by a reinforcement learning loop. What training data do they use? Both can use preference data such as: The difference is how that preference signal is turned into model updates. Does RLHF need a reward model? Traditional RLHF usually does. The reward model learns to assign higher scores to responses that match human preferences. The LLM is then optimized to increase those scores. Does DPO need a reward model? Not in the same explicit way. DPO directly adjusts the language model so the preferred response becomes more likely than the rejected response. This removes one major training component from the pipeline. Why can DPO be simpler? RLHF may require: Training a reward model Running reinforcement learning Generating responses during optimization Managing reward stability Preventing excessive policy drift DPO turns preference learning into a more direct optimization problem, which can make training easier to implement and stabilize. Does DPO still use a reference model? In the standard DPO formulation, yes. A reference model helps control how far the tuned model moves away from its original behavior. So DPO is simpler than traditional RLHF, but it still includes mechanisms to prevent uncontrolled changes. Which one is better? Neither method is automatically better in every situation. RLHF can offer flexible reward based optimization and may be useful when a learned reward signal needs to guide broader exploration. DPO can be attractive when you already have high quality preference pairs and want a simpler training pipeline. The right choice depends on: Dataset quality Training infrastructure Desired behavior Model size Evaluation results Do they have similar risks? Yes. Both can inherit problems from poor preference data, including: Bias Inconsistent judgments Overfitting to evaluator preferences Excessive agreeableness Reduced performance on underrepresented tasks RLHF also introduces reward model specific risks such as reward hacking. A simple comparison For the individual methods, see “What is RLHF in Large Language Models?” and “What is Direct Preference Optimization (DPO)?” In short: RLHF usually converts human preferences into a learned reward signal and then uses reinforcement learning, while DPO trains directly from preferred and rejected responses without the same separate reward model and RL pipeline.
What is Parameter-Efficient Fine-Tuning (PEFT)?
Parameter Efficient Fine Tuning (PEFT) is a family of techniques that adapt a pretrained LLM by training only a small fraction of its parameters instead of updating the entire model. The goal is to reduce memory, compute, and storage requirements while still achieving useful task specific adaptation. Why is PEFT needed? Full fine tuning can be expensive because large LLMs may contain billions of parameters. Updating all of them requires significant: GPU memory Training compute Optimizer state storage Checkpoint storage PEFT reduces this burden by keeping most of the pretrained model frozen. How does PEFT work? A simplified idea is: The exact trainable parameters depend on the PEFT method. What parts of the model can PEFT train? Different PEFT methods use different strategies. Examples include: Adding small trainable adapter modules Training low rank matrices Learning prompt like vectors Updating only selected parameter subsets The base model remains mostly unchanged. What is LoRA? LoRA (Low Rank Adaptation) is one of the most widely used PEFT techniques. Instead of modifying a large weight matrix directly, LoRA learns small low rank updates that influence the model's behavior. Conceptually: LoRA is covered separately in “What is LoRA in LLM fine tuning?” Why does PEFT use less memory? In full fine tuning, training must store gradients and optimizer states for a large number of model parameters. With PEFT, gradients and optimizer states are needed only for the smaller trainable portion. This can dramatically reduce training memory requirements. Does PEFT reduce checkpoint size? Yes. Instead of saving a complete copy of the full model for every task, you can often save only the small PEFT parameters. For example: This makes it easier to maintain multiple specialized versions of the same base model. Is PEFT always as good as full fine tuning? Not always. PEFT can perform very well for many tasks, but full fine tuning may still be useful when: The target task is very different from the base model's behavior Large changes to the model are required Maximum task specific performance is more important than training cost The best approach depends on the model, dataset, and task. What are the advantages of PEFT? PEFT can provide: Lower GPU memory usage Faster and cheaper training Smaller task specific checkpoints Easier experimentation Multiple adaptations from one base model Where does PEFT fit in the training lifecycle? PEFT is not a separate stage like pretraining or post training. It is a way of performing fine tuning more efficiently. So: In short: PEFT adapts an LLM by training only a small set of parameters while keeping most of the pretrained model frozen, making fine tuning much more memory and compute efficient.
What is LoRA in LLM fine-tuning?
LoRA (Low Rank Adaptation) is a parameter efficient fine tuning technique that adapts an LLM by training small additional matrices instead of updating the model's full weight matrices. Most of the original model remains frozen. Why is LoRA useful? Full fine tuning can be expensive because gradients and optimizer states must be maintained for billions of parameters. LoRA reduces that cost by training only a much smaller number of parameters. Conceptually: What does "low rank" mean? Large neural network layers often contain large weight matrices. Instead of learning a complete update to one of those matrices, LoRA represents the update using two much smaller matrices. Conceptually: Because the intermediate dimension of A and B is small, the number of trainable parameters is much lower than updating W directly. Are the original model weights changed? During LoRA training, the original pretrained weights are typically frozen. Only the LoRA parameters are updated. This allows the model to learn task specific changes while preserving the original base model. Where is LoRA usually added? LoRA modules are commonly attached to selected linear layers inside Transformer blocks. Depending on the training setup, these may include projections used in attention or other feed forward layers. You do not need to add LoRA to every parameter in the model. Why does LoRA use less memory? Training memory is reduced because gradients and optimizer states are required only for the small LoRA parameters. This can make fine tuning practical on hardware that would not be able to perform full fine tuning. What is the rank parameter? The rank controls the size of the low dimensional matrices used for the update. A smaller rank means: Fewer trainable parameters Lower memory usage Less adaptation capacity A larger rank provides more capacity but increases training cost. So rank is an important LoRA hyperparameter. Can LoRA adapters be stored separately? Yes. Instead of saving a complete copy of the fine tuned LLM, you can often save only the LoRA adapter weights. For example: This makes it easier to maintain several specialized versions of the same base model. What happens during inference? The LoRA update can be applied alongside the frozen base weights. In some deployments, the LoRA weights can also be merged into the base weights so the model can run without separate adapter operations. Is LoRA always as good as full fine tuning? Not always. LoRA often works very well for many adaptation tasks, but full fine tuning gives the optimizer more freedom because all model parameters can change. The best choice depends on the task, dataset, available compute, and required performance. How does LoRA relate to PEFT? LoRA is one specific method inside the broader category of Parameter Efficient Fine Tuning (PEFT) . So: For the broader concept, see “What is Parameter Efficient Fine Tuning (PEFT)?” In short: LoRA fine tunes an LLM by freezing the original model and learning small low rank weight updates. This greatly reduces trainable parameter count, memory use, and checkpoint size while still allowing strong task specific adaptation.
How does LoRA fine-tuning work?
LoRA fine tuning works by keeping the original LLM weights frozen and learning small low rank updates for selected weight matrices. Instead of changing a large matrix directly, LoRA represents the task specific update using two much smaller trainable matrices. Step 1: Freeze the pretrained model The original model parameters are usually not updated. So during training: This greatly reduces the number of parameters that need gradients and optimizer states. Step 2: Add low rank matrices Suppose a Transformer layer contains a large weight matrix W. LoRA keeps W unchanged and learns an additional update: where A and B are much smaller matrices. The effective weight becomes: The low rank structure is what makes LoRA parameter efficient. Step 3: Run normal training examples The fine tuning data is processed in the usual way: But during backpropagation, the frozen base weights remain unchanged. Only the LoRA parameters receive updates. Step 4: Learn task specific changes Across many training examples, the LoRA matrices learn how the model should change for the target task. For example, LoRA could adapt the same base model for: Customer support Code generation A domain specific writing style Structured output Instruction following Where are LoRA modules added? They are commonly attached to selected linear projections inside Transformer layers. Attention projections are frequent targets, although the exact layers depend on the model and fine tuning setup. You do not need to add LoRA to every layer. What does the rank control? The rank determines the size of the low dimensional update. Conceptually: The rank is therefore one of the main LoRA hyperparameters. What happens after training? The trained LoRA adapter can be saved separately from the base model. For example: This is useful because the same base model can support multiple small adapters. Can LoRA weights be merged? Yes. For deployment, the learned LoRA update can often be merged with the original weights: After merging, the model can run without loading the adapter separately. Why is LoRA more efficient than full fine tuning? Full fine tuning updates billions of model parameters. LoRA updates only a small fraction. That reduces: GPU memory usage Optimizer state memory Training compute Checkpoint size For the broader concept, see “What is LoRA in LLM fine tuning?” In short: LoRA fine tuning freezes the original LLM and trains small low rank matrices that represent task specific weight updates. This preserves the pretrained model while making adaptation much cheaper and easier to store.
What is QLoRA?
QLoRA (Quantized Low Rank Adaptation) is a parameter efficient fine tuning technique that combines 4 bit quantization with LoRA . The base LLM is stored in a quantized form to reduce memory usage, while small LoRA adapter parameters are trained to adapt the model. Why is QLoRA useful? Even LoRA still requires loading the full base model into memory. If the base model is very large, that can still require substantial GPU memory. QLoRA reduces this further by storing the frozen base model in 4 bit precision. Conceptually: What stays frozen? The quantized base model weights are usually kept frozen. Only the LoRA parameters are trained. So: This keeps the number of trainable parameters small. How does QLoRA work? A simplified training flow is: The model still performs normal forward and backward training operations, but the frozen base weights use much less memory. Does the model calculate everything in 4 bit? Not necessarily. The base weights may be stored in 4 bit form, but computations can use higher precision values when needed. The important point is that quantization reduces the memory footprint of the frozen model while LoRA handles the trainable adaptation. What is NF4? QLoRA introduced a 4 bit data type called NormalFloat 4 (NF4) . It is designed for values that roughly follow the distribution commonly seen in neural network weights. Using NF4 helps preserve useful model information while keeping storage small. What is double quantization? QLoRA can also use double quantization . This means some of the quantization related scaling information is itself quantized. That reduces memory usage further. Why is QLoRA more memory efficient than standard LoRA? Standard LoRA: QLoRA: The LoRA part is similar, but QLoRA reduces the memory needed to store the base model. What are the benefits of QLoRA? QLoRA can provide: Much lower GPU memory requirements Fine tuning of larger models on limited hardware Small trainable adapter checkpoints Lower training cost Performance that can remain close to higher precision fine tuning for many tasks Are there trade offs? Yes. QLoRA can introduce: Additional quantization and dequantization overhead Possible quality loss from aggressive quantization Hardware and software compatibility requirements More sensitivity to implementation details So it is not automatically the best choice for every task. How does QLoRA relate to LoRA? QLoRA is essentially: LoRA provides the trainable low rank adapters. Quantization makes the frozen base model much cheaper to keep in memory. For the underlying technique, see “What is LoRA in LLM fine tuning?” In short: QLoRA fine tunes an LLM by keeping the base model frozen in 4 bit quantized form while training small LoRA adapters. This allows large models to be adapted with far less GPU memory than standard full fine tuning or regular LoRA with a higher precision base model.
What is catastrophic forgetting during LLM fine-tuning?
Catastrophic forgetting during LLM fine tuning happens when adapting a model to a new task causes it to lose some capabilities or knowledge it learned earlier. The model improves on the new fine tuning data, but its performance can get worse on other tasks. Why does catastrophic forgetting happen? Fine tuning changes the model's parameters. If those updates are too strong or too narrowly focused, they can overwrite parameter patterns that were useful for broader capabilities. Conceptually: Simple example Suppose a general LLM is fine tuned heavily on legal documents. It may become better at: Legal terminology Legal style answers Contract related tasks But if the training is too narrow, it could become worse at: General conversation Coding Creative writing Other domains That loss is a form of catastrophic forgetting. What makes it more likely? Forgetting can become more severe when: The fine tuning dataset is very narrow The learning rate is too high Training continues for too many steps The new task differs greatly from pretraining behavior All model parameters are updated aggressively The fine tuning data lacks diversity Is catastrophic forgetting the same as overfitting? No. Overfitting means the model learns the fine tuning dataset too closely and performs poorly on unseen examples from the same task. Catastrophic forgetting means the model loses performance on previously learned capabilities. The two can happen at the same time, but they describe different problems. How can catastrophic forgetting be reduced? Common approaches include: Using a smaller learning rate Limiting the number of training steps Mixing some general purpose data into fine tuning Using parameter efficient methods such as LoRA Freezing more of the base model Regularly testing general capabilities during training The goal is to adapt the model without unnecessarily disturbing useful pretrained behavior. Why can PEFT help? Methods such as LoRA keep most base model weights frozen. That limits how much of the original model is changed during adaptation. This can reduce the risk of forgetting, although it does not guarantee that all previous capabilities will be preserved. How do you detect catastrophic forgetting? Evaluate the model both: On the new target task On important tasks it could already perform before fine tuning For example: That pattern suggests the adaptation improved specialization at the cost of older capabilities. Is some trade off normal? Yes. Fine tuning intentionally changes model behavior, so small changes elsewhere are expected. The concern is when those regressions are large enough to damage capabilities you wanted to preserve. In short: catastrophic forgetting occurs when LLM fine tuning improves a new task but unintentionally degrades previously learned capabilities. Careful training settings, broader data mixtures, PEFT, and regression evaluation can help reduce it.