How does LoRA fine-tuning work?

LoRA fine-tuning works by keeping the original LLM weights frozen and learning small low-rank updates for selected weight matrices.

Instead of changing a large matrix directly, LoRA represents the task-specific update using two much smaller trainable matrices.

Step 1: Freeze the pretrained model

The original model parameters are usually not updated.

So during training:

base model weights → frozen
LoRA weights       → trainable

This greatly reduces the number of parameters that need gradients and optimizer states.

Step 2: Add low-rank matrices

Suppose a Transformer layer contains a large weight matrix W.

LoRA keeps W unchanged and learns an additional update:

ΔW = B × A

where A and B are much smaller matrices.

The effective weight becomes:

W + ΔW

The low-rank structure is what makes LoRA parameter-efficient.

Step 3: Run normal training examples

The fine-tuning data is processed in the usual way:

training example
      ↓
forward pass
      ↓
calculate loss
      ↓
backpropagation

But during backpropagation, the frozen base weights remain unchanged.

Only the LoRA parameters receive updates.

Step 4: Learn task-specific changes

Across many training examples, the LoRA matrices learn how the model should change for the target task.

For example, LoRA could adapt the same base model for:

• Customer support
• Code generation
• A domain-specific writing style
• Structured output
• Instruction following

Where are LoRA modules added?

They are commonly attached to selected linear projections inside Transformer layers.

Attention projections are frequent targets, although the exact layers depend on the model and fine-tuning setup.

You do not need to add LoRA to every layer.

What does the rank control?

The rank determines the size of the low-dimensional update.

Conceptually:

lower rank
→ fewer trainable parameters
→ lower memory usage
→ less adaptation capacity

higher rank
→ more trainable parameters
→ more adaptation capacity
→ higher training cost

The rank is therefore one of the main LoRA hyperparameters.

What happens after training?

The trained LoRA adapter can be saved separately from the base model.

For example:

Base LLM
  +
LoRA adapter
  ↓
specialized model behavior

This is useful because the same base model can support multiple small adapters.

Can LoRA weights be merged?

Yes.

For deployment, the learned LoRA update can often be merged with the original weights:

Wmerged = W + ΔW

After merging, the model can run without loading the adapter separately.

Why is LoRA more efficient than full fine-tuning?

Full fine-tuning updates billions of model parameters.

LoRA updates only a small fraction.

That reduces:

• GPU memory usage
• Optimizer-state memory
• Training compute
• Checkpoint size

For the broader concept, see “What is LoRA in LLM fine-tuning?”

In short: LoRA fine-tuning freezes the original LLM and trains small low-rank matrices that represent task-specific weight updates. This preserves the pretrained model while making adaptation much cheaper and easier to store.