What is LoRA in LLM fine-tuning?

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that adapts an LLM by training small additional matrices instead of updating the model's full weight matrices.

Most of the original model remains frozen.

Why is LoRA useful?

Full fine-tuning can be expensive because gradients and optimizer states must be maintained for billions of parameters.

LoRA reduces that cost by training only a much smaller number of parameters.

Conceptually:

frozen base model
      +
small trainable LoRA parameters
      ↓
adapted model

What does "low-rank" mean?

Large neural-network layers often contain large weight matrices.

Instead of learning a complete update to one of those matrices, LoRA represents the update using two much smaller matrices.

Conceptually:

original weight matrix: W

LoRA update:
ΔW = B × A

adapted weight:
W + ΔW

Because the intermediate dimension of A and B is small, the number of trainable parameters is much lower than updating W directly.

Are the original model weights changed?

During LoRA training, the original pretrained weights are typically frozen.

Only the LoRA parameters are updated.

This allows the model to learn task-specific changes while preserving the original base model.

Where is LoRA usually added?

LoRA modules are commonly attached to selected linear layers inside Transformer blocks.

Depending on the training setup, these may include projections used in attention or other feed-forward layers.

You do not need to add LoRA to every parameter in the model.

Why does LoRA use less memory?

Training memory is reduced because gradients and optimizer states are required only for the small LoRA parameters.

This can make fine-tuning practical on hardware that would not be able to perform full fine-tuning.

What is the rank parameter?

The rank controls the size of the low-dimensional matrices used for the update.

A smaller rank means:

• Fewer trainable parameters
• Lower memory usage
• Less adaptation capacity

A larger rank provides more capacity but increases training cost.

So rank is an important LoRA hyperparameter.

Can LoRA adapters be stored separately?

Yes.

Instead of saving a complete copy of the fine-tuned LLM, you can often save only the LoRA adapter weights.

For example:

one base model
   ├─ finance LoRA
   ├─ coding LoRA
   └─ support LoRA

This makes it easier to maintain several specialized versions of the same base model.

What happens during inference?

The LoRA update can be applied alongside the frozen base weights.

In some deployments, the LoRA weights can also be merged into the base weights so the model can run without separate adapter operations.

Is LoRA always as good as full fine-tuning?

Not always.

LoRA often works very well for many adaptation tasks, but full fine-tuning gives the optimizer more freedom because all model parameters can change.

The best choice depends on the task, dataset, available compute, and required performance.

How does LoRA relate to PEFT?

LoRA is one specific method inside the broader category of Parameter-Efficient Fine-Tuning (PEFT).

So:

PEFT
  ├─ LoRA
  ├─ adapters
  └─ other efficient fine-tuning methods

For the broader concept, see “What is Parameter-Efficient Fine-Tuning (PEFT)?”

In short: LoRA fine-tunes an LLM by freezing the original model and learning small low-rank weight updates. This greatly reduces trainable parameter count, memory use, and checkpoint size while still allowing strong task-specific adaptation.