What is Parameter-Efficient Fine-Tuning (PEFT)?

Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques that adapt a pretrained LLM by training only a small fraction of its parameters instead of updating the entire model.

The goal is to reduce memory, compute, and storage requirements while still achieving useful task-specific adaptation.

Why is PEFT needed?

Full fine-tuning can be expensive because large LLMs may contain billions of parameters.

Updating all of them requires significant:

• GPU memory
• Training compute
• Optimizer state storage
• Checkpoint storage

PEFT reduces this burden by keeping most of the pretrained model frozen.

How does PEFT work?

A simplified idea is:

pretrained LLM
    ↓
freeze most parameters
    ↓
train only a small set of parameters
    ↓
task-adapted model

The exact trainable parameters depend on the PEFT method.

What parts of the model can PEFT train?

Different PEFT methods use different strategies.

Examples include:

• Adding small trainable adapter modules
• Training low-rank matrices
• Learning prompt-like vectors
• Updating only selected parameter subsets

The base model remains mostly unchanged.

What is LoRA?

LoRA (Low-Rank Adaptation) is one of the most widely used PEFT techniques.

Instead of modifying a large weight matrix directly, LoRA learns small low-rank updates that influence the model's behavior.

Conceptually:

frozen base weights
      +
small trainable LoRA weights
      ↓
adapted model behavior

LoRA is covered separately in “What is LoRA in LLM fine-tuning?”

Why does PEFT use less memory?

In full fine-tuning, training must store gradients and optimizer states for a large number of model parameters.

With PEFT, gradients and optimizer states are needed only for the smaller trainable portion.

This can dramatically reduce training-memory requirements.

Does PEFT reduce checkpoint size?

Yes.

Instead of saving a complete copy of the full model for every task, you can often save only the small PEFT parameters.

For example:

one base model
   ├─ adapter for task A
   ├─ adapter for task B
   └─ adapter for task C

This makes it easier to maintain multiple specialized versions of the same base model.

Is PEFT always as good as full fine-tuning?

Not always.

PEFT can perform very well for many tasks, but full fine-tuning may still be useful when:

• The target task is very different from the base model's behavior
• Large changes to the model are required
• Maximum task-specific performance is more important than training cost

The best approach depends on the model, dataset, and task.

What are the advantages of PEFT?

PEFT can provide:

• Lower GPU-memory usage
• Faster and cheaper training
• Smaller task-specific checkpoints
• Easier experimentation
• Multiple adaptations from one base model

Where does PEFT fit in the training lifecycle?

PEFT is not a separate stage like pretraining or post-training.

It is a way of performing fine-tuning more efficiently.

So:

Fine-tuning
   ├─ full fine-tuning
   └─ parameter-efficient fine-tuning

In short: PEFT adapts an LLM by training only a small set of parameters while keeping most of the pretrained model frozen, making fine-tuning much more memory- and compute-efficient.