How does supervised fine-tuning work for LLMs?

Supervised fine-tuning (SFT) works by training a pretrained LLM on examples that contain an input and the response the model is expected to produce.

The model compares its prediction with that target response and updates its trainable parameters to reduce the error.

Step 1: Prepare instruction-response examples

A training example might look like:

Instruction:
Explain gravity in simple words.

Target response:
Gravity is the force that pulls objects toward each other.

The dataset should represent the type of behavior expected from the final model.

Step 2: Convert the text into tokens

The model's tokenizer converts the instruction and response into token IDs.

These token sequences are then passed into the model for training.

Step 3: Run a forward pass

The model processes the input and predicts the next token at each relevant position.

For example:

Instruction + previous response tokens
                ↓
          predict next token

Step 4: Compare predictions with the target response

The predicted token probabilities are compared with the actual target tokens.

A loss function measures how far the model's predictions are from the desired response.

Step 5: Backpropagate the loss

Backpropagation calculates how the trainable parameters contributed to the error.

An optimizer then updates those parameters slightly.

This process repeats across many examples.

Which tokens contribute to the loss?

In many instruction-tuning setups, the model receives both the instruction and the target response as input, but the training loss is mainly applied to the response tokens.

Conceptually:

Instruction:
Explain gravity.

Response:
Gravity pulls objects together.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tokens the model is trained to generate

The exact masking strategy can vary between training pipelines.

What changes during SFT?

The model already knows broad language patterns from pretraining.

SFT mainly teaches it how to use those capabilities in the desired way, such as:

• Following instructions
• Answering directly
• Producing specific formats
• Maintaining a certain tone
• Performing domain-specific tasks

How long does SFT take?

Usually much less time than pretraining.

SFT uses a smaller dataset and starts from an already capable model.

Training may update:

• All model parameters, or
• A smaller set of parameters using techniques such as LoRA

How do you know whether SFT worked?

After training, the model should be evaluated on examples it did not see during fine-tuning.

You want to check both:

• Improvement on the target behavior
• Whether important general capabilities were preserved

This helps detect overfitting or regressions.

For the broader concept, see “What is supervised fine-tuning (SFT) in LLMs?”

In short: SFT works by showing a pretrained LLM high-quality input-output examples, measuring how its generated response differs from the target, and updating its parameters until its behavior better matches those examples.