What data is needed to fine-tune an LLM?

The data needed to fine-tune an LLM depends on what behavior or capability you want to improve.

Fine-tuning usually uses a much smaller and more targeted dataset than pretraining.

What does fine-tuning data look like?

For many instruction-following tasks, the dataset contains pairs such as:

Input / instruction
        ↓
Desired response

For example:

Instruction:
Summarize this customer complaint in one sentence.

Target response:
The customer reports that the payment was charged twice.

The model learns by comparing its generated tokens with the desired output.

What kinds of data can be used?

Depending on the use case, fine-tuning data may include:

• Instruction-response pairs
• Question-answer examples
• Classification examples
• Domain-specific documents converted into training examples
• Code and expected outputs
• Conversations
• Structured input-output pairs
• Examples of preferred tone or formatting

The format should closely reflect the behavior expected from the final model.

Does the data need labels?

For supervised fine-tuning, yes.

Each training example needs a desired target output.

Those targets may be:

• Written by humans
• Curated from trusted datasets
• Generated and then reviewed
• Derived from existing structured data

The important part is that the target represents behavior the model should learn.

How much data is needed?

There is no universal number.

The amount depends on:

• Model capability
• Task complexity
• Dataset quality
• Diversity of examples
• How different the desired behavior is from the base model
• Fine-tuning method

A narrowly defined task may need fewer examples than a broad behavioral change.

Is data quality more important than quantity?

Often, yes.

Thousands of inconsistent or incorrect examples can teach the model undesirable patterns.

High-quality data should ideally be:

• Accurate
• Consistent
• Representative of real inputs
• Clear about the desired output
• Free from unnecessary duplication
• Balanced across important cases

Why should the dataset include edge cases?

If the model will encounter difficult or unusual inputs in production, those cases should be represented during training and evaluation.

For example, a support model might need examples involving:

• Incomplete questions
• Ambiguous requests
• Unsupported requests
• Multiple issues in one message

Without representative examples, performance can become uneven.

Should training and evaluation data be separate?

Yes.

Some examples should be kept outside the training set so you can evaluate whether the model generalizes to unseen inputs.

Conceptually:

dataset
  ├─ training data
  ├─ validation data
  └─ test / evaluation data

Mixing evaluation examples into the training set can make results misleading.

Is raw company or domain data enough?

Not always.

Raw documents contain information, but fine-tuning usually works best when the data is transformed into examples that demonstrate the behavior you want.

If the main goal is to give the model access to frequently changing factual information, retrieval may be more appropriate than fine-tuning.

In short: fine-tuning data should contain high-quality, representative examples of the exact behavior you want the model to learn. The right dataset is usually smaller than pretraining data, but its relevance, consistency, coverage, and target quality are critical.