What is a validation dataset?

A validation dataset is a separate portion of data used during model development to evaluate how well a machine learning model performs on examples it did not train on.

It helps guide decisions about the model without using the final test dataset.

Why validation data is needed

A model can perform very well on its training data but still perform poorly on new data.

The validation dataset provides an early check of how well the model is generalizing.

What is the validation dataset used for?

It is commonly used to:

• compare different models
• choose hyperparameters
• select features or preprocessing choices
• decide when to stop training
• choose classification thresholds
• detect signs of overfitting

Simple example

Suppose you train several models to predict customer churn.

You might:

• train each model on the training dataset
• evaluate each one on the validation dataset
• choose the model or settings that perform best on validation data

The validation examples are not used as the ordinary training examples from which the model directly learns its parameters.

Validation data is not the final test

Because model choices are repeatedly influenced by validation results, the model-development process can gradually become tuned to that validation set.

For that reason, a separate test dataset is usually kept untouched until the final evaluation.

Training vs validation

• Training data: used to fit the model's learned parameters.
• Validation data: used to evaluate and tune model-development choices.

In short, a validation dataset is unseen development data used to compare, tune, and monitor a model before its final evaluation on the test dataset.