What is a test dataset?

A test dataset is a separate portion of data used to evaluate the final performance of a machine learning model on examples it has not seen during training or model tuning.

It provides an unbiased estimate of how well the finished model may perform on new data.

Why a test dataset is needed

Training and validation data influence how the model is built and tuned.

If the same data is also used for the final evaluation, the reported performance may be overly optimistic.

The test dataset is therefore kept separate until the model-development process is complete.

What is the test dataset used for?

It is commonly used to:

• measure final model performance
• estimate generalization to unseen data
• compare the finished model against required performance goals
• report final evaluation metrics

Simple example

Suppose you are building a customer-churn model.

You might:

• train the model using the training dataset
• tune model choices using the validation dataset
• evaluate the final selected model once on the test dataset

The test results then provide a more realistic indication of how the model may perform on future customers.

The test set should remain untouched

The test dataset should not be used to:

• train the model
• select features
• tune hyperparameters
• repeatedly compare model variants

If test results are repeatedly used to make development decisions, the model-development process starts adapting to the test set, and it no longer provides a clean final evaluation.

Test data vs validation data

• Validation data: helps make choices during model development.
• Test data: evaluates the final chosen model after those choices are complete.

In short, a test dataset is held-out data used at the end of model development to estimate how well the final model performs on unseen examples.