How do you compare two regression models fairly?

To compare two regression models fairly, evaluate both on the same unseen observations, with the same target and the same metrics. Otherwise, a better score may reflect an easier dataset rather than a better model.

1. Use identical evaluation data

• Train and tune the models without using the final test set.
• Evaluate both on the same test examples, in the same target units.
• For time-based predictions, use a chronological split; for grouped records, keep related observations from leaking across splits.

2. Keep preprocessing leakage-free

Fit imputation, scaling, feature selection, and other learned preprocessing steps using training data only. Apply the fitted transformations to validation and test data.

If models need different preprocessing, give each model its own pipeline while preserving the same evaluation split.

3. Choose a primary metric before comparing

Select the metric based on the real-world impact of prediction errors:

• MAE: average absolute error in target units.
• RMSE: emphasizes occasional large mistakes.
• R²: provides an additional view of fit relative to a mean baseline.

Use the same metric definitions and calculation method for both models.

4. Compare against a simple baseline

A model should improve meaningfully over a basic prediction rule, such as always predicting the training-set mean or median. A more complicated model is not necessarily worthwhile if its gain is tiny.

5. Look beyond one overall score

Check errors for important subgroups, unusually large mistakes, and residual patterns. Also consider inference time, model size, maintenance cost, and interpretability.

Practical example

Suppose two house-price models are tested on the same 200 homes:

• Model A: MAE = ₹4.0 lakh.
• Model B: MAE = ₹3.7 lakh.

Model B has lower average absolute error on that test set. But before choosing it, check whether the improvement is consistent across validation folds, whether it makes expensive outlier mistakes, and whether its extra complexity is justified.

Avoid selecting the winner by repeatedly checking the test set

Use validation or cross-validation for model selection and hyperparameter tuning. Reserve the final test set for an unbiased comparison after choices are finalized.

In short, a fair regression comparison keeps the evaluation data and metrics consistent, prevents leakage, checks performance against a baseline, and considers whether the improvement matters in practice.