How do you choose the right regression metric?

Choose a regression metric based on which prediction mistakes matter most for your problem—not simply which metric gives the best-looking score.

Start with the cost of errors

• MAE: use when you want the typical absolute error in the target's units and every unit of error should count proportionally.
• RMSE: use when occasional large mistakes are especially costly and should receive more weight.
• MSE: useful as a training objective or when squared-error penalties match your goal, but its squared units are harder to interpret.
• MAPE: useful when percentage error is meaningful and actual targets are positive and comfortably away from zero.
• R²: useful as a complementary measure of improvement over a mean-prediction baseline; it does not show the size of errors.

Consider the target distribution

For example, if you forecast sales for products that frequently have zero demand, MAPE is a poor choice because division by zero makes it undefined. MAE or another domain-appropriate metric is safer.

Ask whether all mistakes cost the same

If underpredicting demand is substantially more expensive than overpredicting it, symmetric metrics such as MAE or RMSE may not capture the business objective. Consider an asymmetric or weighted loss aligned with those costs.

Evaluate on unseen data

Use the same validation or test examples when comparing models, and choose a simple baseline such as predicting the training-target mean or median.

Report a primary metric that reflects your goal, plus a secondary metric when it reveals an important trade-off. For example, MAE alongside RMSE can show whether large errors are a concern.

Practical example

For house-price prediction:

• Use MAE in rupees if stakeholders need to know the average absolute pricing mistake.
• Track RMSE too if a few very large price errors could be especially damaging.

In short, let the real-world consequences of prediction errors determine your primary regression metric, then validate it on unseen data and compare against a baseline.