Which machine learning algorithms are sensitive to feature scaling?

Machine learning algorithms are especially sensitive to feature scaling when their calculations depend on distances, gradients, coefficient magnitudes, or feature variance.

K-Nearest Neighbors (KNN)

KNN compares observations using distance.

If one feature ranges from 0–1 and another from 0–100,000, the larger-scale feature can dominate the distance calculation.

Scaling helps each feature contribute more fairly.

K-Means clustering

K-Means assigns points to clusters using distance from cluster centers.

Features with larger numerical ranges can therefore dominate how clusters are formed.

Scaling is usually important before applying K-Means.

Support Vector Machines (SVM)

SVMs can be sensitive to differences in feature magnitude because feature scale affects distances, margins, and kernel calculations.

Scaling is especially important when using kernels such as the RBF kernel.

Linear and logistic regression

Scaling can be useful for linear and logistic regression, particularly when:

• gradient-based optimization is used
• regularization such as L1 or L2 is applied
• coefficient magnitudes need to be compared more meaningfully

Without scaling, regularization may penalize features unevenly simply because they use different units.

Principal Component Analysis (PCA)

PCA looks for directions of maximum variance.

A feature with a much larger numerical scale can have much greater variance and dominate the principal components even when it is not inherently more important.

Scaling is therefore commonly performed before PCA when features use different units.

Other distance-based methods

Algorithms that rely on Euclidean or similar distance measures are generally scale-sensitive.

Examples can include:

• nearest-neighbor methods
• many clustering approaches
• some anomaly-detection techniques

Scaling is not equally important for every algorithm

Some model families are largely unaffected by monotonic rescaling of individual features.

Those algorithms are covered separately in the next article.

Fit the scaler on training data only

Whatever scaling method is used, its parameters must be learned from the training set and then applied unchanged to validation and test data.

In short, KNN, K-Means, SVM, PCA, and many regularized or gradient-trained linear models are sensitive to feature scaling because their calculations depend directly on numerical magnitude, distance, variance, or coefficient size.