Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
What is the difference between normalization and standardization?
Normalization and standardization are both feature scaling techniques, but they transform numerical data in different ways. Normalization Normalization usually rescales values into a fixed range, commonly 0 to 1. A common formula is: After normalization: the minimum observed value becomes 0 the maximum observed value becomes 1 all other values fall between them Normalization is useful when a bounded range is desirable. Standardization Standardization transforms values according to the feature's mean and standard deviation. A common formula is: After standardization, the feature typically has: a mean close to 0 a standard deviation close to 1 The values are not restricted to a fixed range. The main difference Normalization is based on the minimum and maximum values of a feature. Standardization is based on the mean and standard deviation . For example, normalization may produce values such as: while standardization may produce: Sensitivity to outliers Min max normalization can be strongly affected by extreme minimum or maximum values. Standardization can also be influenced by outliers because both the mean and standard deviation are affected by extreme values. Neither method is automatically best for every dataset. Which one should you use? The choice depends on the algorithm, data distribution, and preprocessing requirements. In general: use normalization when a bounded range is useful use standardization when centering and variance based scaling are more appropriate Whichever method is used, its parameters must be learned from the training data only and then applied unchanged to validation and test data. In short, normalization usually maps values into a fixed range, while standardization expresses values relative to the feature's mean and standard deviation.
Which machine learning algorithms are sensitive to feature scaling?
Machine learning algorithms are especially sensitive to feature scaling when their calculations depend on distances, gradients, coefficient magnitudes, or feature variance . K Nearest Neighbors (KNN) KNN compares observations using distance. If one feature ranges from 0–1 and another from 0–100,000, the larger scale feature can dominate the distance calculation. Scaling helps each feature contribute more fairly. K Means clustering K Means assigns points to clusters using distance from cluster centers. Features with larger numerical ranges can therefore dominate how clusters are formed. Scaling is usually important before applying K Means. Support Vector Machines (SVM) SVMs can be sensitive to differences in feature magnitude because feature scale affects distances, margins, and kernel calculations. Scaling is especially important when using kernels such as the RBF kernel. Linear and logistic regression Scaling can be useful for linear and logistic regression, particularly when: gradient based optimization is used regularization such as L1 or L2 is applied coefficient magnitudes need to be compared more meaningfully Without scaling, regularization may penalize features unevenly simply because they use different units. Principal Component Analysis (PCA) PCA looks for directions of maximum variance. A feature with a much larger numerical scale can have much greater variance and dominate the principal components even when it is not inherently more important. Scaling is therefore commonly performed before PCA when features use different units. Other distance based methods Algorithms that rely on Euclidean or similar distance measures are generally scale sensitive. Examples can include: nearest neighbor methods many clustering approaches some anomaly detection techniques Scaling is not equally important for every algorithm Some model families are largely unaffected by monotonic rescaling of individual features. Those algorithms are covered separately in the next article. Fit the scaler on training data only Whatever scaling method is used, its parameters must be learned from the training set and then applied unchanged to validation and test data. In short, KNN, K Means, SVM, PCA, and many regularized or gradient trained linear models are sensitive to feature scaling because their calculations depend directly on numerical magnitude, distance, variance, or coefficient size.
Which machine learning algorithms usually do not require feature scaling?
Some machine learning algorithms usually do not require feature scaling because their predictions do not depend directly on distances or the numerical magnitude of features. The most important examples are tree based models . Decision trees A decision tree learns rules such as: If a feature is rescaled, the threshold changes accordingly, but the ordering of the values remains the same. For example: could become: The tree can still make essentially the same split. Random forests Random forests are collections of decision trees, so they usually do not need feature scaling either. Features can often remain in their original units without one large scale feature automatically dominating another. Gradient boosted decision trees Tree based boosting methods also usually do not require scaling. Examples include: Gradient Boosting Trees XGBoost LightGBM CatBoost These models learn threshold based splits rather than relying on Euclidean distance between observations. Why scaling matters less for tree models Tree based models mainly care about the ordering of feature values and where useful split points occur. Multiplying a feature by a constant or converting it to another consistent scale usually preserves that ordering. Scaling can still appear in the same pipeline Even if the final model is tree based, scaling may still be needed when another preprocessing step depends on scale. For example, a pipeline that uses PCA, K Means, or a distance based transformation before the tree model may still require scaling. Do not assume every non tree model is scale independent Whether scaling matters depends on how an algorithm performs its calculations. Distance based, gradient based, margin based, and variance based methods are often more sensitive to feature scale. In short, decision trees, random forests, and tree based gradient boosting models such as XGBoost, LightGBM, and CatBoost usually do not require feature scaling because their decisions are based mainly on feature ordering and split thresholds rather than feature magnitude.
What is categorical data in machine learning?
Categorical data represents values that belong to distinct groups or categories rather than measuring a continuous numerical quantity. Examples include: color: Red, Blue, Green city: Delhi, Mumbai, Bengaluru subscription type: Free, Basic, Premium payment method: Cash, Card, UPI Categorical data is different from numerical data Numerical data usually represents measurable quantities such as: age height income temperature Categorical data instead represents labels or groups. For example: is a category, while: is numerical. Two common types of categorical data Categorical features are commonly divided into: Nominal data: Categories have no natural order. Example: Red, Blue, Green. Ordinal data: Categories have a meaningful order. Example: Low, Medium, High. These two types are covered separately in the following articles. Why categorical data needs special handling Many machine learning algorithms expect numerical input. A category such as: cannot always be given directly to the model. It may need to be converted into a numerical representation using an encoding technique. Common approaches include one hot encoding, ordinal encoding, and other categorical encoding methods. Numbers can still represent categories A column containing numbers is not automatically numerical in meaning. For example: These numbers are category identifiers, not quantities. Treating them as ordinary numerical values could incorrectly imply that 3 is greater than 1 in a meaningful mathematical sense. Understand the meaning before preprocessing Before encoding a categorical feature, determine: what each category represents whether the categories have a meaningful order how many unique categories exist whether new or unseen categories may appear in production In short, categorical data represents distinct groups or labels rather than measured quantities. Understanding whether those categories are nominal or ordinal is important before converting them into a form a machine learning model can use.
What is nominal data?
Nominal data is categorical data whose values represent different groups or labels without any natural order or ranking . Simple examples Examples of nominal data include: color: Red, Blue, Green city: Delhi, Mumbai, Bengaluru blood group: A, B, AB, O payment method: Cash, Card, UPI These categories are different from one another, but one category is not inherently greater or smaller than another. There is no meaningful order For example: does not have a natural sequence such as: Any numerical codes assigned to these categories are usually just identifiers. For example: does not mean that Green is three times larger than Red or that Blue lies between them. Nominal data vs ordinal data Nominal categories have no meaningful ranking. Ordinal categories, by contrast, have a meaningful order such as: Ordinal data is covered separately in the next article. How nominal data is used in machine learning Many machine learning algorithms require nominal categories to be converted into numerical features before training. A common approach is one hot encoding , where each category gets its own binary feature. The exact encoding method depends on the algorithm and the number of unique categories. In short, nominal data represents categories that are different in name or type but have no natural ordering, such as colors, cities, or payment methods.
What is ordinal data?
Ordinal data is categorical data whose categories have a meaningful order or ranking . Unlike nominal data, the categories are not just different labels—they can be placed in a logical sequence. Simple examples Examples of ordinal data include: satisfaction level: Dissatisfied, Neutral, Satisfied size: Small, Medium, Large priority: Low, Medium, High education level: School, Graduate, Postgraduate These categories have an order. For example: The gaps are not necessarily equal Although ordinal categories can be ranked, the distance between them is usually not precisely measurable. For example, the difference between: does not have to be mathematically equal to: This is an important difference from ordinary numerical data. Ordinal data vs nominal data Nominal data: Categories have no natural order, such as Red, Blue, Green. Ordinal data: Categories have a meaningful order, such as Beginner, Intermediate, Advanced. How ordinal data is handled in machine learning Ordinal categories can often be encoded using ordered numerical values. For example: This preserves the ranking information. However, assigning numbers also introduces an apparent numerical spacing, so the encoding should match the assumptions of the model being used. More details about categorical encoding are covered in the following articles. In short, ordinal data represents categories with a meaningful ranking, but the numerical distance between those categories is not necessarily equal or measurable.
How do you encode categorical variables?
Categorical variables are encoded by converting categories into a numerical representation that a machine learning model can use. The right encoding depends on whether the categories have an order, how many unique values exist, and which algorithm is being used . Start by understanding the category type First determine whether the feature is: Nominal: categories have no natural order, such as Red, Blue, Green Ordinal: categories have a meaningful order, such as Low, Medium, High This distinction helps determine which encoding method is appropriate. One hot encoding For nominal features with a manageable number of categories, one hot encoding is commonly used. Each category becomes its own binary feature. For example: can become separate features such as: One hot encoding is covered separately in the following articles. Ordinal encoding When categories have a meaningful order, they can be mapped to ordered numbers. For example: This preserves the ranking information. Integer or label style encoding Sometimes each category is assigned a unique integer. For example: For nominal data, these numbers are only identifiers and should not be treated as a meaningful ranking unless the model can handle them appropriately. Target based encoding For high cardinality categorical features, methods such as target encoding may replace each category with a statistic derived from the target. This can be useful, but it introduces a serious risk of data leakage if it is not implemented carefully. Target encoding is covered separately later. Some models can handle categories directly Certain algorithms and libraries can work with categorical features without manually applying standard one hot encoding. In those cases, preserving the original categorical information may be preferable. Handle unseen categories Production data may contain categories that were not present during training. The preprocessing pipeline should define how these unseen categories are handled rather than failing unexpectedly. Fit encoders using training data only Any learned encoding information should be created from the training data and then applied consistently to validation and test data. This is especially important for encodings that use frequencies or target values. In short, categorical variables are encoded by choosing a numerical representation that matches the category type, cardinality, and model. Common approaches include one hot encoding, ordinal encoding, integer encoding, and target based encoding.
What is one-hot encoding?
One hot encoding is a technique for converting categorical values into separate binary features that a machine learning model can use. It is commonly used for nominal categories , where the categories do not have a natural order. Simple example Suppose a feature contains three colors: One hot encoding creates one binary feature for each category: A row with: could be represented as: Only the feature corresponding to the observed category is set to 1; the others are set to 0. Why one hot encoding is useful If nominal categories were encoded as: some models might incorrectly interpret those numbers as having an order or numerical distance. One hot encoding avoids introducing that artificial ranking. When one hot encoding works well It is especially useful when: categories have no natural order the number of unique categories is manageable the model expects numerical input Main limitation: more features A categorical feature with many unique values can create a very large number of one hot columns. For example, a city feature with 10,000 unique cities could create thousands of additional features. This is why other encoding strategies may be more appropriate for high cardinality categorical data. Unseen categories Production or test data may contain a category that was not present during training. The encoder should be configured to handle these unseen categories safely rather than failing unexpectedly. Fit the encoder on training data only The set of known categories should be learned from the training data and then applied consistently to validation and test data. In short, one hot encoding represents each category as its own binary feature, allowing nominal categorical data to be used by machine learning models without creating a false numerical ordering between categories.
When should one-hot encoding be used?
One hot encoding should be used when a categorical feature has no natural order and the number of unique categories is reasonably small. It is especially suitable for nominal data , where categories are different labels rather than ranked values. Use it for unordered categories One hot encoding works well for features such as: color: Red, Blue, Green payment method: Cash, Card, UPI browser type: Chrome, Firefox, Safari region: North, South, East, West These categories do not have a meaningful ranking. Use it when the number of categories is manageable If a feature has only a few or a few dozen categories, one hot encoding is often simple and effective. For example, a feature with five payment methods would create five binary columns, which is usually easy to handle. Use it when the model needs numerical input Many machine learning algorithms cannot work directly with text categories. One hot encoding converts those categories into numerical binary features without introducing a false order. Avoid it for high cardinality features If a feature contains thousands of unique categories, one hot encoding can create thousands of additional columns. This can: increase memory usage make training slower create very sparse feature matrices make the model harder to manage In such cases, other encoding methods may be more suitable. Do not use it when category order matters For ordinal features such as: one hot encoding removes the ordering information. An ordinal encoding may be more appropriate when preserving that order is important. Consider whether the model handles categories natively Some algorithms can work directly with categorical features. In those cases, manual one hot encoding may not be necessary and can sometimes be less efficient. Handle unseen categories safely The encoder should define what happens when validation, test, or production data contains a category that was not seen during training. In short, use one hot encoding for nominal categorical features with a manageable number of unique categories, especially when the model requires numerical input and no meaningful order exists between the categories.
What is ordinal encoding?
Ordinal encoding converts ordered categorical values into numerical values while preserving their natural ranking. It is mainly used for ordinal data , where the categories have a meaningful order. Simple example Suppose a feature contains: Ordinal encoding could map them as: The numerical values preserve the original order: Why ordinal encoding is useful Some machine learning algorithms require numerical input. Ordinal encoding converts ordered categories into numbers without losing the ranking information. This makes it different from one hot encoding, which creates separate binary features for each category. The numerical distance may not be meaningful Ordinal encoding preserves order, but it does not prove that the gaps between categories are equal. For example: does not necessarily mean the difference between Low and Medium is exactly the same as the difference between Medium and High. Some models may nevertheless treat those values as ordinary numbers, so this assumption should be considered carefully. Use a meaningful order The mapping should reflect the real domain order. For example: is meaningful because the categories have a clear progression. Assigning an arbitrary order to nominal categories such as cities or colors would usually be inappropriate. Handle unknown categories Validation, test, or production data may contain categories that were not seen during training. The preprocessing pipeline should define how those unknown categories are represented. Fit the encoding from training data Any learned category mapping should be established using the training data and applied consistently to later datasets. In short, ordinal encoding converts ordered categories into numerical values that preserve their ranking, but the resulting numerical distances should not automatically be interpreted as equal or quantitatively meaningful.