Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

What is label encoding and when is it appropriate?

Label encoding converts categorical labels into integer values. For example: Each category receives a unique number. When label encoding is appropriate Label encoding is commonly appropriate for target labels in classification problems. For example, a target such as: can be converted to: The numbers simply represent class identities. It can also work with ordinal input features If an input feature has a real order, numerical codes can preserve that ranking. For example: In this case, the ordering is meaningful. Be careful with nominal input features For unordered categories such as: mapping them to: creates numbers that may look ordered even though the cities have no natural ranking. Some models may interpret those values as if 2 is greater than 1 in a meaningful way. For nominal input features, one hot encoding or another categorical encoding method is often safer. Model choice matters Some tree based models can work reasonably with integer coded categories in certain implementations, while other models are more likely to treat those codes as numerical quantities. The encoding strategy should therefore match the model and the meaning of the feature. Keep the mapping consistent The same category to number mapping must be used for training, validation, test, and production data. Unknown categories should also have a defined handling strategy. In short, label encoding assigns a unique integer to each category. It is commonly appropriate for classification targets and ordered categories, but arbitrary integer codes for nominal input features can introduce a misleading sense of order.

What is target encoding?

Target encoding converts a categorical value into a numerical value calculated from the target variable for that category. It is often used for categorical features with many unique values, where one hot encoding would create too many columns. Simple example Suppose the feature is: and the target is whether a customer purchased a product. If the historical purchase rates are: target encoding may replace each city with its corresponding purchase rate. What statistic is used? For a regression problem, a common target encoding is the mean target value for each category. For a binary classification problem, it may be the proportion of positive target values for each category. The exact statistic depends on the problem. Why target encoding can be useful Target encoding can represent a categorical feature using only one numerical column instead of creating one column per category. This can be useful for high cardinality features such as: city product ID merchant ID category codes with many unique values Rare categories need care A category seen only a few times can produce an unreliable target average. For this reason, target encoding often uses smoothing , which moves estimates for rare categories closer to the overall target average. Unseen categories If a new category appears during validation, testing, or production, a common fallback is the overall target mean or another predefined default. Target encoding must be computed carefully Because the encoding uses the target variable, calculating it directly from all available data can leak information into the model. In practice, training data is often encoded using out of fold or similar leakage safe techniques, while validation and test data are transformed using statistics learned only from training data. The specific risks of target encoding are covered in the next article. In short, target encoding replaces a category with a target based statistic, making it especially useful for high cardinality categorical features, but it must be calculated carefully to avoid leakage and unreliable estimates.

What risks does target encoding introduce?

Target encoding can be powerful, but it introduces several risks because the encoded feature is calculated using the target variable itself . 1. Target leakage The biggest risk is leakage. If the target encoding is calculated using the full dataset before splitting or validation, information from validation or test rows can influence the encoded feature. This can make model performance look much better than it really is. 2. Overfitting on rare categories A category with only a few observations can produce an extreme target average. For example: That value may look highly predictive even though it is based on almost no data. This can cause the model to overfit rare categories. 3. Unstable estimates Target statistics can change significantly when category counts are small. A category's encoded value may therefore vary a lot between different training samples. Smoothing or minimum count rules are often used to make these estimates more stable. 4. Leakage inside cross validation Even when cross validation is used, target encoding can still leak information if each training row is encoded using a statistic that includes its own target. A safer approach is out of fold target encoding : split the training data into folds calculate category statistics from the other folds use those statistics to encode the held out fold This prevents each row's target from directly influencing its own encoded value. 5. Unseen categories A category that appears in validation, test, or production data may not exist in the training data. A fallback strategy is needed, such as using the overall target mean. 6. Distribution changes If the relationship between a category and the target changes over time, previously learned target encodings may become misleading. This is especially important in production systems where customer behavior, products, or business conditions can change. How to reduce the risks Common safeguards include: calculate encodings from training data only use out of fold encoding for training rows smooth estimates for rare categories define a fallback for unseen categories keep the encoding process inside the model validation pipeline In short, target encoding can introduce leakage, overfitting, and unstable estimates because it uses information from the target. Careful train only, out of fold computation and smoothing are essential for reliable use.

How do you handle high-cardinality categorical features?

High cardinality categorical features contain a large number of unique categories, such as thousands of product IDs, customer IDs, cities, or merchant codes. They need special handling because simple one hot encoding can create too many columns. 1. Group rare categories Categories that appear only a few times can be combined into a common group such as: This reduces the number of unique values and can make the feature more stable. 2. Use frequency or count encoding Each category can be replaced by how often it appears. For example: This keeps the feature compact, although it does not directly capture the relationship with the target. 3. Use target encoding carefully Target encoding can replace each category with a statistic derived from the target. It can work well for high cardinality features, but it must be implemented using leakage safe techniques such as out of fold encoding and smoothing. 4. Use hashing Feature hashing maps many categories into a fixed number of numerical features. This keeps memory usage under control even when the number of possible categories is very large. The trade off is that different categories can occasionally map to the same hashed representation. 5. Use models with native categorical support Some algorithms can handle categorical variables directly or with specialized internal encoding. Examples include implementations such as CatBoost and some configurations of LightGBM. This can avoid creating thousands of one hot columns manually. 6. Check whether the feature is actually useful Some high cardinality columns are identifiers rather than meaningful predictive features. For example: may simply identify rows rather than provide reusable predictive information. Such features should be evaluated carefully before encoding. Handle unseen categories Production data may contain categories that were not present during training. The preprocessing pipeline should define a safe fallback for these unseen values. Validate the encoding choice Different encoding methods can change model performance, memory usage, and leakage risk. The best choice should therefore be evaluated using proper validation. In short, high cardinality categorical features can be handled by grouping rare values, using frequency or target encoding, applying feature hashing, or choosing models with native categorical support, while carefully controlling leakage and unseen categories.

What is feature engineering?

Feature engineering is the process of creating, transforming, or selecting input features so they represent the problem in a way that helps a machine learning model learn useful patterns. It often combines domain knowledge with data understanding. Simple example Suppose a dataset contains: A model may find it more useful to work with: rather than the raw date of birth. Creating age from date of birth is an example of feature engineering. Feature engineering can create new features New features can be derived from existing data. Examples include: calculating price per unit from price and quantity extracting day of week from a timestamp calculating account age from an account creation date creating a ratio such as debt / income These new features can make useful relationships easier for the model to learn. It can also transform existing features Feature engineering may change how an existing feature is represented. Examples include: scaling numerical values encoding categorical variables applying a logarithmic transformation to a highly skewed feature grouping rare categories Domain knowledge can be important A useful engineered feature often comes from understanding the real world problem. For example, in a lending problem, debt to income ratio may be more informative than debt or income considered separately. Avoid using information that would not exist at prediction time An engineered feature must only use information that would actually be available when the prediction is made. Otherwise, feature engineering can introduce data leakage. Feature engineering is not just adding more columns Creating many features does not automatically improve a model. Useful features should represent meaningful information and should be evaluated using proper validation. In short, feature engineering turns raw data into more useful model inputs by creating, transforming, or combining features in ways that better represent the prediction problem.

Why is feature engineering important in machine learning?

Feature engineering is important because a machine learning model can only learn from the information that is represented in its input features. Good features can make useful patterns easier for the model to detect. It can expose relationships hidden in raw data Raw columns are not always in the most useful form. For example, instead of giving a model: you may derive: Similarly, instead of using debt and income separately, a feature such as: may represent the real business relationship more directly. It can improve model performance A well designed feature can provide a clearer signal than the original raw values. This can help the model: learn patterns more accurately separate classes more clearly reduce prediction error generalize better to unseen data It can help simpler models work well Sometimes a simple model with strong features can perform better than a complex model trained on poorly represented data. For example, adding meaningful ratios, counts, or time based features can allow a linear model to capture relationships that were difficult to learn from raw columns alone. It can incorporate domain knowledge Feature engineering allows knowledge about the real problem to be included in the dataset. For example: days since last purchase in customer analytics average transaction amount in fraud detection machine age in predictive maintenance These features may reflect the problem more directly than the original raw records. It can make data easier for an algorithm to use Feature engineering can also include transformations such as: encoding categorical variables scaling numerical features transforming skewed variables combining related columns These transformations can make the data more suitable for the chosen model. Poor feature engineering can also hurt Adding irrelevant, noisy, redundant, or future derived features can reduce performance or introduce data leakage. Every engineered feature should therefore be validated rather than assumed to be useful. In short, feature engineering is important because it turns raw data into representations that expose useful patterns, incorporate domain knowledge, and make it easier for machine learning models to learn the relationships that matter.

What is feature extraction?

Feature extraction is the process of transforming raw data into a set of informative features that a machine learning model can use more effectively. It is especially useful when the original data is large, complex, or high dimensional. Simple idea Instead of giving a model all raw values directly, feature extraction creates a more compact representation. For example, an image may contain thousands of pixel values. A feature extraction method can transform those pixels into a smaller set of values that capture important patterns such as shapes, edges, or overall structure. Feature extraction can reduce dimensionality Some methods create fewer features while preserving as much useful information as possible. For example, Principal Component Analysis (PCA) can transform many correlated numerical features into a smaller number of principal components. It can transform raw data into useful measurements Examples include: converting text into numerical representations such as word frequency features extracting frequency related measurements from audio signals deriving compact components from many correlated numerical variables summarizing raw sensor signals into useful statistical features Why feature extraction is useful It can help: reduce the number of input features remove some redundant information make complex data easier for a model to process reduce memory and computation requirements expose patterns that are difficult to use directly from raw data Information can be lost Feature extraction changes the original representation. If too much information is compressed or discarded, important predictive signals may be lost. The extracted representation should therefore be evaluated using validation data. Fit learned transformations on training data only If the feature extraction method learns information from the dataset, such as PCA components, it should be fitted using the training data only. The same learned transformation should then be applied to validation and test data to avoid leakage. In short, feature extraction transforms raw or high dimensional data into a more compact and useful set of features that can make machine learning models easier and more efficient to train.

What is the difference between feature engineering and feature extraction?

Feature engineering and feature extraction both improve how data is represented for a machine learning model, but they are not exactly the same. Feature engineering Feature engineering is the broader process of creating, transforming, combining, or selecting features to better represent the prediction problem. Examples include: creating age from date of birth calculating debt to income ratio extracting day of week from a timestamp grouping rare categories applying a logarithmic transformation It often uses domain knowledge to design features that make important relationships easier for the model to learn. Feature extraction Feature extraction transforms existing raw data into a new representation that captures useful information. Examples include: using PCA to convert many correlated variables into principal components converting text into numerical representations extracting statistical summaries from sensor signals converting raw image information into more compact feature representations The extracted features may be less directly interpretable than the original inputs. Key differences Scope: Feature engineering is the broader concept. Feature extraction is one way of creating a useful representation from existing data. Typical goal: Feature engineering often introduces domain informed features, while feature extraction often compresses or transforms raw data into informative components. Interpretability: Engineered features such as age or price per unit are often easy to understand. Extracted features such as PCA components may be harder to interpret. Dimensionality: Feature engineering can increase, reduce, or keep the same number of features. Feature extraction often reduces or reorganizes the original feature space. Method: Feature engineering may involve manual rules and domain knowledge, while feature extraction is often performed using systematic statistical or algorithmic transformations. They can be used together A machine learning pipeline may use both. For example: Both must respect the training boundary If either process learns information from the dataset, it should be fitted using training data only and then applied consistently to validation and test data. In short, feature engineering is the broader process of designing useful model inputs, while feature extraction specifically transforms existing data into a new, often more compact or informative, representation.

How can date and time columns be converted into useful features?

Date and time columns can be converted into useful machine learning features by extracting patterns that are easier for a model to learn than a raw timestamp. Extract calendar components A timestamp such as: can be converted into features such as: year month day of month day of week hour minute quarter These features can reveal seasonal or recurring patterns. Create business relevant flags Date and time values can also be converted into indicators such as: weekend vs weekday business hours vs outside business hours holiday vs non holiday beginning or end of month These can be useful when behavior changes across different parts of the calendar. Calculate elapsed time features Often the most useful feature is the amount of time between two events. Examples include: customer age from date of birth account age from account creation date days since last purchase time since previous transaction delivery duration These features usually represent the underlying relationship more directly than the original timestamps. Represent cyclical time properly Some time features repeat in cycles. For example, hour 23 and hour 0 are close in time even though their raw numerical values are far apart. Such features can be represented using cyclical transformations so the model can preserve that circular relationship. Preserve chronological information when needed For time dependent problems, features can also describe trends or sequence position, such as: time since the start of the dataset week number month number over multiple years These can help a model capture long term changes. Avoid future information leakage Every time based feature must be calculated using only information available at the moment the prediction would be made. For example, days until customer churns would leak future information because that value would not be known at prediction time. Keep time zones consistent When timestamps come from multiple regions or systems, convert them to a consistent time zone before deriving features. Otherwise, features such as hour of day or day of week may be incorrect. In short, date and time columns become more useful when they are transformed into calendar components, elapsed time measures, business flags, or cyclical representations that reflect the real temporal patterns in the problem.

How can numerical features be transformed?

Numerical features can be transformed to make their scale, distribution, or relationship with the target easier for a machine learning model to use. 1. Scale the feature Numerical values can be rescaled using methods such as normalization or standardization. This is useful when features have very different ranges and the algorithm is sensitive to magnitude. 2. Apply a logarithmic transformation A log transformation can reduce strong right skew in features such as: income transaction amount population website traffic For example: becomes roughly: when using a base 10 logarithm. This compresses very large values more than small ones. 3. Use square root or power transformations Square root, Box Cox, or Yeo Johnson style transformations can also change the shape of a numerical distribution. They may help when the feature is strongly skewed or its relationship with the target is nonlinear. 4. Clip or cap extreme values Very large or very small values can sometimes be limited to chosen boundaries. For example, values above the 99th percentile may be capped at that threshold when there is a clear modeling reason to reduce extreme influence. This should not be done automatically without understanding the outliers. 5. Bin continuous values A continuous feature can be grouped into ranges. For example: Binning can simplify relationships, but it also loses some numerical detail. 6. Create ratios or interaction features Existing numerical features can be combined to create more meaningful values. Examples include: price per unit = price / quantity debt to income = debt / income distance per hour = distance / time These derived features may represent the real world relationship better than the original columns separately. Choose transformations using validation A transformation should have a clear reason and should be evaluated on validation data. Transforming a feature does not automatically improve a model. Fit learned transformations on training data only Any transformation that learns statistics from the data—such as scaling parameters, percentile thresholds, or power transform parameters—should be fitted on the training set and then applied unchanged to validation and test data. In short, numerical features can be transformed through scaling, log or power transforms, clipping, binning, or derived ratios to make useful patterns easier for a model to learn.