Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

What are embedded feature-selection methods?

Embedded feature selection methods select important features as part of the model training process itself. Unlike filter methods, which score features before model training, and wrapper methods, which repeatedly test different feature subsets, embedded methods make feature selection decisions while the model is being fitted. How embedded methods work The learning algorithm includes a mechanism that reduces the influence of weak features or identifies which features contribute most to prediction. After training, features with little contribution can be removed or given very low importance. L1 regularization is a common example Models such as Lasso regression use L1 regularization . L1 regularization can push some feature coefficients exactly to zero. For example: Feature B contributes nothing to the fitted model and can effectively be treated as unselected. Tree based models can also provide embedded selection Decision trees and tree based ensembles choose features while creating splits. Features that repeatedly produce useful splits may receive higher importance, while features that are rarely or never used may be candidates for removal. Why embedded methods are useful They often provide a good balance between filter and wrapper approaches because they: consider the behavior of the actual model can capture relationships that simple statistical filters may miss usually require fewer repeated model fits than wrapper methods can perform selection and training in one process They are model dependent The selected features depend on the algorithm. For example, features selected by Lasso may differ from those considered important by a tree based model. There is therefore no single embedded feature set that is automatically best for every model. Regularization strength matters For methods such as Lasso, the amount of regularization affects how aggressively features are removed. Too little regularization may retain many weak features, while too much may remove useful ones. This setting should be chosen using validation. Avoid data leakage Embedded feature selection must be performed inside the training process. When cross validation is used, feature selection should be repeated within each training fold rather than selecting features once using the full dataset. In short, embedded methods perform feature selection during model training. They are usually more model aware than filter methods and more efficient than exhaustive wrapper methods, but the selected features depend on the algorithm and its settings.

What is correlation-based feature selection?

Correlation based feature selection uses correlation values to identify features that are strongly related to the target or highly redundant with other features. It is a common filter based approach for numerical data. Correlation with the target A feature with a strong correlation to a numerical target may contain useful predictive information. For example: Feature A appears to have a much stronger linear relationship with the target than Feature B. Correlation between features Correlation can also be used to detect redundancy. If two features are highly correlated with each other, such as: they may be carrying almost the same information. In that case, one of them may be removed if it does not add additional value. Positive and negative correlation both matter A strong negative correlation can be just as informative as a strong positive correlation. For example: still represents a strong linear relationship. Feature selection therefore often considers the absolute correlation value. Main advantage Correlation based selection is: fast simple to calculate easy to interpret useful as an initial screening method It works well when linear relationships are important. Main limitation Correlation mainly measures linear relationships. A feature can have very low correlation with the target and still be highly predictive through a nonlinear relationship. For example, if the target depends on x², the raw feature x may have weak linear correlation even though it is important. Correlation does not prove causation A strong correlation means two variables move together statistically. It does not prove that one variable causes the other. Avoid leakage If correlation with the target is used for feature selection, calculate it using training data only. Using the entire dataset before validation can leak information from validation or test data. In short, correlation based feature selection uses correlation to keep features that show useful linear relationships with the target and to remove highly redundant features, but it can miss important nonlinear relationships.

What is mutual information in feature selection?

Mutual information measures how much knowing one variable reduces uncertainty about another. In feature selection, it is used to estimate how much information a feature provides about the target. Simple intuition If knowing a feature tells you a lot about the target, their mutual information is high. If the feature and target are independent, their mutual information is theoretically zero. For example: Why mutual information is useful Unlike simple correlation, mutual information can detect both: linear relationships nonlinear relationships This makes it useful when a feature influences the target in a more complex way. Example Suppose the target depends on: The linear correlation between x and y can be weak, even though x clearly contains information about y. Mutual information can still detect that dependency. How it is used for feature selection A mutual information score can be calculated between each candidate feature and the target. Features can then be ranked according to their scores, and stronger features can be kept for model training. This makes mutual information a filter method because the ranking is calculated independently of the final model. It can be used for different problem types Mutual information can be estimated for: classification problems regression problems The exact estimation method depends on whether the feature and target are discrete or continuous. Main limitation Mutual information is often calculated one feature at a time. This means it may miss a feature that is weak by itself but becomes useful when combined with another feature. It may also select several features that contain similar information unless redundancy is handled separately. The score does not show direction Correlation can indicate whether a relationship is positive or negative. Mutual information only measures the strength of dependency; it does not tell you the direction of the relationship. Estimation quality depends on the data Mutual information estimates can be noisy when the dataset is small or when continuous variables are difficult to estimate reliably. The selected features should therefore still be validated using model performance. Avoid data leakage Because mutual information uses the target, calculate it using training data only and perform selection inside the validation process. In short, mutual information measures how much information a feature provides about the target. It can detect nonlinear dependencies that correlation may miss, but it is usually a univariate filter and can miss interactions or redundancy between features.

What is recursive feature elimination (RFE)?

Recursive Feature Elimination (RFE) is a wrapper based feature selection method that repeatedly trains a model and removes the least important features. The process continues until the desired number of features remains. How RFE works RFE typically follows these steps: train a model using all available features calculate feature importance or model coefficients remove the least important feature or group of features train the model again using the remaining features repeat until the required number of features is reached Simple example Suppose a dataset starts with: After training, the model considers D the least useful feature. RFE removes it: The model is trained again, another weak feature is removed, and the process continues. Why RFE can be useful RFE evaluates features in the context of a real model. This means the selected feature set is based on how features contribute to the chosen algorithm rather than only on a simple statistical score. RFE depends on the model The underlying model must provide some way to rank features, such as: model coefficients feature importance scores For example, linear models can use coefficient magnitudes, while some tree based models provide feature importance measures. Main limitation RFE can be computationally expensive because the model is trained repeatedly. The cost becomes more significant when: the dataset has many features the underlying model is slow to train cross validation is also used Choosing the number of features The number of features to keep can be chosen manually or evaluated using validation. A common extension, RFECV , combines RFE with cross validation to help choose the number of selected features. Avoid data leakage RFE should be performed inside the training and validation process. Running RFE once on the full dataset before evaluating the model can leak information from validation or test data. In short, RFE repeatedly trains a model, removes the least important features, and continues until a smaller feature set remains. It is model aware and often effective, but can be computationally expensive.

How can tree-based feature importance be used for feature selection?

Tree based feature importance can be used for feature selection by ranking features according to how much they contribute to the decisions made by a tree model. Features with higher importance are usually considered more useful, while features with very low importance may be candidates for removal. How tree based importance is obtained Models such as decision trees, random forests, and gradient boosted trees can estimate how much each feature contributes to prediction. A common importance measure is based on how much a feature reduces impurity when it is used for a split. Simple example Suppose a model produces: Feature D appears to contribute much less than the others and may be a candidate for removal. Using importance for feature selection A typical process is: train a tree based model on the training data calculate feature importance scores rank the features remove the weakest features or keep the top ranked subset retrain and validate the model The final decision should be based on validation performance rather than the importance ranking alone. Feature importance is model dependent The ranking reflects how a particular fitted tree model used the features. A feature considered unimportant by one model may still be useful for another algorithm. Correlated features can split importance If two features contain similar information, a tree may choose one of them more often than the other. This can make one feature appear highly important while the other appears weak, even though both contain useful information. Impurity based importance can be biased Traditional tree importance can favor features that offer many possible split points, such as continuous variables or high cardinality features. For this reason, importance scores should be interpreted carefully. Validate after removing features Removing low importance features can reduce complexity and computation, but it may also remove useful information. Always retrain the model and compare validation performance after feature selection. Avoid data leakage Feature importance used for selection should be calculated from training data only. If cross validation is used, feature selection should happen inside each training fold rather than once on the full dataset. In short, tree based feature importance can rank features and identify weak candidates for removal, but the scores are model dependent, can be distorted by correlated or high cardinality features, and should always be confirmed with validation.

What is multicollinearity?

Multicollinearity occurs when two or more input features in a regression model are strongly linearly related to each other. This means the features contain overlapping information, making it difficult for the model to determine the separate effect of each one. Simple example Suppose a dataset contains: These two features contain almost exactly the same information. If both are included in a linear regression model, the model has little basis for deciding how much effect should be assigned to each feature separately. Another example Imagine predicting house price using: total area number of rooms These variables may be strongly related because larger houses usually have more rooms. If several predictors move together strongly, multicollinearity can appear. Perfect vs near multicollinearity Perfect multicollinearity occurs when one feature can be calculated exactly from other features. For example: Near multicollinearity occurs when features are not exact combinations but are still very strongly related. Multicollinearity is mainly about predictors The issue concerns relationships among the input features, not simply whether a feature is correlated with the target. Two predictors can each be useful for predicting the target while still being highly correlated with each other. It matters most for coefficient interpretation Multicollinearity is especially important in models such as linear regression, where we often want to interpret the individual coefficient of each feature. When predictors overlap heavily, separating their individual contributions becomes difficult. How can it be detected? Common approaches include: examining correlations between predictors checking whether features are derived from one another using diagnostic measures such as Variance Inflation Factor (VIF) VIF is covered separately later in this section. Not every correlation means severe multicollinearity Some correlation between features is normal. Multicollinearity becomes more concerning when predictors are strongly related enough that their separate effects become difficult to estimate reliably. In short, multicollinearity occurs when predictor variables contain strongly overlapping linear information, making it difficult for a model—especially a regression model—to estimate their individual effects independently.

Why can multicollinearity be a problem?

Multicollinearity can be a problem because strongly related predictors make it difficult for a regression model to estimate the separate effect of each feature reliably. Coefficients can become unstable When two predictors contain similar information, the model may struggle to decide how much weight to assign to each one. Small changes in the training data can then cause large changes in coefficient values. For example: The overall prediction may remain similar even though the individual coefficients change a lot. Standard errors can increase Multicollinearity can increase the uncertainty around coefficient estimates. This makes it harder to determine whether a feature has a statistically reliable individual effect. Coefficient signs can become misleading A feature that should logically have a positive relationship may sometimes receive a negative coefficient because its effect overlaps strongly with another predictor. This can make the model difficult to interpret. Feature importance becomes harder to explain When several predictors carry similar information, it becomes difficult to say which one is actually responsible for the prediction. This is especially problematic when the goal is explanation, policy decisions, or understanding individual feature effects. Statistical significance can become unreliable Because coefficient uncertainty increases, p values and confidence intervals can become less useful for judging individual predictors. A useful feature may appear statistically insignificant simply because another feature contains nearly the same information. Prediction may still be good Multicollinearity does not always mean poor predictive performance . If the goal is only prediction, a model may still perform well because the combined information from the correlated features is useful. The bigger problem is often the stability and interpretation of individual coefficients. Ways to reduce the problem Common approaches include: removing one of several strongly redundant features combining related features using domain knowledge to keep the most meaningful variable applying regularization such as Ridge regression checking diagnostics such as Variance Inflation Factor (VIF) In short, multicollinearity makes individual feature effects unstable and difficult to interpret. It can inflate uncertainty and distort coefficients even when overall predictions remain reasonably accurate.

What is Variance Inflation Factor (VIF)?

Variance Inflation Factor (VIF) measures how much the variance of a regression coefficient is increased because a predictor is linearly related to the other predictors. It is commonly used to diagnose multicollinearity in regression models. How VIF is calculated For a feature Xj, first predict Xj using all the other input features and calculate the resulting R². Then: If the other predictors explain a large portion of Xj, the R² is high and the VIF becomes large. Simple example If: then: This means the variance of that coefficient is inflated relative to a situation where the predictor is not linearly related to the other predictors. How VIF is interpreted Common rules of thumb are: VIF = 1: no linear relationship with the other predictors VIF between 1 and 5: some correlation, often not severe VIF above 5: worth investigating VIF above 10: often treated as a strong multicollinearity warning These are guidelines rather than universal cutoffs. The acceptable level depends on the modeling goal and domain. Why a high VIF matters A high VIF can indicate that a regression coefficient may be: unstable sensitive to small changes in the data associated with a larger standard error difficult to interpret independently High VIF does not automatically mean remove the feature A feature with a high VIF may still contain useful information. Before removing it, consider: whether another feature contains nearly the same information which feature is easier to interpret whether prediction or coefficient interpretation is the main goal whether regularization could help VIF mainly diagnoses linear dependence among predictors VIF is designed around linear relationships between input variables. It does not detect every possible nonlinear dependency between features. In short, VIF measures how strongly multicollinearity inflates the uncertainty of a regression coefficient. A larger VIF indicates that the predictor overlaps more strongly with the other predictors and may be harder to interpret reliably.

What is regression in machine learning?

Regression is a type of supervised machine learning used to predict a continuous numerical value . Instead of predicting a category such as spam or not spam, a regression model predicts a quantity. Examples of regression problems Regression can be used to predict values such as: house price monthly sales temperature delivery time customer lifetime value electricity consumption For example: How regression works A regression model learns a relationship between input features and a numerical target from historical examples. During training, it tries to find a function that produces predictions close to the known target values. In simple form: Regression target The target variable in a regression problem is usually continuous or ordered numerical data. Examples include: Regression vs classification Regression: predicts a numerical quantity, such as price or demand. Classification: predicts a category or class, such as fraud/not fraud or cat/dog. The important question is what kind of output the problem requires. Common regression algorithms Examples include: linear regression Ridge regression Lasso regression decision tree regression random forest regression gradient boosted regression models Different algorithms can model different types of relationships between the features and target. Regression predictions are not always exact A regression model estimates the target based on patterns learned from data. Its quality is evaluated by comparing predictions with actual values using regression metrics such as MAE, MSE, RMSE, or R². These evaluation metrics are covered later in this section. In short, regression is supervised learning for predicting numerical quantities. It learns from examples where the target is a number and estimates that number for new observations.

When should a regression model be used?

Use a regression model when the target you want to predict is a numerical quantity rather than a class or category. Typical regression targets Regression is appropriate for targets such as: house price monthly revenue temperature delivery time energy consumption customer lifetime value For example: Because the target is a number, this is a regression problem. Use regression when the output can vary continuously A regression target usually has many possible numerical values. For example, predicting: is different from predicting a fixed category such as: The first is naturally a regression task, while the second is usually classification. Regression is useful when the size of the prediction matters Sometimes you need more than a yes/no decision. For example: not just whether sales will increase, but by how much not just whether a delivery will be late, but how many minutes it will take not just whether demand is high, but the expected demand value Regression provides that numerical estimate. Historical examples should contain the numerical target Supervised regression requires training examples where the target value is already known. For example: The model learns from these past relationships and predicts values for new observations. Regression is not limited to straight line relationships Although linear regression models linear relationships, regression as a problem type can also be solved using: decision trees random forests gradient boosted trees neural networks These models can capture more complex relationships. Do not use regression just because the target is stored as a number A numerical code can still represent a category. For example: These values are category labels, not continuous quantities, so the problem is not regression. In short, use regression when the target is a meaningful numerical quantity and you want the model to estimate how much, how many, how long, or what value.