Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
What is train-test contamination?
Train test contamination happens when information from the test dataset influences the training or model development process. This breaks the independence of the test set and can make the final evaluation look better than the model will perform on truly unseen data. Simple example Suppose the same customer record, or a nearly identical duplicate, appears in both the training set and the test set. The model may effectively see that example during training and then be evaluated on it again. That makes the test result less trustworthy. Common causes of train test contamination Contamination can happen when: duplicate or related records appear in both training and test sets preprocessing is fitted on the full dataset before splitting feature selection uses information from the test set imputation or scaling statistics are calculated using test data the test set is repeatedly used to choose models or tune hyperparameters Why contamination is a problem The purpose of the test set is to simulate data the model has never seen before. If test information influences training, the evaluation is no longer independent. This can lead to: inflated test scores hidden overfitting incorrect model comparisons unrealistic expectations for production performance Example with preprocessing Suppose missing values are filled using the mean calculated from the entire dataset before creating the train test split. The training process has now indirectly used information from the test set. A safer approach is to calculate the mean using only the training data and then apply that learned value to the test data. Test data should remain isolated The test set should be treated as unseen data until the final evaluation. Training, preprocessing, feature selection, and model tuning should be completed without using test set information. In short, train test contamination occurs when information from the test set leaks into training or model development, causing the test score to overestimate real world performance.
How can data leakage be detected?
Data leakage can be detected by looking for signs that the model has access to information it should not have during training or evaluation. Leakage is often discovered by combining unusually strong performance signals with a careful review of features, timestamps, preprocessing, and data splits. Look for suspiciously high performance If a model achieves results that seem far better than expected, investigate whether some feature is revealing the answer. For example: near perfect validation accuracy extremely high precision or recall with little effort a sudden large improvement after adding one feature Strong performance is not proof of leakage, but it is a useful warning sign. Inspect highly predictive features Check which features contribute most to the model's predictions. A feature deserves investigation if it: is almost perfectly correlated with the target is created after the outcome occurs contains status information that indirectly reveals the outcome would not exist at the real prediction time Check feature timestamps For time dependent problems, compare when each feature becomes available with when the prediction is supposed to be made. If a feature uses information recorded after the prediction point, it may be leaking future information. Check for overlap between data splits Look for: exact duplicate rows nearly identical records the same customer, patient, device, or account appearing across splits when that should not happen Such overlap can make evaluation data too similar to training data. Review preprocessing carefully Check whether preprocessing steps were fitted using only training data. Potential leakage can occur if the full dataset was used to calculate: means or medians for imputation scaling statistics feature selection scores encodings aggregate features Compare with a more realistic holdout If possible, evaluate the model on a truly independent or later time period. A large performance drop on this production like data can indicate that earlier validation results were benefiting from leakage or contamination. Ask the prediction time question For every feature and transformation, ask: If not, investigate it as a possible leakage source. In short, data leakage is detected by auditing features and preprocessing, checking split overlap and timestamps, and investigating performance that looks unrealistically strong compared with truly unseen data.
How can data leakage be prevented?
Data leakage can be prevented by designing the machine learning pipeline so that information from the future, the target, or evaluation data cannot influence training. The main principle is simple: the model should only learn from information that would genuinely be available when a real prediction is made. Split the data before learning from it Create the training, validation, and test splits before applying preprocessing steps that learn statistics from the data. For example, if missing values are filled using a median, calculate that median from the training set only. Then apply the learned value to the validation and test sets. Fit preprocessing only on training data Steps such as these should usually be fitted using training data only: scaling imputation feature selection dimensionality reduction learned encodings aggregate statistics This prevents validation or test information from leaking into the model development process. Respect the prediction time boundary For every feature, ask whether it would actually be available when the prediction is made. Do not use: future events post outcome information variables created after the target has already occurred Time based problems should preserve chronological order. Keep related records together when necessary If multiple rows belong to the same customer, patient, device, household, or account, placing related rows in different splits can create leakage. Use a group aware split when the real prediction scenario requires those entities to remain separate. Protect the test set The test dataset should not be used to: choose features tune hyperparameters select models adjust preprocessing repeatedly guide development decisions Use validation data for development decisions and keep the test set for final evaluation. Use pipelines Machine learning pipelines can help ensure that preprocessing is fitted inside each training split and then applied correctly to validation or test data. This is especially important during cross validation. Review feature creation carefully Document how and when each feature is produced. A feature that looks harmless may still contain future information or a hidden version of the target. In short, prevent data leakage by isolating evaluation data, fitting learned transformations only on training data, respecting time and group boundaries, and ensuring every feature would truly be available at prediction time.
Why is data preprocessing important in machine learning?
Data preprocessing is important because raw data is often incomplete, inconsistent, noisy, or stored in a form that machine learning algorithms cannot use directly. Preprocessing prepares the data so the model can learn from it more reliably. Raw data is rarely ready for training A real world dataset may contain: missing values duplicate records inconsistent categories incorrect data types extreme values features measured on very different scales text or categories that need numerical representation If these issues are ignored, the model may learn misleading patterns or fail to train correctly. Preprocessing improves data quality Common preprocessing tasks include: handling missing values removing or correcting invalid records converting data into suitable formats encoding categorical variables scaling numerical features when required handling duplicates and outliers Each task should be chosen based on the data and the model being used. It helps models interpret features correctly Some algorithms are sensitive to how features are represented. For example, if one feature ranges from 0 to 1 and another ranges from 0 to 1,000,000, certain algorithms may be strongly influenced by the larger numerical scale unless the data is transformed appropriately. It can improve model reliability Good preprocessing can help: reduce noise improve training stability make features more meaningful improve generalization to unseen data make evaluation more trustworthy However, preprocessing cannot fix a fundamentally poor or unrepresentative dataset. Preprocessing must avoid leakage Any preprocessing step that learns information from the data should be fitted using the training set only. For example, if missing values are replaced using the median, that median should be calculated from the training data and then applied to validation and test data. This prevents evaluation data from leaking into the training process. In short, data preprocessing turns raw data into a cleaner, consistent, and model ready form so machine learning algorithms can learn more reliably and be evaluated fairly.
How do you handle missing values in a dataset?
Missing values should be handled based on why the data is missing, how much is missing, and how important the affected feature is . There is no single best method for every dataset. Start by understanding the missingness Before filling or removing values, check: which columns contain missing values how many values are missing whether missingness is concentrated in certain groups whether the missing value itself carries useful information Missing data can happen because a value was not collected, was not applicable, was lost during processing, or was unavailable at the time. Common ways to handle missing values 1. Remove rows Rows with missing values may be removed when: only a very small number of rows are affected enough data remains after removal the missing rows are not systematically different from the rest Removing too many rows can waste information or introduce bias. 2. Remove columns A feature may be dropped when most of its values are missing and it provides little useful information. However, an important feature should not be removed only because it contains some missing values. 3. Impute missing values Imputation replaces missing values with estimated or representative values. Common simple approaches include: mean imputation for numerical data median imputation for numerical data with skew or outliers mode imputation for categorical data a separate category such as Unknown for some categorical features More advanced methods can estimate missing values using other features. Missingness can itself be informative Sometimes whether a value is missing contains useful predictive information. For example, the fact that a customer did not provide a certain field may be meaningful. In such cases, a separate missing value indicator can be added. Prevent leakage during imputation Any imputation value learned from the data should be calculated using the training set only. For example, if the median is used to fill missing values, calculate the median from the training data and then apply that same value to validation and test data. Consider models that handle missing values Some machine learning algorithms can handle missing values directly, depending on the implementation. Even then, the meaning and pattern of missingness should still be investigated. In short, handle missing values by first understanding their cause and extent, then choose an approach such as removal, imputation, missing indicators, or native model handling while avoiding data leakage.
What is mean imputation?
Mean imputation is a simple method for handling missing numerical values by replacing each missing value with the arithmetic mean of the observed values in that feature. Simple example Suppose a feature contains: The mean of the observed values is: The missing value can therefore be replaced with 12.5. When mean imputation can be useful It can be a reasonable baseline when: the feature is numerical the amount of missing data is relatively small the values are not strongly skewed there are no extreme outliers dominating the mean It is simple, fast, and keeps all rows in the dataset. Limitations of mean imputation Mean imputation does not recreate the true missing values. It can also: reduce the natural variation in the feature distort relationships between variables be strongly affected by outliers hide meaningful patterns in why values are missing For skewed numerical data, median imputation is often more robust. Avoid data leakage The mean should be calculated using the training data only. That learned mean can then be applied to the validation and test sets. Using the full dataset to calculate the mean would allow evaluation data to influence preprocessing. In short, mean imputation replaces missing numerical values with the feature's average value. It is simple and useful as a baseline, but it should be used carefully when data is skewed, contains outliers, or has substantial missingness.
What is median imputation?
Median imputation is a method for handling missing numerical values by replacing each missing value with the median of the observed values in that feature. The median is the middle value after the observed values are sorted. Simple example Suppose a feature contains: The observed values are: The median is the average of the two middle values: The missing value can therefore be replaced with 13. Why use median instead of mean? The median is less affected by extreme values and skewed distributions. In the example above, the value 100 pulls the mean upward, but it has much less effect on the median. This makes median imputation useful when: the feature is numerical the data is skewed the feature contains outliers only a limited amount of data is missing Limitations of median imputation Median imputation is still a simple replacement method. It can: reduce the natural variation in the feature distort relationships between variables hide patterns in why values are missing become less reliable when a large portion of the feature is missing Avoid data leakage The median should be calculated using the training data only. The learned median can then be applied to the validation and test sets. Calculating the median from the entire dataset would allow evaluation data to influence preprocessing. In short, median imputation replaces missing numerical values with the feature's median. It is especially useful for skewed data or data with outliers because the median is more robust than the mean.
What is mode imputation?
Mode imputation is a method for handling missing values by replacing each missing value with the most frequently occurring value in that feature. It is especially common for categorical data. Simple example Suppose a feature contains: The most frequent value is: So the missing value can be replaced with Red. When mode imputation is useful Mode imputation can be a reasonable choice when: the feature is categorical one category occurs more often than the others only a limited amount of data is missing keeping all rows is preferable to removing incomplete records It can also be used with discrete numerical values when using the most common value makes sense. Limitations of mode imputation Replacing missing values with the most common category can distort the original distribution. It may: make the dominant category even more common reduce variation in the feature hide useful information about why values are missing be misleading when multiple categories have similar frequencies In some cases, creating a separate category such as Unknown may preserve more information than replacing missing values with the mode. What if there is more than one mode? A feature can have multiple values tied for the highest frequency. In that case, the imputation rule should be defined consistently, such as choosing one category using a fixed rule rather than making an arbitrary choice each time. Avoid data leakage The mode should be calculated using the training data only. The learned mode can then be applied to validation and test data. In short, mode imputation replaces missing values with the most frequently occurring value in a feature and is commonly used for categorical data, but it can overrepresent the dominant category if used carelessly.
When should missing rows or columns be removed?
Missing rows or columns should be removed only when doing so does not discard too much useful information or introduce bias into the dataset. Deletion can be appropriate, but it should not be the default response to missing data. When removing rows can make sense Rows with missing values may be removed when: only a small percentage of rows are affected enough data remains after removal the missing rows are not systematically different from the rest the missing value affects a feature that is essential for the task and cannot be imputed reliably For example, if 20 rows are incomplete in a dataset with 100,000 otherwise similar records, removing them may have little impact. When removing columns can make sense A column may be removed when: most of its values are missing the feature provides little predictive or business value the data cannot be recovered or imputed meaningfully another feature already provides similar information For example, a column that is 95% missing and has little relevance to the target may not be worth keeping. When rows or columns should not be removed automatically Deletion can be harmful when: a large portion of the dataset would be lost the missing records belong disproportionately to an important group the missing feature is strongly related to the target the fact that a value is missing may itself contain useful information In these cases, removing data can change the dataset distribution and introduce bias. Check the effect before deleting Before dropping rows or columns, measure: the percentage of missing values how much data will remain whether missingness is concentrated in particular classes or groups whether model performance changes after removal Prefer the least destructive option If the missing values can be handled reliably through imputation, a missing indicator, or a model that supports missing values, those approaches may preserve more information than deletion. In short, remove missing rows or columns when the amount of affected data is small or the feature has little value, and when deletion is unlikely to distort the dataset or remove important information.
How can missingness itself be useful as a feature?
Missingness itself can be useful as a feature when the fact that a value is missing carries information about the outcome being predicted . In such cases, instead of only filling the missing value, you can also create a separate indicator showing whether the original value was missing. Simple example Suppose a loan dataset contains an income feature, but some applicants do not provide their income. You might create: Then you can still impute the missing income value while allowing the model to learn whether the missingness itself is informative. Why missingness can contain information A value may be missing for a reason. For example: a customer skipped an optional form field a medical test was not ordered a sensor failed under certain operating conditions a user did not complete part of a process If those situations are related to the target, the missing value pattern can help the model. Missing indicator + imputation A common approach is: create a binary missing value indicator impute the missing value using an appropriate method give both features to the model This separates two pieces of information: the estimated value used in place of the missing entry the fact that the original value was missing When this may not help A missing indicator is not automatically useful. If values are missing completely at random and the missingness has no relationship with the target, the indicator may add little or no predictive value. It should be evaluated using validation data. Watch for leakage Missingness indicators must also respect the prediction time boundary. If a value becomes missing only because of an event that happens after the prediction point, using that missingness could leak future information. In short, missingness can be useful as a feature when the absence of a value carries meaningful information. A separate missing value indicator lets the model learn from that signal while the original feature is handled through imputation or another method.