Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
Why should training and test data be separated?
Training and test data should be separated so that a machine learning model is evaluated on examples it did not learn from. If the same data is used for both training and testing, the evaluation can give an unrealistically high estimate of model performance. Why separation matters During training, the model adjusts itself using patterns in the training dataset. If those same examples are later used for testing, the model may perform well partly because it has already seen them. This does not tell us how well the model will handle new data. The test set provides an independent check A separate test dataset helps answer: This is important because the real goal of machine learning is usually to make useful predictions on future or previously unseen data. What can go wrong without separation? Using the same data for training and testing can: produce overly optimistic performance metrics hide overfitting make different models appear better than they really are give false confidence about real world performance Simple example Suppose a model is trained on 10,000 customer records. If those same 10,000 records are used to measure final accuracy, the result does not provide a clean estimate of how the model will perform on new customers. If a separate test set is held back and used only after training is complete, the evaluation is more meaningful. Separation must apply to preprocessing too Information from the test set should also not influence training time preprocessing decisions. For example, statistics used for scaling, imputation, or feature selection should be learned from the training data rather than from the full dataset. Otherwise, information from the test set can leak into the training process. In short, training and test data are separated so the model can be trained on one set of examples and evaluated fairly on different, unseen examples.
What is a train-validation-test split?
A train validation test split is the process of dividing a dataset into three separate parts so a machine learning model can be trained, tuned, and evaluated on different examples. Each split has a different role. Training set The training set is used to fit the model. The learning algorithm uses these examples to adjust the model's learned parameters. Validation set The validation set is used during model development. It helps with tasks such as: comparing model choices tuning hyperparameters selecting features choosing preprocessing options detecting overfitting The model is not normally trained directly on these examples. Test set The test set is used for the final evaluation after model development is complete. It should remain untouched while the model is being trained and tuned so it can provide a cleaner estimate of performance on unseen data. Simple example Suppose a dataset contains 10,000 samples. A possible split might be: 7,000 samples for training 1,500 samples for validation 1,500 samples for testing This is only an example. The best proportions depend on factors such as dataset size, task complexity, and how much data is needed for reliable evaluation. Why use three separate sets? If the same examples influence both learning and final evaluation, the measured performance may be overly optimistic. Keeping the three roles separate helps answer three different questions: Training: What patterns can the model learn? Validation: Which model or settings should we choose? Test: How well does the final model perform on unseen data? In short, a train validation test split separates data for learning, model selection, and final evaluation so model performance can be measured more reliably.
How should data be split for machine learning?
Data should be split in a way that lets the model learn from one portion while being evaluated on separate, representative examples. There is no single split ratio that is correct for every machine learning problem. The right approach depends on dataset size, class balance, time structure, grouping, and how the model will be used. A common starting point For many datasets, a reasonable starting point is: Training data: about 70–80% Validation data: about 10–15% Test data: about 10–15% For example: These percentages are guidelines, not rules. Large datasets may need a smaller percentage for validation and testing because even a small fraction can still contain many examples. Keep the roles separate Training set: used to fit the model. Validation set: used to compare and tune model development choices. Test set: kept untouched until the final evaluation. Make the splits representative Each split should reflect the kind of data the model will encounter in practice. For ordinary independent observations, random splitting is often suitable. If the target classes are imbalanced, stratified splitting can help preserve similar class proportions across the splits. If multiple rows belong to the same person, device, account, or other group, related rows should often stay in the same split so nearly identical information does not appear in both training and evaluation data. For time dependent data, the split should usually follow time order rather than randomly mixing past and future records. Prevent leakage Any preprocessing step that learns information from the data should be fitted using the training data only. For example, values used for imputation, scaling, feature selection, or other learned transformations should not be calculated using the test set. Choose the split based on the real prediction scenario The most important principle is that validation and test data should realistically simulate the unseen data the model will receive after deployment. In short, split data into separate training, validation, and test portions while preserving the structure of the real problem and preventing information from evaluation data from leaking into training.
What is random sampling?
Random sampling is a method of selecting or assigning data points by chance rather than choosing them according to a fixed pattern. In machine learning, it is commonly used when creating training, validation, and test datasets from independent observations. Simple example Suppose a dataset contains 10,000 customer records. Instead of taking the first 7,000 records for training and the last 3,000 for evaluation, the records can be shuffled and assigned randomly to different splits. This helps prevent the original ordering of the data from creating unintended bias in the split. Why random sampling is useful Random sampling can help: give each observation a fair chance of appearing in a particular split reduce bias caused by the original order of the dataset create subsets that are more representative of the overall data support a fairer comparison between training and evaluation performance When random sampling works well It is most appropriate when individual observations are reasonably independent and the dataset does not have an important ordering or grouping structure. For example, randomly splitting a dataset of independent customer records may be reasonable when each customer appears only once. When simple random sampling may not be enough Random sampling should be used carefully when: classes are highly imbalanced multiple rows belong to the same person, account, device, or other group the order of observations matters the data represents events over time In those cases, methods such as stratified, grouped, or time based splitting may be more appropriate. In short, random sampling selects or assigns observations by chance so machine learning datasets are not split according to an arbitrary original ordering.
What is stratified sampling?
Stratified sampling is a sampling method that divides data into meaningful groups, called strata , and then samples from each group so their proportions are preserved. In machine learning, it is commonly used when splitting classification data so the class distribution remains similar across training, validation, and test sets. Simple example Suppose a dataset contains 1,000 transactions: 900 legitimate transactions 100 fraudulent transactions The overall class distribution is: 90% legitimate 10% fraudulent With stratified splitting, each resulting subset is created so it contains approximately the same 90/10 proportion. For example, a 20% test set might contain about: 180 legitimate transactions 20 fraudulent transactions Why stratification helps A purely random split can sometimes produce uneven class distributions, especially when one class is relatively rare. Stratified sampling helps: preserve important group proportions make training and evaluation subsets more representative reduce the chance that a small class is underrepresented in one split make performance comparisons more reliable What is the stratum? The variable used to define the groups depends on the problem. In supervised classification, the target class is commonly used as the stratum. For example: Stratified sampling vs simple random sampling With simple random sampling, observations are selected without explicitly preserving group proportions. With stratified sampling, the data is first grouped and each group is deliberately represented in the resulting sample. In short, stratified sampling preserves the proportions of important groups when creating a sample or dataset split, helping the subsets better reflect the original data.
When should stratified sampling be used?
Stratified sampling should be used when important groups in the dataset need to be represented in similar proportions across the resulting samples or data splits. It is especially useful in classification problems where some classes are much more common than others. Use it when classes are imbalanced Suppose a fraud dataset contains: 98% legitimate transactions 2% fraudulent transactions A simple random split might accidentally place too few fraud examples in the validation or test set. Stratified sampling helps preserve approximately the same 98/2 class distribution in each split. Use it when the dataset is relatively small With smaller datasets, random variation can easily create uneven class distributions. Stratification helps ensure that each subset contains enough examples from important classes to support meaningful training and evaluation. Use it when evaluation depends on class representation Stratification is useful when model performance needs to be measured fairly across all target classes. If one evaluation split contains very few examples of a minority class, metrics such as precision, recall, or F1 score can become unstable or misleading. When stratified sampling may not be the right choice Stratification should not override more important structure in the data. For example: Time dependent data: preserve chronological order instead of randomly stratifying future and past records. Grouped data: keep related records from the same customer, patient, device, or account in the same split when needed. Very rare classes: stratification can preserve proportions, but it cannot create enough examples if the original dataset contains too few. What should usually be stratified? In supervised classification, the target label is most commonly used for stratification. For example, if the target contains spam and not spam, the split can preserve the proportion of those two classes. In short, use stratified sampling when maintaining the representation of important classes or groups is necessary for reliable training and evaluation, particularly with imbalanced or smaller classification datasets.
What is time-based data splitting?
Time based data splitting is a method of dividing a dataset according to chronological order rather than randomly shuffling the observations. It is commonly used when the data changes over time and the goal is to predict future outcomes from past information. Simple example Suppose you have monthly sales data from January 2023 to December 2025. A time based split might use: earlier months for training later months for validation the most recent months for testing For example: The model learns from the past and is evaluated on later periods. Why not randomly shuffle time dependent data? Randomly mixing records from different dates can allow information from the future to influence training. This can produce an unrealistic evaluation because, in real use, future information is not available when predicting the future. When time based splitting is useful It is commonly used for data such as: sales and demand forecasting stock or market data website traffic over time sensor measurements customer behavior histories fraud or transaction data where patterns change over time What time based splitting helps measure It gives a better estimate of how the model will perform when deployed on future data. It can also reveal whether model performance degrades when patterns change over time. Keep preprocessing time aware Any preprocessing or feature engineering that learns from the data should also respect the time boundary. For example, statistics calculated using future records should not be used to transform earlier training data. In short, time based data splitting keeps earlier observations for learning and later observations for evaluation so the experiment reflects real future prediction conditions.
Why should future data not be used to predict the past?
Future data should not be used to predict the past because that information would not have been available at the time the prediction was supposed to be made. Using future information gives the model an unfair advantage and creates an unrealistic evaluation. Simple example Suppose you want to predict on January 1 whether a customer will cancel their subscription during January. The model may use information available up to January 1, such as: past login activity previous payments earlier support requests current subscription details But it should not use: a cancellation request submitted on January 20 February usage data a refund recorded after the customer already cancelled Those events happen after the prediction point. What goes wrong if future data is used? The model may appear extremely accurate because some future variables directly or indirectly reveal what eventually happened. This can cause: inflated validation or test scores misleading confidence in the model poor performance after deployment data leakage Real predictions move forward in time In production, a model normally uses information available now to predict something that happens later. A realistic experiment should reproduce that same information boundary. Data to the right of the prediction point should not be used to construct features for a prediction made on the left. This applies beyond the train test split Future information can leak into the model through: feature engineering preprocessing rolling statistics aggregations labels created with incorrect time windows For example, a 30 day average used for a January 1 prediction must not include observations from dates after January 1. In short, future data must not be used to predict the past because it would not exist at prediction time. Using it creates leakage and produces performance estimates that cannot be reproduced in the real world.
What is data leakage in machine learning?
Data leakage in machine learning happens when information that should not be available to the model influences the training process or model evaluation. It can make a model appear much more accurate during development than it will be in real world use. Simple example Suppose a model is built to predict whether a customer will cancel a subscription. If one input feature records whether the customer later submitted a cancellation request, the model is indirectly being given information about the outcome it is supposed to predict. That is leakage because the information would not have been available at the actual prediction time. Common ways leakage happens Data leakage can occur when: future information is included in input features information closely related to the target is accidentally used as a feature test or validation data influences training preprocessing is fitted using the full dataset before splitting duplicate or related records appear across training and evaluation sets Why leakage is dangerous A leaked model may show excellent validation or test metrics because it has access to information that makes the task artificially easy. After deployment, that information may no longer be available, so model performance can drop sharply. Leakage can therefore lead to: overly optimistic performance estimates poor generalization incorrect model selection false confidence in deployment readiness Leakage can be subtle Some leakage is obvious, such as directly including the answer in the input. Other cases are harder to notice, such as using statistics calculated from future observations or allowing related records to appear in both training and test sets. In short, data leakage occurs when information that would not legitimately be available at prediction time or during independent evaluation enters the machine learning process, causing misleadingly strong results.
What is target leakage?
Target leakage happens when one or more input features contain information that directly or indirectly reveals the target the model is supposed to predict. It is a specific type of data leakage that can make model performance look unrealistically strong. Simple example Suppose you are building a model to predict whether a loan applicant will default. A feature such as: would be a problem if that value is recorded only after the borrower has already defaulted or repaid the loan. The feature contains information about the outcome itself, so the model is not truly learning to predict default in advance. Another example Imagine a model that predicts whether a customer will churn. If one feature is: and the account is marked closed only after the customer leaves, that feature effectively gives away the answer. Why target leakage is dangerous Target leakage can cause: unusually high validation or test scores features that appear extremely predictive false confidence in the model large performance drops after deployment The model may seem excellent during development because it is using information that will not be available when a real prediction must be made. Target leakage can be direct or indirect Direct leakage occurs when a feature explicitly contains the target or a near copy of it. Indirect leakage occurs when a feature is created from information collected after the outcome has happened or from a process strongly tied to the outcome. The key question For every feature, ask: If the answer is no, the feature may be leaking the target. In short, target leakage occurs when the model receives information that reveals the outcome it is supposed to predict, causing deceptively strong performance that usually does not hold in real use.