Machine Learning
Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.
How do you detect duplicate records in a dataset?
Duplicate records are detected by checking whether two or more rows represent the same observation, entity, or event. Some duplicates are exact copies, while others differ slightly because of formatting, timestamps, spelling, or data entry differences. Start with exact duplicates An exact duplicate occurs when all relevant fields are identical. For example: If the same row appears more than once, it may be an exact duplicate. Most data processing tools can identify rows that repeat across all columns. Check duplicates using unique identifiers If the dataset contains a reliable identifier such as: customer ID transaction ID order ID device ID check whether the same identifier appears more times than expected. A repeated identifier is not always an error. For example, one customer may legitimately have many transactions. The meaning of the identifier must therefore be understood before declaring records duplicates. Check combinations of columns Sometimes no single column uniquely identifies a record. In that case, use a combination such as: Repeated combinations can reveal duplicate events. Look for near duplicates Near duplicates represent the same real world record but are not exactly identical. Examples include: Rahul Sharma vs Rahul Sharma Bengaluru vs Bangalore different capitalization small spelling differences slightly different timestamps for the same event These may require normalization, fuzzy matching, or domain specific rules. Check duplicates across data splits Duplicate or nearly identical records appearing in both training and test data can make model evaluation overly optimistic. For grouped data, also check whether multiple records from the same entity should remain in the same split. Always investigate before removing A repeated row is not automatically incorrect. Some datasets legitimately contain repeated events or measurements, so duplicate detection should be followed by a check of what each row represents. In short, detect duplicate records by checking exact row matches, repeated identifiers, repeated combinations of key fields, and near duplicate patterns, while confirming that the repeated records are not legitimate observations.
How should duplicate records be handled?
Duplicate records should be handled only after confirming that they are truly redundant and not legitimate repeated observations. Removing every repeated looking row automatically can delete useful information. First determine whether the duplicate is real Ask what one row represents. For example: two identical customer profiles may be accidental duplicates two identical product purchases may be legitimate separate transactions repeated sensor readings may be valid observations The business meaning of the data should guide the decision. Remove exact accidental duplicates If two or more rows represent the same real world observation and were duplicated because of data collection, import, or processing errors, keep one valid record and remove the redundant copies. This prevents some observations from being counted more than once. Resolve conflicting duplicates carefully Sometimes records refer to the same entity but contain different values. For example, two rows with the same customer ID may contain different addresses or timestamps. In that case, do not simply delete one row. Instead, determine which value is correct or define a consistent rule such as: keep the most recent record keep the record from the trusted source merge complementary fields send the conflict for manual review Standardize before detecting near duplicates Near duplicates may differ because of formatting rather than meaning. Useful preparation can include: trimming extra spaces standardizing capitalization normalizing dates and phone numbers correcting known category variations This can reveal records that represent the same entity but were written differently. Handle duplicates before data splitting When possible, identify duplicate or related records before creating training and evaluation splits. If copies of the same observation appear in both training and test data, the model may be evaluated on information it effectively already saw during training. This can inflate test performance. Keep a record of what was removed Document the duplicate rule and the number of records affected. This makes preprocessing reproducible and helps detect upstream data quality problems. In short, handle duplicate records by first confirming whether they are truly redundant, removing confirmed accidental copies, resolving conflicting records with clear rules, and preventing duplicates from leaking across training and evaluation data.
What are outliers in machine learning data?
Outliers are data points that are unusually far from most other observations in a dataset. They may represent rare but valid cases, measurement errors, data entry mistakes, or unusual events. Simple example Suppose a feature contains these monthly incomes: The value 2,500,000 is very different from the others and may be considered an outlier. Why outliers occur Outliers can appear because of: genuine rare events measurement or sensor errors data entry mistakes incorrect units unusual customer or system behavior natural variation in the population An outlier is not automatically bad data. Why outliers matter in machine learning Some algorithms are sensitive to extreme values. Outliers can: pull the mean away from typical values affect scaling increase prediction error influence regression lines strongly make some distance based methods behave poorly Other models may be more robust to outliers. Outliers can also be important Sometimes the unusual cases are exactly what the model needs to learn. For example: fraudulent transactions equipment failures unusual medical cases abnormal network activity Removing these observations could remove valuable information. Investigate before changing them When an outlier is found, ask: Is the value possible? Was it recorded correctly? Does it belong to the same population as the rest of the data? Is it important to the prediction problem? Depending on the answer, an outlier may be kept, corrected, transformed, capped, or removed. In short, outliers are unusually extreme observations. They can distort some machine learning models, but they may also represent important real world cases, so they should be investigated rather than automatically deleted.
How do you detect outliers?
Outliers can be detected by looking for observations that are unusually far from the rest of the data. The best method depends on the feature distribution, number of variables, and the meaning of the data. Start with domain checks Some outliers can be found using known valid ranges. For example: an age of 250 years a negative product price a temperature outside the possible range of the sensor These may indicate data entry or measurement errors. Use visual inspection Simple plots can reveal unusual values quickly. Useful visualizations include: box plots histograms scatter plots A point far away from the main cluster may be an outlier worth investigating. Use the IQR method The interquartile range (IQR) is a common robust method for numerical data. First calculate: A common rule marks values outside these limits as potential outliers: Values beyond these bounds are candidates for investigation. Use z scores For data that is reasonably close to a normal distribution, a z score measures how many standard deviations a value is from the mean. Very large positive or negative z scores can indicate possible outliers. A threshold such as |z| 3 is sometimes used as a practical rule, but it should not be treated as universal. Detect multivariate outliers Some records may look normal in each feature separately but become unusual when several features are considered together. Methods such as: Isolation Forest Local Outlier Factor clustering based approaches can help detect these more complex patterns. Investigate before acting An outlier flag means the record is unusual, not necessarily wrong. After detection, ask: Is the value valid? Is it a data error? Is it an important rare case? Does it belong to the same population as the rest of the dataset? In short, outliers can be detected using domain rules, visualizations, statistical methods such as IQR and z scores, or multivariate techniques. Detection should always be followed by investigation before deciding what to do with the record.
When should outliers be removed?
Outliers should be removed only when there is a strong reason to believe they are incorrect, irrelevant, or harmful to the intended machine learning task . Being unusual is not enough by itself to justify deletion. Remove outliers that are clear data errors Removal is often appropriate when an extreme value is impossible or obviously incorrect. For example: an age of 250 years a negative quantity where negative values are impossible a sensor reading caused by a known device failure a value entered in the wrong unit If the correct value cannot be recovered, removing the affected record may be reasonable. Remove records that belong to the wrong population An observation may be valid but irrelevant to the problem being modeled. For example, a dataset intended to model ordinary retail customers may accidentally contain internal test accounts or system generated transactions. Those records can behave like outliers even though their values are technically valid. Consider removal when a small number of extreme points distort the model Some algorithms are very sensitive to extreme values. If a few unusual observations have a large and unjustified influence on the fitted model, removing them may improve stability. However, first confirm that those observations are not important real world cases. Do not remove outliers only because a statistical rule flags them A point outside an IQR boundary or with a large z score is only a candidate outlier . It may still represent a valid observation. For example, very large transactions may be rare but legitimate. Check the effect of removal Before permanently removing outliers, compare the dataset and model with and without them. Check whether removal: improves validation performance changes important relationships disproportionately removes a particular group makes the training data less representative of production data Use alternatives when possible Instead of deleting valid extreme values, you may consider: transforming the feature capping extreme values using robust scaling choosing models or loss functions that are less sensitive to outliers In short, remove outliers when they are confirmed errors, irrelevant records, or unjustified extreme points that distort the model. Valid rare cases should usually be retained rather than deleted simply because they are unusual.
When should outliers be retained?
Outliers should be retained when they are valid observations that represent real situations the model may encounter . An observation should not be removed only because it is rare or far from the average. Retain valid rare cases If an extreme value is correct and belongs to the population being modeled, it may contain important information. For example: a very large but legitimate purchase an unusually high value customer a rare equipment failure an uncommon medical case Removing these records can make the training data less representative of reality. Retain outliers when rare events are the prediction target In some machine learning problems, unusual observations are exactly what matters. Examples include: fraud detection anomaly detection fault detection cybersecurity monitoring Removing unusual records from these datasets could remove the cases the model is supposed to identify. Retain them when the distribution is naturally skewed Some real world variables naturally contain a long tail of extreme values. Examples include: income transaction amounts website traffic insurance claims In such datasets, extreme observations may be part of the true distribution rather than errors. Retain them when they may occur in production The training data should reflect the conditions the model will face after deployment. If valid extreme values can appear in production, removing all of them from training can leave the model poorly prepared for those cases. Use robust methods instead of deletion when appropriate If valid outliers strongly affect the model, consider alternatives such as: transforming the feature using robust scaling capping only when justified choosing models or loss functions that are less sensitive to extreme values This can reduce their unwanted influence without discarding real information. Investigate first Before retaining an outlier, confirm that: the value is valid it was measured or recorded correctly it belongs to the population of interest it is relevant to the machine learning task In short, retain outliers when they are valid, relevant, and representative of situations the model may encounter. Rare does not mean incorrect, and removing genuine extreme cases can make a model less useful in the real world.
What is feature scaling?
Feature scaling is the process of transforming numerical features so they are expressed on comparable scales. It is useful when different features have very different ranges. Simple example Suppose a dataset contains: age: values around 18–80 annual income: values around 200,000–5,000,000 Without scaling, the income feature has much larger numerical values than age. For algorithms that depend on distances, gradients, or the magnitude of coefficients, this difference in scale can strongly affect learning. What feature scaling does Feature scaling changes the numerical representation of a feature while preserving the information it carries. For example, values might be transformed from: into a smaller comparable range or into values centered around a common reference point. Common feature scaling approaches Two widely used approaches are: Normalization: often rescales values into a fixed range such as 0 to 1. Standardization: typically transforms values so the feature has a mean near 0 and a standard deviation near 1. These methods are covered separately in more detail. When feature scaling matters Scaling is particularly important for algorithms that use: distances between data points gradient based optimization feature magnitudes when calculating model coefficients Not every machine learning algorithm requires scaling to the same degree. Fit scaling using training data only Any values needed for scaling, such as a minimum, maximum, mean, or standard deviation, should be learned from the training data. The same learned transformation should then be applied to validation and test data. This prevents evaluation data from influencing preprocessing. In short, feature scaling transforms numerical features into comparable scales so differences in units or magnitude do not unnecessarily influence how certain machine learning algorithms learn.
Why is feature scaling important for some machine learning algorithms?
Feature scaling is important for some machine learning algorithms because features with much larger numerical ranges can influence calculations more strongly than smaller scale features, even when they are not more important. Different scales can distort distance calculations Suppose a dataset contains: age: 18–80 annual income: 200,000–5,000,000 If an algorithm measures distance between data points, the income feature can dominate the distance simply because its numbers are much larger. Scaling puts the features on more comparable numerical ranges. Scaling can improve optimization Many models are trained using iterative optimization methods such as gradient descent. When features have very different scales, the optimization path can become inefficient and require more steps to reach a good solution. Scaled features often help the optimizer move more smoothly and converge faster. Scaling affects regularization Some models use regularization to penalize large coefficients. If features are measured on very different scales, their coefficients are also expressed on different numerical scales. This can cause the regularization penalty to affect features unevenly. Scaling makes coefficient penalties more comparable. Scaling can improve numerical stability Very large differences in feature magnitude can sometimes make calculations less stable or harder for an algorithm to optimize. Bringing features onto similar scales can reduce these numerical issues. Not every algorithm is equally sensitive Some algorithms depend strongly on distances, gradients, or coefficient magnitudes and therefore benefit greatly from scaling. Other algorithms are much less affected by feature scale. The specific algorithms that are sensitive to scaling are covered separately. Fit the scaler on training data only The scaling parameters should be learned from the training set and then applied unchanged to validation and test data. This prevents evaluation data from influencing preprocessing. In short, feature scaling matters because it prevents large numerical ranges from dominating distance calculations, can make optimization more efficient, and helps some models treat features more consistently.
What is normalization in data preprocessing?
Normalization is a data preprocessing technique that rescales numerical values into a common range, often between 0 and 1. It is commonly used when features have very different numerical ranges. Simple example Suppose a feature contains: Using min max normalization to map the values into the range 0 to 1 gives: Min max normalization A common normalization formula is: where: x is the original value min is the minimum value of the feature max is the maximum value of the feature With this transformation, the minimum observed value becomes 0 and the maximum becomes 1. Why normalization is useful Normalization can help when an algorithm is sensitive to the magnitude of numerical features. It can make features easier to compare and prevent a feature with a large numerical range from dominating calculations simply because of its scale. Normalization does not change the ordering If one original value is larger than another, it remains larger after min max normalization. The relative spacing is also transformed consistently within the observed range. Be careful with outliers Min max normalization depends on the minimum and maximum values. An extreme outlier can stretch the range and compress most other values into a small portion of the normalized scale. Fit normalization on training data only The minimum and maximum values should be calculated from the training set. The same values should then be used to transform validation, test, and future production data. This prevents information from evaluation data from leaking into preprocessing. In short, normalization rescales numerical features into a common range, commonly 0 to 1, so differences in raw units or magnitude have less influence on scale sensitive machine learning algorithms.
What is standardization in data preprocessing?
Standardization is a data preprocessing technique that transforms a numerical feature so its values are expressed relative to the feature's mean and standard deviation. It is commonly used to put features with different units or scales onto a more comparable basis. Standardization formula A common formula is: where: x is the original value mean is the average value of the feature standard deviation measures how spread out the feature values are After standardization, the transformed feature typically has a mean close to 0 and a standard deviation close to 1. Simple example Suppose a value is: Then: The standardized value is 2, meaning the original value is two standard deviations above the mean. Why standardization is useful Standardization can help algorithms that are sensitive to feature magnitude, distances, gradients, or coefficient sizes. It allows features measured in very different units to contribute on a more comparable scale. Standardization does not force values into a fixed range Unlike min max normalization, standardized values are not limited to a range such as 0 to 1. Values can be negative, positive, or greater than 1 in magnitude. Outliers can still affect standardization Because standardization uses the mean and standard deviation, extreme values can influence both statistics. For heavily skewed data or strong outliers, a more robust scaling approach may sometimes be preferable. Fit standardization on training data only The mean and standard deviation should be calculated from the training set. The same learned values must then be used to transform validation, test, and future production data. This prevents evaluation data from leaking into preprocessing. In short, standardization rescales a numerical feature by subtracting its mean and dividing by its standard deviation, producing values that are easier for many scale sensitive machine learning algorithms to work with.