Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

What is a polynomial kernel in SVM?

A polynomial kernel is a similarity function that lets an SVM model polynomial relationships between input features without explicitly creating every polynomial feature. It can learn curved decision boundaries when a straight line is insufficient. Formula x · z: dot product of two feature vectors. degree: polynomial degree (for example, 2 or 3), controlling the order of feature interactions. gamma: scales the dot product. coef0: constant offset; affects how lower order and higher order terms contribute. In scikit learn, use SVC(kernel='poly'). Its polynomial parameters are degree, gamma, and coef0; the regularization parameter C is tuned separately. Beginner friendly numerical example Let x = (1, 2), z = (2, 3), gamma = 0.5, coef0 = 1, and degree = 2: Why is this different from a linear kernel? Expanding the square gives: The squared dot product term implicitly incorporates products between features, such as x1 x2. These interactions allow a linear separator in the implicit feature space to become nonlinear in the original space. Important: the kernel output 25 is a similarity calculation, not a class probability . Practical example: a center surrounded by alerts Imagine a sensor with two readings. A central reading (0, 0) is Normal , while readings at (±2, 0) and (0, ±2) are Alert . A single straight line cannot place the center on one side and all surrounding points on the other. A degree 2 polynomial SVM can learn a boundary around the center. When should you use a polynomial kernel? Use it when feature interactions or polynomial shaped boundaries are plausible. Start with lower degrees , then tune degree, gamma, coef0, and C using cross validation. Higher degree is not automatically better. Scale numerical features using training only preprocessing inside a pipeline; kernel values are sensitive to feature magnitudes. Compare against linear and RBF kernels on the same validation splits . Polynomial kernels can overfit or be computationally expensive on large datasets. In short: a polynomial kernel computes (gamma x·z + coef0)^degree, allowing SVMs to capture polynomial feature interactions without manually expanding them. References scikit learn: SVM kernel functions scikit learn: Classification boundaries with different SVM kernels scikit learn: SVC parameters

How do you choose an SVM kernel?

Choose an SVM kernel by comparing how well plausible kernels generalize on validation data—not by assuming nonlinear kernels are always better. Start with a linear baseline , then consider RBF and polynomial kernels when the data suggests nonlinear patterns. Practical selection guide Linear kernel: good first choice when the boundary is approximately linear, features are high dimensional and sparse (such as text), or simplicity and speed matter. For large datasets, compare LinearSVC as a more scalable linear alternative. RBF (Gaussian) kernel: good candidate when a linear boundary underfits and class relationships are smoothly nonlinear. It has gamma and C to tune. Polynomial kernel: consider when interactions among features and a particular polynomial degree are plausible. Tune degree, gamma, coef0, and C; higher degree is not inherently better. Beginner friendly example Imagine classifying customer risk using income and debt ratio : If low and high risk examples are separated roughly by a straight line, a linear SVM may be sufficient. If risk rises in several curved regions, RBF may capture the pattern better. If the relevant relationship depends on interactions such as income multiplied by debt ratio, a polynomial kernel may be worth testing. These are hypotheses, not guarantees. The decision comes from held out performance. A reliable way to choose 1. Reserve a test set for final evaluation. 2. Use a pipeline that scales features inside training folds, preventing leakage. 3. Compare kernels under the same cross validation splits and task appropriate metric. 4. Tune each kernel's parameters, rather than comparing defaults alone. 5. Consider prediction latency, memory, interpretability, and the number of training examples—not just the cross validation score. Python example: compare three kernels The make moons dataset deliberately contains nonlinear structure, so an RBF or polynomial kernel may outperform linear here; the winner must still be measured , not assumed. Important cautions Scale features correctly : RBF distances and polynomial dot products are sensitive to measurement scales. Tune C jointly with kernel settings : for RBF, gamma affects how local the influence of training examples is. Large gamma and large C can produce overly intricate boundaries. Watch scalability : kernel SVC training can become expensive on very large datasets. Try LinearSVC or another scalable baseline where appropriate. Prevent test leakage : choose the kernel using validation/cross validation only; evaluate the final selection on untouched test data. In short: begin with linear SVM, test RBF or polynomial when justified, and choose using cross validated generalization plus practical operating costs . References scikit learn: SVM user guide scikit learn: RBF SVM parameter effects scikit learn: GridSearchCV

What is Naive Bayes?

Naive Bayes is a supervised probabilistic classifier based on Bayes' theorem. It is called naive because it assumes features are conditionally independent given the class. How it works Estimate class priors, such as P(Spam). Estimate feature likelihoods, such as P(free | Spam). For a new example, multiply each class prior by the likelihoods for its features. Predict the class with the highest score. Core formula The denominator is the same for all candidate classes for one observation, so it can be omitted when selecting the largest score. Implementations often sum log probabilities instead of multiplying tiny numbers. Beginner friendly example A simplified email classifier uses binary indicators for the words free and win . Prior probabilities: Spam = 0.4; Not spam = 0.6. In spam emails: P(free=1)=0.8; P(win=1)=0.7. In non spam emails: P(free=1)=0.1; P(win=1)=0.2. For an email containing both words: Since 0.224 is larger, the prediction is Spam . Under this simplified model, the normalized estimate is 0.224 / (0.224 + 0.012) = 94.9% . This is a model estimate, not a guarantee of well calibrated probability. Main variants Gaussian Naive Bayes: for continuous features modeled with Gaussian distributions. Multinomial Naive Bayes: for nonnegative count like features, often document word counts. Bernoulli Naive Bayes: for binary presence/absence features. Strengths and limitations Naive Bayes is fast to train and predict, often works well in text classification, and is a useful small data baseline. Its conditional independence assumption can be unrealistic; correlated features may produce poorly calibrated probabilities. Smoothing helps prevent zero likelihoods for unobserved features. In short: Naive Bayes classifies by combining prior class frequencies with the likelihood of the observed features in each class. Reference scikit learn: Naive Bayes

Why is Naive Bayes called naive?

Naive Bayes is called “naive” because it makes a strong simplifying assumption: all features are independent of one another once the class label is known. Real world features are often related, so this assumption is usually only an approximation. What does conditional independence mean? Suppose a spam classifier looks for the words free and discount . Its naive assumption is: That says knowing whether free appears would not change the likelihood of discount , after we already know the email is spam . It does not mean the words must be independent across all emails; independence is assumed conditional on the class . Beginner friendly example: duplicate evidence Imagine two features extracted from an email: Feature A: contains the word free . Feature B: contains the word free (an exact duplicate of A). Suppose P(A=1 | Spam)=0.8 and P(A=1 | Not Spam)=0.2. Because B is a duplicate, the real joint probability of both being present is the same as A alone: True joint likelihood given Spam: 0.8. Naive estimate: 0.8 0.8 = 0.64. True joint likelihood given Not Spam: 0.2. Naive estimate: 0.2 0.2 = 0.04. With equal class priors, the actual likelihood ratio for the duplicated evidence is 0.8 / 0.2 = 4; the naive calculation makes it 0.64 / 0.04 = 16. The evidence is counted twice , even though the second feature adds no information. This can make the estimated probabilities excessively confident. Why use the assumption at all? It greatly simplifies the calculations : the classifier estimates each feature's class conditional likelihood instead of all combinations of features. It can train efficiently with many features , such as word presence signals in text classification. It can still classify accurately even when conditional independence is violated, because ranking the class scores can be easier than estimating perfectly calibrated probabilities. Does “naive” mean inaccurate? No. The word describes the simplifying statistical assumption, not that the classifier is automatically poor. The assumption can work surprisingly well for spam filtering and document categorization, although correlated features may distort probability estimates. In short: Naive Bayes is called naive because it treats features as conditionally independent given the class —a convenient but often unrealistic simplification. References scikit learn: Naive Bayes NLTK: The Naivete of Independence

What is Gaussian Naive Bayes?

Gaussian Naive Bayes (GaussianNB) is a probabilistic classifier designed for continuous numerical features . For each class, it assumes each feature follows a Gaussian (normal) distribution , described by a mean and variance. Like other Naive Bayes models, it assumes features are conditionally independent given the class. How does it work? 1. Calculate class priors: estimate how often each class occurs in the training set. 2. Learn Gaussian parameters: calculate the mean and variance of each numerical feature separately for each class. 3. Evaluate a new observation: calculate its Gaussian likelihood under each class. 4. Choose the class with the largest prior weighted product of feature likelihoods (usually computed using log scores for numerical stability). Gaussian likelihood formula Here, mean and variance are estimated from the training observations belonging to that class . The formula gives a probability density , not a probability of an exact continuous value. Numerical example: predict a sensor condition Suppose a sensor measurement is used to identify two conditions: Normal: measurements 1, 2, 3. Alert: measurements 7, 8, 9. Both classes have three observations, so the estimated class priors are 0.5 each . Normal: mean = (1+2+3)/3 = 2; variance = ((1 2)^2 + (2 2)^2 + (3 2)^2)/3 = 2/3. Alert: mean = 8; variance = 2/3. For a new reading 2.5 , its Gaussian density is far higher under the Normal class (centered near 2) than the Alert class (centered near 8). With equal priors, Gaussian Naive Bayes predicts Normal . Python example When should you use it? Useful for continuous measurements such as temperature, duration, or physical sensor readings when the per class distributions are reasonably Gaussian. Often provides a fast, simple baseline with limited training data. Less suitable when feature distributions are strongly non Gaussian or features remain highly correlated within classes. Practical detail: scikit learn's GaussianNB uses var smoothing to add a small positive quantity to feature variances for numerical stability, particularly when a feature has little or no variance. In short: Gaussian Naive Bayes learns a mean and variance for every feature in every class , then uses Gaussian likelihoods plus class priors to classify new numerical observations. References scikit learn: Gaussian Naive Bayes guide scikit learn: GaussianNB API

What is Multinomial Naive Bayes?

Multinomial Naive Bayes (MultinomialNB) is a probabilistic classifier for nonnegative count like features , especially word counts in text classification. It learns how frequently each word appears in each class, then uses these frequencies and the class priors to classify new documents. How it works Convert each document into a vector of word counts, such as [count(free), count(meeting)]. For every class, total the occurrences of each word across its training documents. Estimate each word's class conditional probability , using smoothing to avoid zero probabilities. Score a new document by combining the class prior and probabilities for its word counts. The highest score determines its predicted class. Mathematical idea With V vocabulary words and additive smoothing alpha: This multinomial model uses how many times a word appears, not simply whether it appears. Numerical example: spam versus normal email Suppose the vocabulary has exactly two words : free and meeting. Their aggregated training counts are: Spam: free = 8, meeting = 2 (10 total). Normal: free = 1, meeting = 9 (10 total). With Laplace smoothing (alpha=1) and V=2: P(free | Spam) = (8+1)/(10+2) = 9/12 = 0.75. P(meeting | Spam) = (2+1)/12 = 3/12 = 0.25. P(free | Normal) = (1+1)/12 = 2/12 ≈ 0.167. P(meeting | Normal) = (9+1)/12 = 10/12 ≈ 0.833. For a new message containing free free meeting , assume equal class priors of 0.5: Since the Spam score is higher, Multinomial Naive Bayes predicts Spam . These are unnormalized class scores, not calibrated probabilities. Python example This is a deliberately tiny example for illustration. Real applications typically use CountVectorizer with many training documents and should evaluate on unseen data. Nonnegative TF IDF features can also work in practice, although the original multinomial interpretation is based on counts. When to use it—and what to watch Good fit: spam detection, topic labeling, and other document classification with nonnegative token count features. Fast and scalable: useful as a baseline for large sparse text matrices. Smoothing: alpha=1 is Laplace smoothing; it prevents an unseen word in one class from automatically giving that class a zero score. Limitation: ignores word order and assumes conditional independence of features. It is not designed for arbitrary negative valued measurements. In short: Multinomial Naive Bayes predicts from word frequencies within each class , using smoothing and class priors to score unseen documents. References scikit learn: Multinomial Naive Bayes scikit learn: MultinomialNB API

What is Bernoulli Naive Bayes?

Bernoulli Naive Bayes is a probabilistic classifier for binary features —each feature is either 1 (present/yes) or 0 (absent/no). It is often used with word presence indicators in text classification. How does it work? Convert each input into binary indicators, such as contains free and contains meeting. Estimate how often each feature is present within each class , along with the class prior. For a new input, account for both present features and absent features . Predict the class with the highest prior weighted score. The important difference Unlike Multinomial Naive Bayes, which models word counts , Bernoulli Naive Bayes models presence or absence . A missing word matters: if a word is common in a class but absent from a document, the classifier explicitly factors that absence into its score. Formula Let p jc = P(feature j=1 | class=c) and x j be either 0 or 1: With Laplace smoothing (alpha=1) for a binary feature: Numerical example: filtering email Suppose the features are free and meeting. From earlier emails, the learned conditional probabilities are: Spam: P(free=1 | Spam)=0.8, P(meeting=1 | Spam)=0.2. Normal: P(free=1 | Normal)=0.1, P(meeting=1 | Normal)=0.7. Consider an email that contains free but does not contain meeting . With equal class priors of 0.5: Prediction: Spam. Crucially, (1 0.2) and (1 0.7) explicitly model the absence of meeting . Both results are unnormalized scores, not calibrated probabilities. Python example Why binarize=None? The example already contains 0/1 values. When working with numeric inputs that are not binary, scikit learn can binarize them using a threshold; choose that preprocessing intentionally. When is it useful? Short documents, spam filtering, or other classification problems where a yes/no signal is meaningful. Large, sparse matrices of binary features, such as whether a word appears in a document. A fast baseline before testing more complex classifiers. Limitations: repeating a word several times gives no additional count information once it is converted to 1. The model also makes the usual conditional independence assumption. Compare it with MultinomialNB on held out data rather than assuming either model always wins. In short: Bernoulli Naive Bayes uses whether each feature is present or absent , while Multinomial Naive Bayes usually focuses on how frequently features occur . References scikit learn: Bernoulli Naive Bayes scikit learn: BernoulliNB API

When is Naive Bayes a good classifier?

Naive Bayes is a good classifier when fast, inexpensive training and prediction are important , especially for high dimensional sparse text features or relatively small labeled datasets. It is an excellent baseline , but its usefulness should be checked on held out data. Good use cases Text classification: spam filtering, support ticket routing, and document categorization with word counts or binary word presence features. High dimensional sparse data: can learn a separate likelihood for each feature and class without modeling every feature interaction. Limited labeled examples: simple parameter estimates can be effective even when more complex methods struggle. Quick baselines: fast to train and easy to compare against logistic regression or linear SVM. Incremental learning: several Naive Bayes implementations support updates through partial fit. Practical example: ticket categorization A company wants to route incoming support tickets into Billing , Account , or Delivery . Words such as refund, password, and parcel can help identify a category. Multinomial Naive Bayes is a sensible first model for word count features. This very small dataset is illustrative, not proof of accuracy. Validate with enough independent examples. Which variant works best? MultinomialNB: nonnegative word counts and similar features. BernoulliNB: yes/no indicators such as word presence. GaussianNB: continuous measurements whose distribution within each class is approximately Gaussian. When might it be a poor choice? Correlated features: its conditional independence assumption may effectively count related evidence more than once. Complex feature interactions: nonlinear models may capture patterns Naive Bayes cannot. Reliable probabilities required: class probability estimates can be poorly calibrated even when classification accuracy is good. Feature distributions mismatch: do not assume one Naive Bayes variant suits every data type. How do you decide? Evaluate Naive Bayes using stratified cross validation , an appropriate metric such as macro F1 for imbalanced multiclass data, and a comparison baseline. Keep text vectorization within each training fold to prevent data leakage. Judge performance on unseen examples, not training accuracy alone. In short: choose Naive Bayes as a fast, practical baseline for text and other compatible feature representations; keep it if validation and deployment requirements support it. Reference scikit learn: Naive Bayes

What is a confusion matrix?

A confusion matrix is a count based summary of a classification model's predictions compared with the true labels. It shows both the correct predictions and the different kinds of mistakes. How do you read it? For a binary classifier, with class 1 treated as positive, scikit learn arranges the counts as: True negative (TN): actual 0, predicted 0. False positive (FP): actual 0, predicted 1. False negative (FN): actual 1, predicted 0. True positive (TP): actual 1, predicted 1. The diagonal counts correct predictions. The off diagonal counts mistakes. The order of axes can differ in other software or diagrams, so always check the labels. Worked example: detecting faulty products A model inspects 20 products , where 1 means faulty and 0 means good: 8 good products correctly predicted good ( TN ). 2 good products wrongly flagged faulty ( FP ). 3 faulty products wrongly predicted good ( FN ). 7 faulty products correctly flagged faulty ( TP ). There are 8 + 7 = 15 correct predictions and 2 + 3 = 5 errors. Accuracy is 15 / 20 = 75%, but the matrix reveals that three faulty products were missed —potentially a bigger concern than the two false alarms. Python example Setting labels=[0, 1] makes the row/column order explicit. Beyond binary classification For three or more classes , a confusion matrix has one row and column per class. Each diagonal cell counts correct predictions for that class; other cells show which classes the model confuses. You can also normalize counts by true class with confusion matrix(y true, y pred, normalize='true'). This helps compare error patterns when class sizes differ. When is it useful? To see which errors a model makes, not only how often it is right. To compare results on imbalanced classes , where accuracy alone can hide poor minority class performance. To identify classes that need better features, labeling, or more training examples. In short: a confusion matrix answers “Which classes did the classifier get right, and which did it confuse?” Later curriculum questions build precision, recall, and related metrics from these counts. References scikit learn: confusion matrix API scikit learn: confusion matrix example

What are true positives, true negatives, false positives, and false negatives?

True positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) describe the four possible outcomes of a binary classifier , assuming one class is explicitly defined as positive . What does each term mean? Consider a disease screening test where Positive means “the patient has the disease.” True Positive (TP): The patient has the disease and the test predicts positive. True Negative (TN): The patient does not have the disease and the test predicts negative. False Positive (FP): The patient does not have the disease, but the test predicts positive ( false alarm ). False Negative (FN): The patient has the disease, but the test predicts negative ( missed case ). “True” means the prediction is correct; “false” means incorrect. “Positive” and “negative” describe the predicted class in TP/FP/TN/FN terminology, not whether an outcome is desirable. Numerical example A screening system examines 100 patients : 30 actually have the disease; 70 do not. Among those with the disease, 24 test positive and 6 test negative. Among those without the disease, 10 test positive and 60 test negative. Therefore: Check the totals: Actual positives: TP + FN = 30. Actual negatives: TN + FP = 70. Predicted positives: TP + FP = 34. Predicted negatives: TN + FN = 66. How do they relate to a confusion matrix? Using actual labels as rows and predicted labels as columns : Be careful: some diagrams reverse the axes. Always read the row/column labels rather than memorizing a layout. Python example Why do these outcomes matter? In medical screening , missed cases (FN) can be especially costly. In spam detection , false positives may hide legitimate emails. These four counts are the building blocks for accuracy, precision, recall, specificity, and F1 score —covered in the next curriculum questions. In short: TP and TN are correct predictions; FP is a false alarm; FN is a missed positive case. Decide which class counts as positive before interpreting the numbers. References scikit learn: Confusion matrix scikit learn: Classification metrics