Machine Learning

Machine learning is a way to build software by fitting models to examples instead of writing every rule by hand. A model sees data, optimizes an objective, and generalizes the learned pattern to new inputs.

Why is feature scaling important for KNN?

Feature scaling is important for KNN because KNN decides which examples are “nearest” using distances between feature values. If one feature has much larger numerical values than another, it can dominate the distance calculation even when it should not dominate the prediction. Why scale matters For two numerical features, Euclidean distance is: Each squared difference contributes directly. A salary measured in dollars may vary by thousands, while age measured in years varies by tens. Without scaling, differences in salary can overwhelm age differences. Practical example: the predicted class can change Suppose we predict whether a customer will Renew using age and annual salary . Our new customer is (age=30, salary=50000). Two labeled customers are: A: (31, 60000) → Renew . B: (45, 50050) → Cancel . Without scaling: Distance to A = sqrt(1^2 + 10000^2) ≈ 10000.00. Distance to B = sqrt(15^2 + 50^2) ≈ 52.20. With K = 1 , B is nearest, so the prediction is Cancel . After expressing salary in thousands (divide each salary by 1000, consistently for all points): Distance to A = sqrt(1^2 + 10^2) ≈ 10.05. Distance to B = sqrt(15^2 + 0.05^2) ≈ 15.00. Now A is nearest, so the prediction is Renew . The underlying customers and labels did not change. Only the feature scale changed—but it changed who counted as nearest. Dividing salary by 1000 here is just an illustrative transformation, not a universally correct scaling rule. How to scale features in practice StandardScaler: subtract the feature's training set mean and divide by its training set standard deviation. MinMaxScaler: rescale features to a chosen range, commonly 0–1; it can be strongly affected by outliers. RobustScaler: use training set median and interquartile range when extreme values make mean based scaling less suitable. Choose scaling based on the data and validation performance. Scaling does not prove that two features are equally informative, and the appropriate distance metric still matters. Correct scikit learn approach Avoid data leakage: fit the scaler only on training data , then use the same fitted transformation for validation, test, and new observations. A pipeline handles this correctly and also helps during cross validation. In short: KNN depends on the geometry of the features . Scaling can prevent arbitrary measurement units from distorting distances, neighbor selection, and predicted labels. References scikit learn: Importance of Feature Scaling scikit learn: StandardScaler scikit learn: Pipelines and data leakage prevention

What is a Support Vector Machine (SVM)?

A Support Vector Machine (SVM) is a supervised learning method used mainly for classification, and also for regression and some outlier detection tasks. In classification, an SVM learns a decision boundary that separates classes while generally trying to keep a large margin between the boundary and nearby training examples. How does SVM classification work? 1. Read labeled training examples. Each example has feature values and a known class. 2. Find a separating boundary. A linear SVM looks for a line in two dimensions, a plane in three dimensions, or a hyperplane in higher dimensions. 3. Prefer a wide margin. When separation is possible, the model favors a boundary with a large distance to the closest examples from both classes. 4. Predict a new example's class. Its position relative to the learned boundary determines the class label. The training examples closest to the boundary are called support vectors . They are especially influential in determining the boundary. Practical example: spam classification Imagine a simplified email dataset with just one feature, a suspicious content score : Legitimate emails have scores 1 and 2. Spam emails have scores 6 and 7. For these perfectly separable examples, a maximum margin boundary is at score = 4 , halfway between the closest opposite class examples (2 and 6). A new email with score 3 falls on the Legitimate side. A new email with score 5 falls on the Spam side. The examples with scores 2 and 6 are the support vectors in this simple hard margin case. Real spam classifiers use many features and are rarely perfectly separable. Soft margin SVMs allow a trade off between a wide margin and classification mistakes; the hyperparameter C helps control that trade off. What about nonlinear patterns? Linear SVM: learns a straight or flat decision boundary in the provided feature space. Kernel SVM: can learn more flexible boundaries using a kernel, such as the radial basis function (RBF). Feature scaling matters: SVM is sensitive to feature magnitudes, so preprocessing should be fit using training data only. Try it with scikit learn Important: An SVM normally returns a class or decision score, not automatically a calibrated probability . Also, kernel SVM training can become expensive for very large datasets. In short: SVM classifies by learning a boundary with a useful margin; the closest training examples—its support vectors —are central to that boundary. References scikit learn: Support Vector Machines scikit learn: SVC API

How does an SVM find a separating hyperplane?

An SVM finds a separating hyperplane by choosing the boundary that leaves the widest possible gap between two classes , subject to its training constraints. This is called maximum margin classification . 1. Represent the boundary For a linear binary SVM, the decision boundary is: x is the feature vector. w determines the boundary's orientation. b shifts the boundary. The sign of w · x + b determines the predicted class ( 1 or +1). With two features, the hyperplane is a line ; with three it is a plane ; with more features it is a higher dimensional hyperplane. 2. Find the maximum margin separator Many different lines may correctly separate linearly separable examples. A hard margin SVM selects one that maximizes the distance to the closest examples from either class . With labels y i in { 1, +1}, its standard optimization is: The constraints place correctly classified examples on or beyond the two supporting margin planes. Minimizing ||w|| maximizes the width 2 / ||w|| of the band between them. 3. Worked example: a line in two dimensions Suppose four labeled points have features (x1, x2): Class 1: ( 3, 1) and ( 2, 0). Class +1: (2, 0) and (3, 1). The maximum margin separating line is x1 = 0 . In the canonical SVM scaling: The closest points, ( 2, 0) and (2, 0) , lie exactly on the two margin boundaries; these are the support vectors . For a new point ( 1, 0), the score is 0.5 → predict Class 1 . For (1, 0), the score is +0.5 → predict Class +1 . 4. What if classes overlap? Real datasets may not be perfectly separable. A soft margin SVM permits some margin violations while penalizing them; parameter C controls the trade off between margin size and those violations. A kernel SVM can also create a nonlinear boundary in the original feature space. Python example In short: an SVM does not choose just any separating line. It solves an optimization problem to find a wide margin boundary , mainly determined by the nearby support vectors. References Stanford CS229: Support Vector Machines scikit learn: Maximum margin separating hyperplane example scikit learn: SVM mathematical formulation

What is the margin in an SVM?

The margin in a Support Vector Machine (SVM) is the distance between its decision boundary and the closest training examples. A maximum margin SVM prefers a separating boundary with as much space as possible around it, rather than one that merely separates the training labels. What does the margin look like? Imagine two groups of points with a line between them: The decision boundary is the middle line where the classifier changes its prediction. The margin boundaries are parallel lines touching the nearest training examples from the two classes. Those nearby examples are called support vectors . Some explanations call the distance from the middle line to the nearest point the margin ; others call the full distance between the two margin boundaries the margin width. It helps to state which measure is being used. Numerical example Suppose we have a two feature dataset: Class A ( 1): (0, 1) and (1, 0). Class B (+1): (0, 5) and (1, 6). The maximum margin decision boundary is the horizontal line x2 = 3 . The closest opposing points are (0, 1) and (0, 5). Distance from x2 = 3 to (0, 1) = 2 units . Distance from x2 = 3 to (0, 5) = 2 units . Full distance between the margin boundaries = 4 units . Compare a boundary at x2 = 2: it still separates the classes, but the nearest Class A point is only 1 unit away. The boundary at x2 = 3 has a larger minimum distance to either class , so SVM prefers it in this perfectly separable example. Margin formula For a linear decision function f(x) = w · x + b, the perpendicular distance from a point x to the hyperplane is: For a correctly separated hard margin SVM in canonical scaling, the closest training points satisfy y i f(x i) = 1: For the example above, w = (0, 0.5) and b = 1.5. Thus ||w|| = 0.5, and the full width is 2 / 0.5 = 4. Verify in Python Why is a large margin useful? A wider margin makes the boundary less dependent on tiny changes close to the training points and is one reason SVMs can generalize well. It does not guarantee better test accuracy , especially with noisy data. For overlapping classes, a soft margin SVM allows some observations to lie inside the margin or even be misclassified. Its C parameter balances margin size against those violations; this is covered separately in the upcoming soft margin and C questions. In short: an SVM's margin measures the clearance around its decision boundary. Maximizing that clearance is central to how a hard margin SVM selects among possible separators. References scikit learn: SVM maximum margin separating hyperplane scikit learn: Support Vector Machines, mathematical formulation

What are support vectors?

Support vectors are the training examples that actively determine an SVM's decision boundary. In the trained SVM's dual representation, they are the examples with nonzero coefficients ; examples with zero coefficients do not appear in its decision function. Why are they called support vectors? In a hard margin linear SVM , support vectors are the closest points from the two classes, lying on the margin boundaries. These points support the position and orientation of the maximum margin separator. Points farther away usually do not determine the separator. However, changing or removing a support vector can change the fitted boundary. In a soft margin SVM , support vectors can also lie inside the margin or be misclassified ; they are not always exactly on the two margin lines. Practical example: temperature alert Imagine a simple classifier trained with one feature, sensor temperature in degrees Celsius: Normal: 15°C and 18°C (class 0). Overheated: 24°C and 28°C (class 1). For this separable dataset, the maximum margin decision boundary is 21°C , halfway between 18°C and 24°C: 18°C and 24°C are support vectors , the two nearest opposing training examples. 15°C and 28°C are not support vectors ; they are farther from the boundary. Removing 15°C leaves the separator at 21°C in this example. Removing 18°C changes the closest opposing pair to 15°C and 24°C, moving the separator to 19.5°C . That illustrates the key idea: not every training point contributes equally to the final SVM boundary . What does the model calculate? In the dual form of a binary SVM, its decision score can be written as: x i is a support vector and y i is its class label, encoded as 1 or +1. alpha i 0 is its learned dual weight. K(x i, x) measures similarity through the chosen kernel; with a linear kernel it is a dot product. Other training points have alpha i = 0, so they contribute nothing directly to this decision score. Find support vectors in Python Do not confuse SVM with KNN: KNN finds nearby examples separately for each prediction. An SVM learns its support vector set during training and uses it for subsequent predictions. In short: support vectors are the influential examples that anchor an SVM's separating boundary; in a soft margin model they may be on or inside the margin. References scikit learn: Support Vector Machines scikit learn: SVC attributes scikit learn: Maximum margin separating hyperplane example

What is a soft-margin SVM?

A soft margin Support Vector Machine (SVM) is a classifier that permits some training examples to fall inside the margin or even on the wrong side of the decision boundary . It trades perfectly separating every training point for a boundary that may generalize better to new data. Why is a soft margin needed? A hard margin SVM requires every training example to be correctly classified and outside its margin. This fails when classes overlap or labels are noisy, and it can produce a brittle boundary even when perfect separation is possible. A soft margin SVM instead balances two objectives: Keep a large margin around the decision boundary. Penalize margin violations , including misclassified examples, without forbidding them outright. Practical example: detecting spam Suppose we use one feature, a suspicious content score, and observe: Legitimate emails ( 1): scores 1 and 4. Spam emails (+1): scores 3 and 7. No single cutoff can perfectly separate these four examples: spam at 3 overlaps legitimate mail at 4. A hard margin linear SVM therefore has no feasible perfect separator . A soft margin SVM can still fit a line, accepting some mistakes. For illustration, consider a candidate decision score f(x) = x 3.5 (not necessarily the fitted optimum): Legitimate x = 1: score 2.5 → correctly classified, outside the margin. Spam x = 7: score +3.5 → correctly classified, outside the margin. Spam x = 3: score 0.5 → misclassified, incurs a penalty. Legitimate x = 4: score +0.5 → misclassified, incurs a penalty. How does a soft margin work mathematically? For class labels y i equal to 1 or +1, a linear soft margin SVM introduces a slack variable xi i for each example: xi i = 0: correctly classified, on or outside its margin boundary. 0 < xi i < 1: correctly classified, but inside its margin. xi i = 1: on the decision boundary. xi i 1: on the wrong side of the decision boundary. For the candidate line above, the two misclassified examples each require xi i = 1.5, while the other two need 0. This follows from xi i = max(0, 1 y i f(x i)). C controls the trade off: increasing it penalizes margin violations more heavily; decreasing it permits greater violations in exchange for stronger regularization. Selecting C is covered in the next curriculum question. Try it with scikit learn Important: soft margin means permitting violations, not deliberately misclassifying every difficult point. The learned boundary depends on the data, kernel, feature scaling, and C. In short: a soft margin SVM handles overlapping or noisy classes by allowing penalized margin violations, rather than insisting on impossible or fragile perfect separation. References scikit learn: Support Vector Machines, mathematical formulation scikit learn: SVC API scikit learn: Hinge loss

What is the C parameter in an SVM?

C is the regularization parameter of a soft margin Support Vector Machine (SVM). It controls how strongly the model penalizes training examples that lie inside the margin or on the wrong side of the decision boundary. Key rule: smaller C = stronger regularization; larger C = weaker regularization. This is the reverse of what beginners sometimes expect from the letter C. What happens when C changes? Small C (for example, 0.01): margin violations are relatively inexpensive. The optimizer is more willing to accept them in exchange for a wider margin and a more strongly regularized model. If C is too small, the classifier may underfit. Large C (for example, 100): violations are expensive. The optimizer tries harder to fit the training labels, potentially choosing a narrower margin. If C is too large, the classifier may overfit noise. Neither direction guarantees a specific result: changing C does not always change the fitted predictions, and higher training accuracy does not necessarily mean better test accuracy. Where does C appear mathematically? A standard linear soft margin SVM solves: ||w||^2 / 2 encourages a wider margin. xi i is the violation (slack) assigned to training example i. C sum(xi i) determines how costly the violations are relative to the margin term. Numerical example Imagine an email spam classifier with one training example whose slack is 0.5 for a particular candidate boundary: At C = 0.1, its contribution to the penalty is 0.1 × 0.5 = 0.05. At C = 10, its contribution is 10 × 0.5 = 5. The same violation costs 100 times more with C = 10 than with C = 0.1. When training, the SVM may change the decision boundary to reduce that cost—even if doing so reduces the margin. The slack value here is illustrative, not a claim that a fitted model keeps its slack fixed as C changes. How should you choose C? Try C values on a logarithmic scale, such as 0.01, 0.1, 1, 10, 100, using cross validation. Fit feature preprocessing inside each fold to avoid data leakage, and evaluate the final selected model once on a separate test set. In scikit learn, SVC defaults to C=1.0 , but that is a starting point rather than an optimal choice for every dataset. In short: C controls how much an SVM prioritizes fitting the training examples versus maintaining a strongly regularized, wide margin solution. Select it based on validation performance , not training accuracy alone. References scikit learn: SVC regularization parameter scikit learn: RBF SVM parameters and the effect of C scikit learn: SVM mathematical formulation

What is the kernel trick?

The kernel trick lets an algorithm work as if it had mapped data into a richer, higher dimensional feature space—without explicitly constructing those new features. In an SVM, this can produce a nonlinear decision boundary in the original input space , even though the separator is linear in the implicit feature space. Why do we need it? A linear SVM separates data with a straight line (in two input dimensions). Some patterns cannot be separated that way. For example, consider an XOR pattern: (0, 0) and (1, 1) → Class 0. (0, 1) and (1, 0) → Class 1. No straight line separates those classes perfectly in the original 2D space. A suitable feature transformation can make a linear separator possible in a richer representation. How does the trick work? 1. Imagine transforming each input x with a feature map phi(x) that creates extra features (for example, feature interactions). 2. Many SVM calculations only require dot products between transformed examples: phi(x) · phi(z). 3. Replace those expensive transformed dot products with a kernel function that computes the same value directly from the original inputs: This is the trick : the algorithm uses the expanded representation implicitly , without materializing phi(x) for every example. Numerical example: polynomial kernel For two dimensional vectors, one valid degree 2 polynomial kernel is: An equivalent explicit feature mapping is: Take x = (1, 2) and z = (3, 4): Direct kernel calculation: (1 3 + 2 4)^2 = 11^2 = 121. Explicit feature dot product: (1, 2 sqrt(2), 4) · (9, 12 sqrt(2), 16) = 9 + 48 + 64 = 121. Both approaches give exactly 121 , but the kernel computes the result without explicitly constructing those three new feature coordinates. The efficiency benefit becomes more important for very large feature mappings. Practical Python example Common kernels and limitations Linear: ordinary dot product, so there is no nonlinear feature expansion. Polynomial: implicitly introduces polynomial feature interactions. RBF (Gaussian): corresponds to a rich, potentially infinite dimensional feature space. It is covered separately later in the curriculum. Important: the kernel trick does not make SVM training free. Computing and storing similarities between many training examples can be costly; kernel choice and parameters still need validation. In short: instead of explicitly creating new features , the kernel trick directly computes their dot products, enabling an SVM to learn more flexible decision boundaries. References Stanford STATS 202: Kernels and the kernel trick scikit learn: SVM kernel functions and mathematical formulation

What is a linear kernel?

A linear kernel is the simplest kernel used by a Support Vector Machine (SVM): it computes the dot product of two input vectors. Unlike nonlinear kernels, it does not introduce a nonlinear feature mapping; a linear SVM learns a straight line boundary in two dimensions or a hyperplane in higher dimensions. Formula Here x and z are two feature vectors of the same length. A linear kernel measures their unnormalized dot product similarity. It is not Euclidean distance or cosine similarity : vector magnitudes affect its value. Simple numerical example Take two vectors x = (2, 3) and z = (4, 1): Nothing has been transformed into extra polynomial or radial features. The calculation uses the original inputs directly. How does it classify? With a linear kernel, an SVM learns a decision score: Positive score: one class. Negative score: the other class. Zero score: on the learned decision boundary. Practical example: classify customer activity Suppose two comparable scale features summarize customer activity. We train on: (1, 1) and (1, 2) → Inactive (0) . (4, 1) and (4, 2) → Active (1) . A maximum margin linear boundary for this symmetric, separable dataset is x1 = 2.5: New customer (2, 1) → Inactive (0) . New customer (3, 1) → Active (1) . The boundary is a straight line: this linear kernel SVM cannot draw an arbitrary curved boundary on the supplied raw features. Python example When should you use a linear kernel? Try it first when the classes are reasonably separable in the supplied feature space, or when you need a simpler, more interpretable boundary. It is often a strong choice for high dimensional sparse features , such as bag of words text representations. For large datasets, consider LinearSVC : it uses a specialized linear solver and often scales better than SVC(kernel='linear'). Their implementations, default loss functions, and multiclass handling differ, so their fitted results need not be identical. Scale numerical features when appropriate, and select the regularization parameter C with validation data. In short: a linear kernel computes K(x, z) = x · z and produces a linear SVM decision boundary. It is a practical baseline before considering RBF or polynomial kernels for nonlinear patterns. References scikit learn: SVM kernel functions scikit learn: Linear kernel scikit learn: LinearSVC API

What is an RBF kernel?

The Radial Basis Function (RBF) kernel, also called the Gaussian kernel, measures similarity based on the distance between two feature vectors. It lets an SVM learn curved, nonlinear decision boundaries when a straight line cannot separate the classes. Formula x and z are two feature vectors. ||x z||^2 is their squared Euclidean distance. gamma (γ) is a positive hyperparameter that controls how quickly similarity falls with distance. exp denotes the exponential function. Interpretation: identical points have kernel similarity 1 . As two points move farther apart, their similarity approaches 0 . This value is a kernel similarity, not a predicted class probability. Worked numerical example Take x = (1, 2) and set gamma = 0.2. Nearby point a = (2, 2): squared distance = (1 2)^2 + (2 2)^2 = 1; similarity = exp( 0.2 1) ≈ 0.819. Farther point b = (2, 4): squared distance = (1 2)^2 + (2 4)^2 = 5; similarity = exp( 0.2 5) ≈ 0.368. The RBF kernel therefore gives the nearer point greater similarity without inventing additional manual features. How does it help an SVM? A linear SVM creates a straight separating boundary in the supplied feature space. An RBF SVM uses these distance based kernel values in its learned decision function, effectively working with a much richer feature representation. This allows nonlinear boundaries in the original input space. Example: Imagine a sensor where readings close to (0, 0) are Normal , while readings around that point are Alert . A single straight line cannot put the center on one side while keeping the surrounding points on the other; an RBF SVM can model such a pattern. Python example What does gamma change? Smaller gamma: similarity decreases slowly with distance; each training point has a broader range of influence. The decision boundary often becomes smoother. Larger gamma: similarity decreases rapidly; points affect a more local neighborhood. The boundary can become intricate and may overfit. Gamma is not the same as C. gamma controls the kernel's distance scale; C controls how strongly SVM penalizes margin violations. Both should be tuned using validation data. When is RBF useful? When classes appear nonlinearly separable and a linear baseline is inadequate. When a moderate size dataset supports kernel calculations. After scaling numerical features , since Euclidean distances are affected by feature units. Limitations: RBF is not always better than linear models, and kernel SVM training can become expensive on very large datasets. In short: RBF compares examples through distance based Gaussian similarity , enabling SVMs to learn nonlinear boundaries. Its most important parameter is gamma , which determines how local that similarity is. References scikit learn: RBF kernel definition scikit learn: RBF SVM parameters scikit learn: SVC API