How does logistic regression work?

Logistic regression learns a relationship between input features and the probability of a class. For binary classification, it fits weights so examples labeled 1 receive higher predicted probabilities and examples labeled 0 receive lower ones.

Step 1: Calculate a linear score

For each observation, combine the feature values with learned weights and an intercept:

z = w1*x1 + w2*x2 + ... + b

Each w is a feature coefficient; b is the intercept. The score `z` can be any real number.

Step 2: Convert the score into a probability

Pass z through the sigmoid function:

p = 1 / (1 + exp(-z))

This produces 0 < p < 1, interpreted by the model as the estimated probability of class 1.

Step 3: Measure prediction error

During training, compare the estimated probabilities with the known labels using binary cross-entropy (log loss):

Loss = -[y*log(p) + (1-y)*log(1-p)]

• For an actual label of 1, predicting a very low p is penalized heavily.
• For an actual label of 0, predicting a very high p is penalized heavily.

Step 4: Learn the weights

An optimizer changes the coefficients and intercept to reduce the total training loss. Regularization, when used, discourages excessively large coefficients and can help generalization. The learning algorithm is not simply fitting a straight line to the 0/1 labels using squared error.

Worked example: customer churn

Suppose a simplified trained model uses missed payments (x) as its only feature:

z = -2 + 0.8*x

For x = 3, we get z = 0.4, so p = sigmoid(0.4) ≈ 0.60.

Using a 0.50 decision threshold, the predicted label is Churn (1). In practice, choose the threshold on validation data based on the costs of false positives and false negatives.

Key takeaway: logistic regression learns weights by minimizing log loss, transforms their linear score into a probability with the sigmoid, and then applies a decision rule to obtain a class label.

Reference

• scikit-learn: LogisticRegression documentation