How does changing the classification threshold affect predictions?

Changing the classification threshold changes which observations receive a positive label. It does not change a trained model’s predicted probabilities or its ranking of observations; it changes the decisions made from those scores.

What happens when the threshold moves?

For a binary classifier that predicts Positive when p >= threshold:

• Lower threshold: more examples are labeled positive. True positives and false positives can increase; recall cannot decrease on the same fixed dataset.
• Higher threshold: fewer examples are labeled positive. False positives and true positives can decrease; recall cannot increase on that same dataset.

Precision can move in either direction as the threshold changes, because both the numerator (true positives) and denominator (all predicted positives) can change. F1-score can also rise or fall.

Worked example: fraud detection

Assume eight transactions have these fixed fraud scores and *actual* labels:

A: 0.95 (Fraud)       E: 0.56 (Fraud)
B: 0.83 (Legitimate)  F: 0.47 (Fraud)
C: 0.72 (Fraud)       G: 0.35 (Legitimate)
D: 0.62 (Legitimate)  H: 0.12 (Legitimate)

There are four fraudulent and four legitimate transactions.

Threshold = 0.70

• Flag A, B, and C.
• True positives = 2; false positives = 1; false negatives = 2; true negatives = 3.
• Precision = 2/3 ≈ 66.7%; recall = 2/4 = 50%.

Threshold = 0.50

• Flag A, B, C, D, and E.
• True positives = 3; false positives = 2; false negatives = 1; true negatives = 2.
• Precision = 3/5 = 60%; recall = 3/4 = 75%.

Threshold = 0.30

• Flag A through G.
• True positives = 4; false positives = 3; false negatives = 0; true negatives = 1.
• Precision = 4/7 ≈ 57.1%; recall = 4/4 = 100%.

In this example, lowering the threshold catches more fraud but also blocks more legitimate transactions. In other datasets, precision need not decline steadily as the threshold falls.

What stays the same?

• Predicted scores and their ranking stay the same when only the threshold changes.
• A fixed model’s threshold-independent ranking summaries, such as ROC-AUC and average precision, do not change just because you select a different operating cutoff.
• The confusion matrix, precision, recall, F1-score, and predicted class labels can change.

How should the threshold be chosen?

Evaluate candidate thresholds on a validation set, considering the actual costs of false positives versus false negatives. Fix the chosen threshold before final evaluation on untouched test data. Avoid tuning it on the test set.

In short: changing the threshold trades off *how many positives are caught* against *how many negatives are mistakenly flagged*. Select the operating point that fits the application's objectives, not automatically 0.50.

References

• scikit-learn: Tuning the decision threshold for class prediction
• scikit-learn: Precision-recall curve