How do you choose classification thresholds for multilabel models?

A multilabel classifier can predict several labels for the same example. It often outputs a separate score for each label; thresholds turn these scores into yes/no decisions. Use one shared threshold as a simple starting point, or tune separate thresholds when labels have different frequencies, score distributions, or error costs.

Beginner-friendly example: shared vs per-label thresholds

A photo receives model scores for Animal = 0.72, Outdoor = 0.43, Person = 0.28.

• Shared threshold = 0.50: predict {Animal}, because only 0.72 reaches 0.50.
• Per-label thresholds: Animal = 0.70, Outdoor = 0.40, Person = 0.30. Predict {Animal, Outdoor}; Person stays absent.
• If the actual labels are {Animal, Outdoor}, the per-label thresholds get the full set right while the shared threshold misses Outdoor. One example does not prove the customized thresholds are better; compare across held-out examples.

How to select thresholds without leakage

1. Split the dataset into training, validation, and untouched test partitions (respect groups or time order where needed).
2. Train the model only on training data. Use an initial shared threshold such as 0.50 as a baseline, not an automatic optimum.
3. Collect validation scores for every label. Try candidate thresholds (for example 0.20, 0.30, …, 0.80), and evaluate against actual validation labels.
4. Choose the objective beforehand: for example macro F1, samples F1, a minimum recall requirement for critical labels, or a cost-sensitive utility. A label-by-label F1 optimization may not maximize an aggregate sample-level objective.
5. Choose shared or per-label thresholds. Separate thresholds are useful when label score distributions differ; consider shrinkage or a common threshold for labels with few positives to avoid overfitting.
6. Freeze thresholds before test evaluation. Report the final scores on the untouched test split and monitor them after deployment.

Precision–recall tradeoff

• Lowering a label's threshold usually produces more positive predictions: it can recover missed positives (higher recall), but may increase false alarms and lower precision.
• Raising a label's threshold typically reduces alerts: it may improve precision but miss more positives and reduce recall.
• The exact movement of precision is not guaranteed to be monotonic for every finite dataset. Inspect the actual precision–recall curve.

Small Python example

PYTHON
import numpy as np
scores = np.array([[0.72, 0.43, 0.28]])
labels = np.array(["Animal", "Outdoor", "Person"])

shared = np.array([0.50, 0.50, 0.50])
per_label = np.array([0.70, 0.40, 0.30])
print(labels[(scores[0] >= shared)])      # ['Animal']
print(labels[(scores[0] >= per_label)])   # ['Animal' 'Outdoor']

Practical cautions

• Avoid leakage: selecting thresholds on test data makes the final test score optimistic. Repeated validation tuning can also overfit; use cross-validation or a second validation set for large searches.
• Data imbalance: a rare label may have too few positive validation examples for a stable individually optimized threshold.
• Calibration: probabilities can be miscalibrated; check calibration when interpreting scores as probabilities or using probability-based decisions.
• Changing prevalence and costs: re-evaluate thresholds when data or business requirements shift.
• Correlated labels: choosing each threshold in isolation may ignore useful constraints between labels.

Takeaway: Start with a shared threshold, tune on validation data for a clearly chosen objective, consider per-label thresholds when supported by enough data, and test only after freezing all choices.

References

• scikit-learn precision_recall_curve
• scikit-learn TunedThresholdClassifierCV — binary estimator threshold tuning; multilabel-specific tuning requires applying the approach per label or implementing a suitable multilabel objective.