How do you evaluate a multilabel classification model?

Evaluate a multilabel classifier by checking both complete label sets and individual label decisions. Unlike multiclass classification, where each example belongs to one class, a multilabel example can have several correct labels simultaneously.

Beginner-friendly example

Imagine three photos, each potentially labeled Dog, Outdoor, and Person. Every label is independently either present (1) or absent (0).

• Photo 1: actual {Dog, Outdoor}; predicted {Dog}. The prediction misses Outdoor.
• Photo 2: actual {Outdoor, Person}; predicted {Outdoor, Person}. Perfect match.
• Photo 3: actual {Dog}; predicted {Dog, Person}. The prediction wrongly adds Person.

Each photo has three label decisions, giving 3 × 3 = 9 decisions in total.

1. Subset accuracy (exact-match accuracy)

A photo counts as correct only when its entire predicted label set exactly matches the actual set.

• Only Photo 2 is an exact match.
• Subset accuracy = 1/3 ≈ 33.3%.
• This is deliberately strict: one extra or missing label makes an entire photo incorrect.

2. Hamming loss

Hamming loss measures the fraction of individual label decisions that are wrong.

• Photo 1 has one missing label; Photo 3 has one extra label; Photo 2 has zero errors.
• Hamming loss = 2/9 ≈ 22.2%. Lower is better; 0 means no label-level mistakes.
• Hamming loss is more forgiving than subset accuracy because it counts errors per label, not per whole photo.

3. Multilabel F1-score

F1 balances precision and recall. In this example, across all photo-label decisions:

• True positives = 4, false positives = 1, false negatives = 1.
• Micro precision = 4/5 = 80%.
• Micro recall = 4/5 = 80%.
• Micro F1 = 80%.

For multilabel tasks, the choice of F1 averaging matters:

• Micro F1: pools label decisions across the dataset.
• Macro F1: averages the separate F1 of each label equally.
• Samples F1: calculates an F1 for each photo's predicted-versus-actual label sets and then averages across photos. In this example: (2/3 + 1 + 2/3)/3 ≈ 77.8%.

Do not assume micro F1 equals subset accuracy in multilabel classification.

Python example

PYTHON
from sklearn.metrics import accuracy_score, hamming_loss, f1_score

# Columns: Dog, Outdoor, Person
y_true = [[1,1,0], [0,1,1], [1,0,0]]
y_pred = [[1,0,0], [0,1,1], [1,0,1]]

print("Subset accuracy:", accuracy_score(y_true, y_pred))  # 0.3333
print("Hamming loss:", hamming_loss(y_true, y_pred))      # 0.2222
print("Micro F1:", f1_score(y_true, y_pred, average="micro"))  # 0.8
print("Samples F1:", f1_score(y_true, y_pred, average="samples"))  # 0.7778

Choosing a metric

• Use subset accuracy when a complete set of labels must be perfect.
• Use Hamming loss when each missed or extra label matters independently.
• Use micro F1 for an overall label-decision summary and macro/per-label F1 to expose weak performance on uncommon labels.
• Always show per-label support, evaluate on representative held-out data, and inspect whether your score changes when prediction thresholds change.
• Watch out for examples with no positive labels, where some per-sample precision/recall/F1 expressions require a defined zero-division convention.

Key takeaway: Multilabel evaluation is not one number. Exact-match accuracy measures perfect sets; Hamming loss measures individual mistakes; F1 measures precision-recall balance.

References

• scikit-learn model evaluation
• scikit-learn hamming_loss