What is subset accuracy in multilabel classification?

Subset accuracy (exact-match accuracy) measures the fraction of examples for which the predicted set of labels matches the true set perfectly. If even one label is missing or incorrectly added, that whole example counts as incorrect.

Formula

Subset accuracy = number of samples with all labels correct / total number of samples.

For multilabel data, compare each sample's full binary indicator vector (or label set), not individual positions in isolation.

Beginner-friendly step-by-step example

Suppose an image-tagging model assigns any combination of Animal, Outdoor, Person, and Vehicle:

• Image 1: actual {Animal, Outdoor}; predicted {Animal, Outdoor} → exact match.
• Image 2: actual {Person, Vehicle}; predicted {Person} → not a match, since Vehicle is missing.
• Image 3: actual {Animal}; predicted {Animal, Outdoor} → not a match, since Outdoor was added.
• Image 4: actual {Outdoor, Person}; predicted {Outdoor, Person} → exact match.
• Image 5: actual {Vehicle}; predicted {Animal} → not a match.

Step 1: Count perfect label sets: Images 1 and 4 → 2 matches.

Step 2: Count all images: 5.

Step 3: Subset accuracy = 2/5 = 0.40 (40%).

The other three images receive no partial credit, even if some of their tags are correct.

Why does it become strict as label count grows?

If each of L label decisions is independently correct with probability 0.9, then under that simplified independence assumption, the chance of all labels being right is 0.9^L:

• With 2 labels: 0.9² = 81%.
• With 5 labels: 0.9⁵ ≈ 59%.
• With 10 labels: 0.9¹⁰ ≈ 35%.

This is an illustration, not a universal formula: real label errors can be dependent, labels vary in difficulty, and label cardinality matters. Still, the example shows why exact-match accuracy may decline as the number of possible labels increases.

Python example

PYTHON
from sklearn.metrics import accuracy_score

# Indicator order: Animal, Outdoor, Person, Vehicle
y_true = [
    [1,1,0,0], [0,0,1,1], [1,0,0,0],
    [0,1,1,0], [0,0,0,1]
]
y_pred = [
    [1,1,0,0], [0,0,1,0], [1,1,0,0],
    [0,1,1,0], [1,0,0,0]
]

print(accuracy_score(y_true, y_pred))  # 0.4

When is it useful?

• When the entire set must be correct before an automated action can proceed, such as a rule-based multi-tag routing system.
• As a strict quality measure alongside less restrictive label-level metrics.

Limitations

• No partial credit: one incorrect label is treated the same as several incorrect labels for that example.
• Sensitive to label count: comparisons across tasks with different label vocabularies can be misleading.
• Can hide label-specific problems: report per-label metrics and class support to locate weak or rare labels.
• Empty label sets matter: correctly predicting no labels also counts as an exact match; inspect prevalence if empty sets dominate.

Key takeaway: Subset accuracy asks, “How often did the model get the complete label set exactly right?”

Reference: scikit-learn accuracy_score.