What is a multilabel confusion matrix?

A multilabel confusion matrix gives a separate 2×2 confusion matrix for each possible label. Each label is treated as present (positive) or absent (negative), because one sample can contain multiple labels at once.

Worked example: 4 photos, 2 possible tags

For each photo, compare the actual and predicted tag sets:

• Photo 1: actual {Dog, Outdoor}; predicted {Dog, Outdoor}.
• Photo 2: actual {Dog}; predicted {Outdoor}.
• Photo 3: actual {Outdoor}; predicted {Outdoor}.
• Photo 4: actual {}; predicted {Dog}.

Step 1 — Evaluate the Dog label independently

• TP = 1: Photo 1 actually has Dog and predicts Dog.
• FP = 1: Photo 4 predicts Dog but does not contain Dog.
• FN = 1: Photo 2 contains Dog but the model misses it.
• TN = 1: Photo 3 neither contains nor predicts Dog.
• The four counts total 4 photos.

Step 2 — Evaluate the Outdoor label independently

• TP = 2: Photos 1 and 3 contain Outdoor and predict it.
• FP = 1: Photo 2 predicts Outdoor, although it is absent.
• FN = 0: no actual Outdoor tag is missed.
• TN = 1: Photo 4 neither contains nor predicts Outdoor.
• Again, the four counts total 4 photos.

How is this different from a multiclass confusion matrix?

• A single-label multiclass task normally has one K×K matrix showing confusion *between* mutually exclusive classes.
• A multilabel task has K separate 2×2 matrices, one per label. Each matrix counts that label's present/absent decisions independently.
• Consequently, one photo can contribute to several label-specific matrices. Do not add all their counts and interpret the sum as the number of distinct photos.

Python example

PYTHON
from sklearn.metrics import multilabel_confusion_matrix

# Columns: Dog, Outdoor
y_true = [[1, 1], [1, 0], [0, 1], [0, 0]]
y_pred = [[1, 1], [0, 1], [0, 1], [1, 0]]

cms = multilabel_confusion_matrix(y_true, y_pred)
for name, cm in zip(["Dog", "Outdoor"], cms):
    tn, fp, fn, tp = cm.ravel()
    print(name, {"TP": tp, "FP": fp, "FN": fn, "TN": tn})
# Dog: TP=1, FP=1, FN=1, TN=1
# Outdoor: TP=2, FP=1, FN=0, TN=1

Applications and limitations

• Useful for diagnosis: discover which tags are frequently missed or wrongly added, and calculate label-specific precision and recall when needed.
• Watch for rare labels: a label with very few positives may have many TNs but still perform poorly on actual positives. Report support alongside the counts.
• Thresholds matter: changing a label's decision threshold changes its TP/FP/FN/TN counts.
• Does not capture label relationships: separate matrices cannot tell you whether two labels are consistently predicted together incorrectly, or whether a whole label set is exactly right.
• Zero support: if a label never occurs in the evaluation set, interpret its positive-class metrics with care.

Key takeaway: In multilabel evaluation, ask “For *this label*, which photos are true positives, false positives, false negatives, and true negatives?” and repeat for every label.

Reference: scikit-learn multilabel_confusion_matrix.