What is sample-averaged F1-score in multilabel classification?

Sample-averaged F1 scores each example's *complete predicted label set* separately, then takes the arithmetic mean across examples. It is useful when the quality of each person's document, image, or other individual prediction matters.

Formula

For sample i, count TPᵢ (correctly predicted positive labels), FPᵢ (extra positive labels), and FNᵢ (missing positive labels).

• F1ᵢ = 2TPᵢ / (2TPᵢ + FPᵢ + FNᵢ) when the denominator is positive.
• Samples F1 = (F1₁ + ... + F1ₙ) / n.

Beginner-friendly example

An image model can attach any of Cat, Outdoor, Person to each photo:

• Photo 1: actual {Cat, Outdoor}, predicted {Cat, Person}. TP=1, FP=1, FN=1, so F1=2/(2+1+1)=0.50.
• Photo 2: actual {Person}, predicted {Person}. TP=1, FP=0, FN=0, so F1=2/(2+0+0)=1.00.
• Photo 3: actual {Cat, Person}, predicted {Cat}. TP=1, FP=0, FN=1, so F1=2/(2+0+1)=0.667.

Now average the three sample scores: (0.50+1.00+0.667)/3 ≈ 0.722 (72.2%).

What makes it different?

• Samples F1: calculates an F1 for each individual example, then averages; a photo with many labels does not automatically count more than another photo.
• Micro F1: pools label-level TP, FP, and FN across all samples before calculating F1; examples with more positive-label decisions can have greater influence.
• Macro F1: calculates F1 separately for each label, then averages over labels; each label has equal weight.

For the example, pooled TP=3, FP=1, FN=2, giving micro F1=6/9≈66.7%, different from samples F1≈72.2%.

Python

PYTHON
from sklearn.metrics import f1_score

# Columns: Cat, Outdoor, Person
y_true = [[1,1,0], [0,0,1], [1,0,1]]
y_pred = [[1,0,1], [0,0,1], [1,0,0]]

print(f1_score(y_true, y_pred, average="samples"))  # 0.722222...
print(f1_score(y_true, y_pred, average="micro"))    # 0.666666...

Empty-label edge cases

• If a sample has no actual positives and no predicted positives, TP=FP=FN=0; its F1 is mathematically undefined, not intrinsically perfect. scikit-learn defaults to reporting 0 and raising an UndefinedMetricWarning; choose zero_division=1 if your evaluation convention counts a correct empty-set prediction as 1, and explain that choice.
• If actual positives exist but the model predicts none, F1 is 0. If it predicts positives but none are actually present, F1 is also 0.
• Compare systems using the same zero-division rule, especially when many examples have no labels.

Applications and limitations

• Good for tagging or recommendation tasks where the completeness and correctness of each example's label set matters.
• It can hide poor performance on rare labels; inspect per-label scores and support.
• It does not account for unequal harm from different missing or extra labels.

Takeaway: Samples F1 is an *average of per-example F1 scores*, not an F1 computed from pooled counts.

Reference: scikit-learn f1_score.