What is ROC-AUC?

ROC-AUC (Area Under the Receiver Operating Characteristic Curve) summarizes how well a binary classifier ranks positive cases above negative cases across decision thresholds. A larger area generally indicates better class discrimination.

What does ROC-AUC measure?

An ROC curve plots true positive rate (recall) against false positive rate while varying the classification threshold. ROC-AUC is the area under that curve, typically between 0 and 1.

• 1.0: perfect ranking of positive cases above negative cases.
• 0.5: chance-level ranking (no discrimination).
• Below 0.5: positive cases tend to receive lower scores than negative cases; this is worse than chance with the current score direction.

For binary classification, ROC-AUC also has a useful ranking interpretation: the probability that a randomly selected positive receives a higher model score than a randomly selected negative, with ties counting as half.

Beginner-friendly numerical example

Suppose a fraud model assigns these risk scores:

• Fraud (positive): 0.90 and 0.60.
• Legitimate (negative): 0.70 and 0.20.

Compare each positive against each negative—there are 2 × 2 = 4 pairs:

• 0.90 > 0.70 → correctly ranked.
• 0.90 > 0.20 → correctly ranked.
• 0.60 < 0.70 → incorrectly ranked.
• 0.60 > 0.20 → correctly ranked.

Three of four pairs are correctly ordered:

ROC-AUC = 3 / 4 = 0.75

Interpretation: On these observations, a randomly selected fraudulent transaction receives a higher risk score than a randomly selected legitimate transaction 75% of the time (counting tied scores as half a win).

Python example

PYTHON
from sklearn.metrics import roc_auc_score

# 1 = fraud, 0 = legitimate
y_true = [1, 1, 0, 0]
scores = [0.90, 0.60, 0.70, 0.20]

print(roc_auc_score(y_true, scores))
# 0.75

Use continuous scores such as positive-class probabilities or decision-function values. Passing only final binary predictions discards most ranking information.

When is ROC-AUC useful?

• Comparing models' discrimination before selecting a classification threshold.
• Evaluating ranking quality when several operating thresholds are plausible.
• Summarizing performance over the full range of false positive rates.

Important limitations

• ROC-AUC does not determine a deployment threshold. A model with good AUC can still perform poorly at the specific false-positive rate your application can tolerate.
• ROC-AUC does not assess probability calibration. A score of 0.9 need not mean a 90% real-world probability.
• Rare positives can require different metrics. On highly imbalanced tasks, investigate precision, recall, alert volume, and precision–recall curves; ROC-AUC can look strong even when positive predictions produce many false alarms.
• Both classes are required for binary ROC-AUC. Multiclass variants exist but require a specified aggregation strategy, such as one-vs-rest or one-vs-one.

In short: ROC-AUC evaluates how well a classifier ranks positives above negatives across thresholds, not how accurate its final labels are at one threshold.

References

• scikit-learn: roc_auc_score
• Google Machine Learning Crash Course: ROC and AUC