What is Naive Bayes?

Naive Bayes is a supervised probabilistic classifier based on Bayes' theorem. It is called *naive* because it assumes features are conditionally independent given the class.

How it works

• Estimate class priors, such as P(Spam).
• Estimate feature likelihoods, such as P(free | Spam).
• For a new example, multiply each class prior by the likelihoods for its features.
• Predict the class with the highest score.

Core formula

P(Class | Features) = P(Features | Class) * P(Class) / P(Features)
Class score = P(Class) * product(P(feature_i | Class))

The denominator is the same for all candidate classes for one observation, so it can be omitted when selecting the largest score. Implementations often sum log-probabilities instead of multiplying tiny numbers.

Beginner-friendly example

A simplified email classifier uses binary indicators for the words free and win.

• Prior probabilities: Spam = 0.4; Not spam = 0.6.
• In spam emails: P(free=1)=0.8; P(win=1)=0.7.
• In non-spam emails: P(free=1)=0.1; P(win=1)=0.2.

For an email containing both words:

Spam score     = 0.4 * 0.8 * 0.7 = 0.224
Not-spam score = 0.6 * 0.1 * 0.2 = 0.012

Since 0.224 is larger, the prediction is Spam. Under this simplified model, the normalized estimate is 0.224 / (0.224 + 0.012) = 94.9%. This is a model estimate, not a guarantee of well-calibrated probability.

Main variants

• Gaussian Naive Bayes: for continuous features modeled with Gaussian distributions.
• Multinomial Naive Bayes: for nonnegative count-like features, often document word counts.
• Bernoulli Naive Bayes: for binary presence/absence features.

Strengths and limitations

Naive Bayes is fast to train and predict, often works well in text classification, and is a useful small-data baseline. Its conditional-independence assumption can be unrealistic; correlated features may produce poorly calibrated probabilities. Smoothing helps prevent zero likelihoods for unobserved features.

In short: Naive Bayes classifies by combining prior class frequencies with the likelihood of the observed features in each class.

Reference

• scikit-learn: Naive Bayes