What is Bernoulli Naive Bayes?

Bernoulli Naive Bayes is a probabilistic classifier for binary features—each feature is either 1 (present/yes) or 0 (absent/no). It is often used with word-presence indicators in text classification.

How does it work?

• Convert each input into binary indicators, such as contains_free and contains_meeting.
• Estimate how often each feature is present within each class, along with the class prior.
• For a new input, account for both present features and absent features.
• Predict the class with the highest prior-weighted score.

The important difference

Unlike Multinomial Naive Bayes, which models word counts, Bernoulli Naive Bayes models presence or absence. A missing word matters: if a word is common in a class but absent from a document, the classifier explicitly factors that absence into its score.

Formula

Let p_jc = P(feature_j=1 | class=c) and x_j be either 0 or 1:

Score(class=c) = P(c) * product_j [p_jc^x_j * (1-p_jc)^(1-x_j)]

With Laplace smoothing (alpha=1) for a binary feature:

P(feature=1 | class) = (present_count_in_class + 1)
                       / (number_of_class_examples + 2)

Numerical example: filtering email

Suppose the features are free and meeting. From earlier emails, the learned conditional probabilities are:

• Spam: P(free=1 | Spam)=0.8, P(meeting=1 | Spam)=0.2.
• Normal: P(free=1 | Normal)=0.1, P(meeting=1 | Normal)=0.7.

Consider an email that contains `free` but does not contain `meeting`. With equal class priors of 0.5:

Spam score   = 0.5 * 0.8 * (1 - 0.2) = 0.32
Normal score = 0.5 * 0.1 * (1 - 0.7) = 0.015

Prediction: Spam. Crucially, (1 - 0.2) and (1 - 0.7) explicitly model the absence of `meeting`. Both results are unnormalized scores, not calibrated probabilities.

Python example

PYTHON
from sklearn.naive_bayes import BernoulliNB

# [contains_free, contains_meeting]
X = [[1, 0], [1, 0], [0, 1], [0, 1]]
y = ['Spam', 'Spam', 'Normal', 'Normal']

clf = BernoulliNB(alpha=1.0, binarize=None)
clf.fit(X, y)
print(clf.predict([[1, 0]]))  # ['Spam']

Why `binarize=None`? The example already contains 0/1 values. When working with numeric inputs that are not binary, scikit-learn can binarize them using a threshold; choose that preprocessing intentionally.

When is it useful?

• Short documents, spam filtering, or other classification problems where a yes/no signal is meaningful.
• Large, sparse matrices of binary features, such as whether a word appears in a document.
• A fast baseline before testing more complex classifiers.

Limitations: repeating a word several times gives no additional count information once it is converted to 1. The model also makes the usual conditional-independence assumption. Compare it with MultinomialNB on held-out data rather than assuming either model always wins.

In short: Bernoulli Naive Bayes uses whether each feature is present or absent, while Multinomial Naive Bayes usually focuses on how frequently features occur.

References

• scikit-learn: Bernoulli Naive Bayes
• scikit-learn: BernoulliNB API