Naive Bayes Classifier

In one sentence: Naive Bayes picks the most likely class by multiplying how common each class is by how well each clue fits that class.

$P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)$

Read it out loud: “The probability of class y, given the features x 1 through x n, is proportional to the probability of y, times the product, for i from 1 to n, of the probability of x sub i given y. The vertical bar means given. The proportional sign is read as proportional to, and it means the two sides scale together. The big pi sign means multiply together.”

Before you start

The ideas and signs this page uses, explained from zero. For powers of ten, Greek letters and other notation in general, see Math basics.

Conditional probability and the word given: A conditional probability, written $P(A \mid B)$ , is read “the probability of $A$ given $B$ ”. It is the chance of $A$ once you know that $B$ happened. In $P(y \mid x)$ the vertical bar means “given”, so the formula asks how likely the class is once the clues are known.
Bayes’ theorem, prior and likelihood: Bayes’ theorem flips a question around: from how clues look inside each class, to how likely each class is once the clues are seen. The prior, $P(y)$ , is the share of training examples in class $y$ before any clues are looked at. The likelihood, $P(x_i \mid y)$ , is how often clue $x_i$ appears among the examples of class $y$ .
Independent clues: Two clues are independent if knowing one tells you nothing about the other. Rolling two dice is independent: a 6 on one die does not change the chance of a 6 on the other. Words in an email are usually not independent, because some words tend to appear together, which is why the naive assumption is only an approximation.
The product sign and the proportional sign: The big pi sign, $\prod$ , means multiply together, in the same way that $\sum$ means add up. $\prod_{i=1}^{n} x_i$ equals $x_1 \times x_2 \times \cdots \times x_n$ . The sign $\propto$ means proportional to: two quantities differ only by a constant factor, so their ratio is the same for every class.
Logarithms turn products into sums: The natural logarithm, $\ln$ , is the power of the number $e$ (about 2.718) needed to make a given number. A key rule is $\ln(ab) = \ln a + \ln b$ , so a logarithm turns a product into a sum. Computers use this trick because multiplying many tiny probabilities gives a number too small to store, while adding their logarithms stays in a safe range.
Spam filter and labeled examples: A spam filter is a program that sorts incoming email into spam or not spam. A classifier like this learns from labeled examples, emails that people have already marked as spam or not spam, and then predicts the label of a new email. The clues are usually the words the email contains.

What each symbol means

Symbol	Meaning
$P(y \mid x_1, \dots, x_n)$	The probability of class $y$ after seeing all the clues. It is a number from 0 to 1 with no unit. A value of 0.99 would mean a 99% chance that the email is spam, given the words it contains.
$\propto$	Is proportional to. The two sides differ only by a constant factor that is the same for every class. This means you can compare the class scores without computing that constant, and divide all the scores by their total to get real probabilities.
$y$	One class label, such as spam or not spam. It is a category rather than a number, so it has no unit. The classifier’s job is to pick the value of $y$ that has the largest score.
$x_1, \dots, x_n$	The features: the $n$ clues observed for one example, such as whether the word free appears in an email. Each clue is a fact about the example, often a yes or no or a count. It has no unit.
$n$	The number of features, a count with no unit. An email in which we check three chosen words has $n = 3$ , and the formula multiplies one likelihood for each of those three clues.
$P(y)$	The prior: the share of training examples that belong to class $y$ , before any clues are seen. It is a number from 0 to 1 with no unit. If 40% of the training emails are spam, then $P(\text{spam}) = 0.4$ .
$\prod_{i=1}^{n}$	Multiply together the terms for every $i$ from 1 to $n$ . It has no unit of its own, and it combines one likelihood per clue. Multiplying the three numbers 0.3, 0.1 and 0.5 gives 0.015, for example.
$P(x_i \mid y)$	The likelihood: the probability that clue $x_i$ appears in an example of class $y$ . It is a number from 0 to 1 with no unit. In the made-up example below, $P(\text{free} \mid \text{spam}) = 0.30$ means the word appears in 30% of the spam examples.

The formula piece by piece

$P(y)$: Start with the prior, which is how common the class is in the training data. A common class starts with a higher score, before any clue has been looked at, because it is more likely in general.
$P(x_i \mid y)$: Each clue’s likelihood: how typical clue $x_i$ is for class $y$ . A clue that appears often in class $y$ multiplies the score up, and a clue that rarely appears in class $y$ multiplies the score down.
$\prod_{i=1}^{n} P(x_i \mid y)$: Multiply the likelihoods of all the clues together. Multiplying treats the clues as independent once the class is known, so each clue’s effect is applied one after another, in a single product.
$P(y) \prod_{i=1}^{n} P(x_i \mid y)$: The class score: the prior times all the clue likelihoods. Compare these scores across the classes, and the largest one is the naive Bayes guess. This score is not yet a probability, because it has not been divided by the total.
$P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)$: The probability of a class given the clues is proportional to how common the class is, times how well each clue fits that class, so the class with the largest score wins.

What the formula is doing, and why

Naive Bayes is a classifier built on Bayes’ theorem. For each possible class $y$ , it asks: given the clues I can see, how probable is this class? It then picks the class with the highest score. Training is fast, because it mainly counts how often each clue appears in each class, and the method often works well on text.

Why start with the prior? The formula starts with $P(y)$ , the share of training examples in that class. A class that is common in the training data begins with a higher score, because it is more likely before any clue is seen. Then, for each clue, it multiplies by the likelihood $P(x_i \mid y)$ , which measures how typical that clue is for the class. Clues that fit a class well raise its score, and clues that rarely appear in it lower its score.

Why multiply? Multiplying is how the chances of independent events combine. If the chance of heads on one coin is 0.5, and the chance of heads on a second, separate coin is also 0.5, the chance of both is $0.5 \times 0.5 = 0.25$ . Once the class is known, the clues are treated as independent, so the joint chance of all the clues becomes a product with one term per clue.

Why “naive”? The word refers to the assumption that, once the class is known, the clues are independent of one another. That assumption lets the joint likelihood split into separate terms, one per clue. In real text, words are not independent: free and offer tend to appear together. The assumption is wrong, yet the classifier often still ranks the classes well.

The $\propto$ sign hides a constant. The probability of the clues, $P(x_1, \dots, x_n)$ , is the same for every class, so it can be dropped when you only need to compare scores. To get actual probabilities, divide each class score by the sum of all the class scores. In practice, sums of logarithms replace the products, which avoids numbers too tiny for a computer to store.

An everyday picture

A doctor starts with how common each illness is in the clinic, then adjusts for each symptom. Some symptoms are common with one illness and rare with another. The illness with the highest adjusted score becomes the first guess. Naive Bayes does the same with classes and word clues in an email.

Worked example: Spam or not spam, with made-up word statistics

All numbers here are made up for this example and are not measured from real email. Assume the training emails are 40% spam and 60% not spam. A new email contains two words, free and prize, so $n = 2$ clues. The invented probabilities are $P(\text{free} \mid \text{spam}) = 0.30$ , $P(\text{free} \mid \text{not spam}) = 0.02$ , $P(\text{prize} \mid \text{spam}) = 0.10$ and $P(\text{prize} \mid \text{not spam}) = 0.01$ .

Write the score for spam. Start with the prior and multiply by each clue’s likelihood under spam: $P(\text{spam}) \times P(\text{free} \mid \text{spam}) \times P(\text{prize} \mid \text{spam}) = 0.4 \times 0.30 \times 0.10 = 0.012$ . Multiplying by 0.30 for free raises the score, because the word is fairly common in spam.
Write the score for not spam. Do the same with the not-spam numbers: $0.6 \times 0.02 \times 0.01 = 0.00012$ . The score is far smaller, because both words are rare in normal email, so each multiplication shrinks the score.
Compare the two scores. The spam score, 0.012, is 100 times the not-spam score, 0.00012. Both scores are small numbers, but the comparison is fair because both were built the same way, with a prior and two likelihoods. The larger score is the better fit for the clues.
Add the two scores. The total is $0.012 + 0.00012 = 0.01212$ . This total is what we divide by in the next step, and it is the same for every class, which is why the $\propto$ sign allows us to skip it while comparing.
Divide each score by the total. $P(\text{spam} \mid \text{words}) = \frac{0.012}{0.01212} \approx 0.990$ , and $P(\text{not spam} \mid \text{words}) = \frac{0.00012}{0.01212} \approx 0.0099$ . The two numbers add to 1, so they are real probabilities of the two classes, as the rule requires.
Read the answer. The class with the highest probability is spam, so the filter labels this email as spam. The answer depends entirely on the made-up numbers, so it shows the method, not a real result.

Answer: Under these invented numbers, the filter gives the email about a 99% chance of being spam. Both words are far more common in the made-up spam examples, so the spam score wins easily, even though spam is the smaller class in the training data.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Forgetting the prior. Ignoring how common each class is throws away information about the base rates.
Letting one unseen word wipe out a class. If a word never appears in the training spam, its probability is 0 and the whole product becomes 0. Smoothing, such as adding a small count to every word, avoids this.
Multiplying many small probabilities directly. Long emails produce numbers too small for a computer to store accurately. Add logarithms instead of multiplying.
Reading the raw class scores as probabilities. The scores must be divided by their total before they add up to 1.

Where you will meet it

Email spam filtering, the classic early use of this method.
Sentiment analysis of short reviews, as a fast baseline before trying heavier models.
Sorting news articles or support tickets into topics, using the words each message contains.
Real-time text classification on small devices, because prediction needs only a few lookups and multiplications.

Questions people ask

Why is it called naive?

Because it assumes the clues are independent once the class is known. That assumption is often false, for example when words tend to appear together, but the method is still useful.

How is it different from logistic regression?

Naive Bayes models how each clue appears within each class and then applies Bayes’ theorem. Logistic regression learns weights that separate the classes directly and does not assume independence. Both can give probabilities.

Does it need a lot of training data?

It often works with modest amounts of data, because it estimates the probability of each clue separately. The estimates are still noisy when a class has very few examples.

Why do we not divide by the probability of the features?

That probability is the same for every class. Dividing by it changes all the scores by the same factor, so it does not change which class wins. It is needed only when you want exact probabilities.

Bayes' Theorem: Bayes’ theorem is a recipe for changing how likely something is after you learn a new fact, by comparing how well that fact fits the idea with how common the fact is overall.
Cross-Entropy Loss: Cross-entropy loss gives a small penalty when the model gave the right answer a high probability, and a big penalty when it gave the right answer a small probability.
Softmax: Softmax takes a list of scores, which can be any numbers at all, and turns them into percentages that add up to 100 percent, so the biggest score gets the biggest share.