Bayes' Theorem

In one sentence: Bayes’ theorem tells you how much to change your belief in something after you see new evidence.

$P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}$

Read it out loud: “The probability of A given B equals the probability of B given A, times the probability of A, divided by the probability of B.”

What each symbol means

Symbol	Meaning
$P(A \mid B)$	The probability that $A$ is true once you know that $B$ happened. This is the updated belief, the answer you want.
$P(B \mid A)$	The probability of seeing the evidence $B$ if $A$ is true. It is often the number that is easiest to measure.
$P(A)$	The prior: the probability of $A$ before you see any evidence.
$P(B)$	The overall probability of seeing the evidence $B$ , counting every way it can happen.
$A$	The thing you want to know about, such as a person being sick.
$B$	The evidence you have recently observed, such as a positive test result.

What the formula is doing

Bayes’ theorem connects two conditional probabilities that are easy to mix up. $P(B \mid A)$ asks: if $A$ is true, how likely is this evidence? $P(A \mid B)$ asks the reverse: now that we have the evidence, how likely is $A$ ? These are different numbers, and mixing them up leads to bad decisions.

The formula starts from the prior $P(A)$ , your belief before the evidence. It multiplies that by $P(B \mid A)$ , which measures how well the evidence fits $A$ . Then it divides by $P(B)$ , the overall chance of seeing that evidence at all. The division keeps the result a proper probability between 0 and 1.

The division by $P(B)$ does important work. It counts the evidence coming from every source, not only from $A$ . When the evidence is common in general, $P(B)$ is large and the update is small. When the evidence is rare, the update can be large.

Bayes’ theorem is a rule for updating beliefs. Once you have the updated belief, it can serve as the prior for the next piece of evidence, so a belief can be refined step by step as new information arrives.

An everyday picture

Think of a detective weighing a clue. A muddy boot print points more strongly to a suspect if muddy boots are rare in town. If everyone owns muddy boots, the print tells the detective very little. Bayes’ theorem weighs the clue against how common it is overall, not only against how well it fits the suspect.

Worked example: A medical test for a rare disease

Let $A$ be the event that a person is sick and $B$ be the event that the test is positive. Suppose 1% of people are sick, so $P(A) = 0.01$ . The test is positive for 90% of sick people, so $P(B \mid A) = 0.9$ . It is also positive for 9% of healthy people, so the false positive rate is $0.09$ . A random person tests positive. What is the chance they are sick?

Find the overall chance of a positive test, $P(B)$ . Add the two ways to test positive: being sick and testing positive, and being healthy and testing positive. $P(B) = 0.9 \times 0.01 + 0.09 \times 0.99 = 0.009 + 0.0891 = 0.0981$ .
Apply Bayes’ theorem: $P(A \mid B) = \frac{0.9 \times 0.01}{0.0981} = \frac{0.009}{0.0981} \approx 0.092$ .

Answer: Even after a positive result, the chance of being sick is only about 9.2%. The test is good, but the disease is rare, so most positive results come from the much larger group of healthy people.

The numbers in this example are recomputed by script every time the site is built.

Common mistakes

Swapping $P(A \mid B)$ and $P(B \mid A)$ . A test that catches 90% of sick people does not mean a positive result is 90% likely to be sick.
Forgetting the prior. Ignoring how rare the disease is makes a positive test look far more conclusive than it is.
Leaving the false positives out of $P(B)$ . The denominator must count every way the evidence can show up, including healthy people who test positive.
Reading a probability of 9.2% as a verdict on one person. It describes a group of cases that look like this one, not a certain answer.

Where you will meet it

Medical testing, where a test result and the disease rate are combined into the chance that a patient is actually sick.
Naive Bayes spam filters, which estimate how likely an email is to be spam given the words it contains.
Robot localization, where a robot updates its guess of its own position with each new sensor reading.
Bayesian statistics, where a prior belief is updated with data to give a posterior belief.

Questions people ask

What is a prior?

The prior is your starting belief before the new evidence. It can come from past data, expert knowledge or a rough estimate. The updated belief after the evidence is called the posterior.

What does naive mean in naive Bayes?

It assumes the features, such as the words in an email, are independent of each other once the class is known. That assumption is rarely exactly true in real text, but the method still works well for spam filtering.

Why can P(B) be hard to compute?

When the evidence can come from many different causes, adding up every way it can happen takes work. Many methods avoid computing it directly by comparing the ratios of the possible answers.

Is Bayes' theorem only for medical tests?

No. It works for any question where a belief is updated with evidence, such as spam filters or robot localization.

Cross-Entropy Loss: Cross-entropy loss gives a small score when the model was confident in the right answer and a big score when it was not.
Softmax: Softmax takes a list of scores, any numbers at all, and turns them into percentages that add up to 100%.
Shannon Entropy: Shannon entropy is one number, in bits, that says how unpredictable a source is. The more predictable it is, the less space its messages need.