Bayes' Theorem is a fundamental rule in probability theory that describes how to update the probability of a hypothesis as more evidence or information becomes available. It provides a mathematical connection between a prior belief and observed data, producing an updated posterior belief.
In simple terms, Bayes' theorem gives us a rigorous mathematical framework for learning from evidence.
Imagine that a patient is tested for a rare disease. Suppose the disease affects 1 in 10,000 people, the test has 99% sensitivity, and 99% specificity. What is the probability that a person who tests positive actually has the disease?
It is tempting to answer "99%", but that would be incorrect. The answer depends not only on the performance of the test, but also on the base rate — how common the disease is in the population.
If we test 10,000 people, approximately 1 person has the disease. That person has a 99% chance of testing positive, giving about 0.99 true positives.
Among the 9,999 healthy people, a 1% false-positive rate produces about 99.99 false positives.
Therefore, among all positive results, only a small fraction correspond to people who actually have the disease:
\[ P(\text{Disease} \mid \text{Positive}) = \frac{ P(\text{Positive} \mid \text{Disease})P(\text{Disease}) }{ P(\text{Positive}) } \approx 0.0098 \]
So the probability that a person who tests positive actually has the disease is only about 0.98%.
This is a classic illustration of the importance of the base rate. It also demonstrates why we must distinguish between:
\[ P(\text{Positive} \mid \text{Disease}) \]
and:
\[ P(\text{Disease} \mid \text{Positive}) \]
These are different probabilities. Bayes' theorem provides the mathematical connection between them.