Bayes' Theorem is a fundamental rule in probability theory that describes how to update the probability of a hypothesis as more evidence or information becomes available. It provides a mathematical connection between a prior belief and observed data, producing an updated posterior belief.
In simple terms, Bayes' theorem gives us a rigorous mathematical framework for learning from evidence.
Imagine that a patient is tested for a rare disease. Suppose the disease affects 1 in 10,000 people, the test has 99% sensitivity, and 99% specificity. What is the probability that a person who tests positive actually has the disease?
It is tempting to answer "99%", but that would be incorrect. The answer depends not only on the performance of the test, but also on the base rate — how common the disease is in the population.
If we test 10,000 people, approximately 1 person has the disease. That person has a 99% chance of testing positive, giving about 0.99 true positives.
Among the 9,999 healthy people, a 1% false-positive rate produces about 99.99 false positives.
Therefore, among all positive results, only a small fraction correspond to people who actually have the disease:
\[ P(\text{Disease} \mid \text{Positive}) = \frac{ P(\text{Positive} \mid \text{Disease})P(\text{Disease}) }{ P(\text{Positive}) } \approx 0.0098 \]
So the probability that a person who tests positive actually has the disease is only about 0.98%.
This is a classic illustration of the importance of the base rate. It also demonstrates why we must distinguish between:
\[ P(\text{Positive} \mid \text{Disease}) \]
and:
\[ P(\text{Disease} \mid \text{Positive}) \]
These are different probabilities. Bayes' theorem provides the mathematical connection between them.
Interactive Bayesian Updating – See the theorem in action
Use the sliders below to adjust your Prior belief about the probability \(p\) that a coin lands heads, and the observed Data (Heads, Tails). The graph instantly updates to show the resulting Posterior distribution.
The example uses a Beta prior and Binomial data. Try changing the prior parameters and then adding more observations to see how the posterior changes.
Figure 1: The blue dashed line is the Prior (initial belief). The green dotted line shows the normalized shape of the Likelihood, while the red solid line is the Posterior (updated belief). Move the sliders to see Bayesian updating in action.
The mathematical formula
Bayes' theorem is expressed as:
\[ P(A|B) = \frac{P(B|A)\cdot P(A)} {P(B)} \]Where:
- P(A|B) – Posterior: the probability of hypothesis A after taking evidence B into account.
- P(B|A) – Likelihood: the probability of observing evidence B assuming that hypothesis A is true.
- P(A) – Prior: the probability assigned to hypothesis A before incorporating the new evidence.
- P(B) – Evidence (or marginal likelihood): the overall probability of observing B, accounting for the possible hypotheses. It acts as the normalizing constant for the posterior.
A useful shorthand is:
\[ \text{Posterior} \propto \text{Likelihood} \times \text{Prior} \]The proportionality symbol is important: the likelihood multiplied by the prior gives an unnormalized posterior. The evidence \(P(B)\) provides the normalization needed to obtain a probability distribution.
Bayesian updating with a coin toss
In the interactive graph above, we use a Beta-Binomial model. The unknown parameter \(p\) represents the probability that the coin lands heads.
We begin with a Beta prior:
\[ p \sim \mathrm{Beta}(\alpha,\beta) \]After observing \(H\) heads and \(T\) tails, the likelihood as a function of \(p\) is proportional to:
\[ L(p|H,T) \propto p^H(1-p)^T \]Notice that the likelihood is not itself a probability distribution over \(p\), so it does not need to integrate to 1.
For the purpose of visualization, the graph displays a normalized version of the likelihood. Its shape is equivalent to a Beta distribution:
\[ \mathrm{Beta}(H+1,T+1) \]Because the Beta distribution is a conjugate prior for the Binomial likelihood, the posterior is also a Beta distribution:
\[ p|H,T \sim \mathrm{Beta}(\alpha+H,\beta+T) \]This is why the Beta-Binomial model is especially useful for building intuition about Bayesian inference: the entire update can be expressed simply by adding the observed numbers of heads and tails to the prior parameters.
Why is Bayes' theorem so useful?
Bayes' theorem provides a formal framework for reasoning under uncertainty and incorporating new evidence.
A Bayesian analysis starts with a prior distribution, combines it with a likelihood representing the observed data, and produces a posterior distribution representing updated uncertainty.
This framework is useful in many areas, including:
- Spam filtering: probabilistic classifiers such as Naïve Bayes can use observed features, such as word frequencies, to estimate whether an email is likely to be spam.
- Medical diagnosis: Bayesian reasoning helps interpret diagnostic evidence while taking disease prevalence and test characteristics into account.
- Machine learning: Bayesian methods are used in areas including Gaussian processes, Bayesian neural networks, and Bayesian optimization.
- Finance and risk assessment: Bayesian models can update estimates of uncertain quantities as new economic or market information becomes available.
- A/B testing: Bayesian methods can update beliefs about competing variants as new observations accumulate.
Bayesian and frequentist statistics
It is common to summarize one distinction between Bayesian and frequentist statistics by saying that Bayesian inference asks about the probability of hypotheses or parameters given the data, whereas frequentist inference focuses on the behavior of statistical procedures and data under fixed, unknown parameters.
This distinction is useful, but Bayesian and frequentist statistics should not be viewed simply as "correct" versus "incorrect" approaches.
They are different frameworks for statistical inference, with different interpretations, assumptions, and strengths. Both are widely used in scientific research and practical data analysis.
Common pitfalls and limitations
- The prior: the choice of a prior can influence the posterior, particularly when little data are available. Priors can be informed by previous research, domain knowledge, or other justified assumptions. Sensitivity analysis can be used to investigate how much conclusions depend on the choice of prior.
- Model assumptions: Bayesian inference depends on the assumed prior and likelihood model. If the model is inappropriate, the resulting posterior may also be misleading.
- Computational complexity: simple conjugate models can often be solved analytically, but more complex Bayesian models may require numerical methods such as MCMC or variational inference.
- Interpreting probability: a posterior probability describes uncertainty under a specified model and set of assumptions. For example, \(P(\text{rain tomorrow}|data)=70\%\) does not mean that it will rain for exactly 70% of tomorrow.
- Garbage in, garbage out: Bayesian methods do not remove the need for good data and appropriate statistical assumptions.
The takeaway
Bayes' theorem is a mathematical framework for updating uncertainty in light of new evidence.
We begin with a prior distribution representing what we know or assume before considering the new data. The likelihood describes how compatible different parameter values or hypotheses are with the observed data. Combining them gives us a posterior distribution representing our updated state of knowledge.
\[ \boxed{ \text{Prior} + \text{Evidence} \rightarrow \text{Posterior} } \]The process can then be repeated. Information accumulated today can become part of the prior information used in a future analysis.
The deeper lesson of Bayesian reasoning is not that our initial beliefs must be correct. It is that we should be willing to update them when the evidence changes.
Your prior is your starting point. The data provide new information. The posterior is your updated understanding.
```