Understanding Bayes’ Theorem: From Basics to Real-World Applications

Bayes Theorem

We first look at the seemingly simple yet powerful Bayes theorem, which contains everything you need to master Bayesian statistics.

Starting from the multiplication rule in probability theory, we obtain the basic relationship that leads to Bayes theorem. In compact form it can be written as P(H|D) = \frac{P(D|H) P(H)}{P(D)} , where H denotes a hypothesis and D denotes observed data.

This equation shows that the probability of a hypothesis given data ( posterior ) is proportional to the likelihood of the data under the hypothesis multiplied by the prior belief about the hypothesis, normalized by the evidence.

It is important to note that P(H) (the prior) and P(D|H) (the likelihood) are not generally equal. For example, the probability of being a two‑legged animal and the probability of having two legs are different: almost everyone has two legs, but many two‑legged animals are not human.

In Bayesian terms, the prior distribution reflects our knowledge about the parameters before observing data. If we know nothing, a uniform distribution can be used. The likelihood measures how plausible the observed data are under a given set of parameters. The posterior distribution is the result of Bayesian analysis, representing our updated knowledge after seeing the data; it is proportional to the prior multiplied by the likelihood.

The posterior can be viewed as the updated prior for the next round of analysis, making Bayesian methods especially suitable for sequential data processing, such as real‑time weather or satellite data for early disaster warning.

The final component, evidence (also called marginal likelihood), is the normalizing factor obtained by integrating the product of prior and likelihood over all possible parameter values. In practice we often treat it as a constant and focus on the relative values of the posterior.

Prior : the distribution before observing data.

Likelihood : the probability of the observed data given the parameters.

Posterior : the updated distribution after incorporating the data.

Evidence : the normalizing factor that ensures the posterior is a proper probability distribution.

Reference: Osvaldo Martin, "Python Bayesian Analysis".