How Bayesian Inference Solves the Classic Coin Toss Problem

Single‑Parameter Inference

Probability measures uncertainty of parameters; Bayes’ theorem updates these probabilities when new data are observed to reduce uncertainty.

We start with a simple example, inferring a single unknown parameter to learn Bayesian statistics.

The Coin‑Toss Problem

Flipping a coin is a classic statistical problem described as follows:

We repeatedly flip a coin, record the number of heads and tails, and answer questions such as whether the coin is fair and how biased it might be.

The coin‑toss example is ideal for learning Bayesian statistics because it is familiar and the model is simple, allowing easy computation. Moreover, many real‑world problems involve two mutually exclusive outcomes (0/1, positive/negative, spam/ham, safe/unsafe, healthy/unhealthy), so the same model applies.

To estimate the coin’s bias using Bayesian methods, we need data and a probability model. Assuming we have performed a number of flips and recorded the heads count, the data are ready; the next step is to specify the model.

General Model

We abstract bias as a parameter θ, where θ = 1 means the coin always lands heads, θ = 0 means always tails, and θ = 0.5 indicates a fair coin. Let n be the number of flips and k the number of heads. Bayes’ theorem gives:

We must specify a prior p(θ) and a likelihood p(k|θ). We start with the likelihood.

Choosing the Likelihood

Assuming each flip is independent and only two outcomes are possible, a natural likelihood is the binomial distribution:

The binomial distribution describes the probability of k heads in n flips, i.e., the probability of success in n independent trials.

If θ is unknown, we assign it a prior. Next we choose the prior.

Choosing the Prior

We use the beta distribution, the most common prior in Bayesian statistics, with the form:

The beta distribution, like the binomial, involves the gamma function Γ. Its two parameters α and β control the shape, and its support is [0,1], matching the range of θ.

The beta distribution is a conjugate prior for the binomial likelihood, meaning the posterior will also be a beta distribution:

Other conjugate priors exist (e.g., Gaussian for Gaussian likelihood), but the beta–binomial pair is widely used because it yields analytically tractable posteriors.

Computing the Posterior

Recall Bayes’ theorem: posterior ∝ likelihood × prior.

For our problem, we multiply the binomial likelihood by the beta prior, simplify, and obtain a posterior that is again a beta distribution with updated parameters α' = α + k and β' = β + n – k.

The result is a posterior distribution, not a single point estimate; it describes the probability of different bias values given the data. With enough data, posteriors from different priors converge to the same distribution, and in the limit of infinite data the influence of the prior vanishes.

Reference:

Osvaldo Martin, Python Bayesian Analysis