Bayes' theorem
Bayes' theorem is a result in probability theory, which gives the conditional probability distribution of a random variable A given B in terms of the conditional probability distribution of variable B given A and the marginal probability distribution of A alone.In the context of Bayesian probability theory and statistical inference, the marginal probability distribution of A alone is usually called the prior probability distribution or simply the prior. The conditional distribution of A given the "data" B is called the posterior probability distribution or just the posterior.
As a mathematical theorem, Bayes' theorem is valid regardless of whether one adopts a frequentist or a Bayesian interpretation of probability. However, there is disagreement as to what kinds of variables can be substituted for A and B in the theorem; this topic is treated at greater length in the articles on Bayesian probability and frequentist probability.
Bayes' theorem is named after the Reverend Thomas Bayes (1702–61). Bayes worked on the problem of computing a distribution for the parameter of a binomial distribution (to use modern terminology); his work was edited and presented posthumously (1763) by his friend Richard Price, in An Essay towards solving a Problem in the Doctrine of Chances. Bayes' results were replicated and extended by Laplace in an essay of 1774, who apparently was not aware of Bayes' work.
One of Bayes' results (Proposition 5) gives a simple description of conditional probability, and shows that it does not depend on the order in which things occur:
What is "Bayesian" about Proposition 9 is that Bayes presented it as a probability for the parameter p. That is, not only can one compute probabilities for experimental outcomes, but also for the parameter which governs them, and the same algebra is used to make inferences of either kind. Interestingly, Bayes actually states his question in a way that might make the idea of assigning a probability distribution to a parameter palatable to a frequentist. He supposes that a billiard ball is thrown at random onto a billiard table, and that the probabilities p and q are the probabilities that subsequent billiard balls will fall above or below the first ball. By making the binomial parameter p depend on a random event, he cleverly escapes a philosophical quagmire that he most likely was not even aware was an issue.
Bayes' theorem is a relation among conditional and marginal probabilities. It can be viewed as a means of incorporating information, from an observation, for example, to produce a modified or updated probability distribution. To derive Bayes' theorem, note first from the definition of conditional probability that
Each term in Bayes' theorem has a conventional name.
The term P(A) is called the prior probability of A.
It is "prior" in the sense that it precedes any information about B.
P(A) is also the marginal probability of A.
The term P(A|B) is called the posterior probability of A, given B. It is "posterior" in the sense that it is derived from or entailed by the specified value of B.
The term P(B|A), for a specific value of B, is called the likelihood function for A given B and can also be written as L(A|B).
The term P(B) is the prior or marginal probability of B, and acts as the normalizing constant. With this terminology, the theorem may be paraphrased as
Bayes' theorem is often embellished by noting that
Historical remarks
The main result (Proposition 9 in the essay) derived by Bayes is the following: assuming a uniform distribution for the prior distribution of the binomial parameter p, the probability that p is between two values a and b is
where m is the number of observed successes and n the number of observed failures. His preliminary results, in particular Propositions 3, 4, and 5, imply the result now called Bayes' Theorem (as described below), but it does not appear that Bayes himself emphasized or focused on that result.Statement of Bayes' theorem
denoting by P(A,B) the joint probability of A and B.
Dividing the left- and right-hand sides by P(B), we obtain
which is the theorem conventionally known as Bayes' theorem.Alternative forms of Bayes' theorem
so the theorem can be restated as
where AC is the complementaryary event of A. More generally, where {Ai} forms a partition of the event space,
See also the law of total probability.
There is also a version of Bayes' theorem for continuous distributions.
It is somewhat harder to derive, since probability densities,
strictly speaking, are not probabilities,
so Bayes' theorem has to be established by a limit process;
see Papoulis (citation below), Section 7.3 for an elementary derivation.
Bayes' theorem for probability densities is formally similar to the theorem for probabilities:
Here we have indulged in a conventional abuse of notation,
using f for each one of these terms,
although each one is really a different function;
the functions are distinguished by the names of their arguments.
Theorems analogous to Bayes' theorem hold in problems with more than two variables.
These theorems are not given distinct names,
as they may be mass-produced by applying the laws of probability.
The general strategy is to work with a decomposition of the joint probability, and to marginalize (integrate) over the variables that are not of interest.
Depending on the form of the decomposition,
it may be possible to prove that some integrals must be 1,
and thus they fall out of the decomposition;
exploiting this property can reduce the computations very substantially.
A Bayesian network is essentially a mechanism for automatically generating the extensions of Bayes' theorem that are appropriate for a given decomposition of the joint probability.
Typical examples that use Bayes' theorem assume the philosophy underlying Bayesian probability that uncertainty and degrees of belief can be measured as probabilities. One such example follows. For additional worked out examples, please see the article on the examples of Bayesian inference.
We wish to know about the proportion r of voters who will vote "yes" on a referendum. It is given that n=10 voters have been located at random, and m=7 say they will vote yes. From Bayes' theorem we have
The prior summarizes what we know about the distribution of r in the absence of any observation. We will assume in this case that the prior distribution of r is uniform over the interval [0, 1]. That is, f(r) = 1. That assumption should be considered provisional -- if some additional background information is found, we should modify the prior accordingly.
Under the assumption of random sampling, choosing voters is just like choosing balls from an urn. The likelihood function for such a problem is just the probability of 7 successes in 10 trials for a binomial distribution.
One may be interested in the probability that more than half the voters will vote "yes". The prior probability that more than half the voters will vote "yes" is 1/2, by the symmetry of the uniform distribution. In comparison, the posterior probability that more than half the voters will vote "yes", i.e., the conditional probability given the outcome of the opinion poll -- that seven of the 10 voters questioned will vote "yes" -- is
Bayes' theorem for probability densities
and there is an analogous statement of the law of total probability:
As in the discrete case,
the terms have standard names.
f(x, y) is the joint distribution of X and Y,
f(x|y) is the posterior distribution of X given Y=y,
f(y|x) = L(x|y) is (as a function of x) the likelihood function of X given Y=y,
and f(x) and f(y) are the marginal distributions of X and Y respectively, with f(x) being the prior distribution of X.Extensions of Bayes' theorem
Example
From this we see that once we have in hand the prior f(r) and the likelihood function f(n=10, m=7|r), we can compute the posterior f(r|n=10, m=7).
As with the prior, the likelihood is open to revision -- more complex assumptions will yield more complex likelihood functions. Maintaining the current assumptions, we compute the normalizing factor,
and the posterior distribution for r is then
for r between 0 and 1, inclusive.
which is about an "89% chance".References
Versions of the essay
Commentaries
Additional material
See also