Conjugate Pairs

Basic Concepts

When the prior and posterior distributions are both from the same distribution family then we say that the pair is conjugate. These conjugate pairs are based on some distribution of the data. E.g. we refer to an exponential-gamma system when the data is an independent random sample from an exponential distribution where both the prior and posterior of the exponential distribution parameter are the pdf of a gamma distribution.

When the distribution of the data has multiple parameters, we also need to specify which of the parameters are known (and so can be treated as constants) and which are unknown (and so require prior and posterior distributions).

Exponential-Gamma

Property 1: If the independent sample data X = x1, …, xn follow an exponential distribution (and so xi ≥ 0 for each i) with an unknown rate parameter λ with a prior gamma distribution of λ ∼ Gamma(α, β) then the posterior distribution is λ|X ∼ Gamma(α′, β′)  where

Posterior gamma parameters

Proof: Note first that λ > 0, which is compatible with a gamma distribution. Since the prior follows a gamma distribution, we know that the pdf is

Gamma pdf

Since the likelihood function is the product of exponential pdf functions

Exponential likelihood function

It follows by Bayes Theorem that

Posterior distribution for lambda

Posterior gamma distribution

which completes the proof since this last formula is proportional to the pdf of the specified gamma distribution.

Poisson-Gamma 

Property 2: If the independent sample data X = x1, …, xn follow a Poisson Distribution (and so xi ≥ 0 for each i) with an unknown mean parameter with a prior gamma distribution of λ ∼ Gamma(α, β) then the posterior distribution is λ|X ∼ Gamma(α′, β′)  where

Posterior gamma parameters

and

n times the mean

Proof: We proceed as in the proof of Property 1.

Pdf of lambda

Likelihood

Posterior pdf part 1

Posterior pdf part 2

Since f(λ|X) is proportional to the pdf of Gamma(α′, β′), this completes the proof.

Example 1: As described in Example 2 of Poisson Distribution, radioactive particles hitting a screen follow a Poisson distribution. We observe that 200 radioactive particles hit a screen in the past 300 seconds. Using a prior gamma distribution with a mean of .75 and standard deviation of 1 for lambda, the mean of the Poisson distribution, what is the posterior distribution for lambda?

Since λ ∼ Gamma(α, β), we know that

mean = αβ = .75

variance = αβ2 = 1.0

Thus, 1.0 = αβ2 = .75β, and so β = 4/3 = 1.3333 and α = .75/(4/3) = 9/16 = .5625.

By Property 2, the posterior distribution λ|X ∼ Gamma(α′, β′)  where

Alpha prime

Beta prime

from which the posterior mean and standard deviation can be calculated

Posterior mean

Posterior standard deviation

This means that the estimated value for lambda is now .6669 with a standard deviation much lower than the previous standard deviation of 1.

Binomial-Beta

See Binomial Distribution Analytic Approach                                                         

Leave a Comment