Gibbs Sampler Algorithm
As described in Gibbs Sampler Two Sample Binomial, suppose we want a sample from the joint distribution f(θ, φ), but we don’t have access to this distribution directly, although we do have access to the distributions for θ|φ and φ|θ. In this case, we can use the following iteration method, called Gibbs Sampler.
- Start with a guess θ0 (which is any permissible value for θ)
- For each i > 0, get sample values θi ∼ f(θ|φi-1) and φi ∼ f(φ|θi)
Then for sufficiently large i, (θi, φi) ∼ f(θ, φ).
Gibbs Sampler for bivariate normal distribution
Example 1: Create a sample of size 2,000 from a bivariate normal distribution with μ1 = 2, μ2 = 1, σ1 = .5, σ2 = 1.5 and ρ = .6 using Gibbs Sampler.
We create a sample of 2,100 points (xi, yi) where we throw away 100 of the initial values. We start with an initial value for x0 of zero, and then at each iteration, we look for random values such that
The result is shown in Figure 1 where only the first five sample pairs (xi, yi) are displayed and where range B2:B6 shows the target parameters of the bivariate normal distribution.
Figure 1 – Gibbs Sample for bivariate normal distribution
Representative Formulas
Representative formulas from Figure 1 are shown in Figure 2. Range E2:E6 shows the calculated parameter values based on the sample generated in range B109:C2108. We see that these values are fairly close to the values in range B2:B6, as expected.
Figure 2 – Representative formulas from Figure 1
References
Peng, R. D. (2021) Advanced statistical computing
https://bookdown.org/rdpeng/advstatcomp/
https://bookdown.org/rdpeng/advstatcomp/gibbs-sampler.html#example-bivariate-normal-distribution
https://bookdown.org/rdpeng/advstatcomp/gibbs-sampler.html#example-normal-likelihood