Bayesian Two-Sided Testing Examples

Objective

In Bayesian Two-Sided Testing we describe how to perform the equivalent of a two-tailed t-test using a Bayesian framework. On this webpage, we provide examples in Excel.

One-sample examples

Example 1: Test the null and alternative hypotheses for the revised data for Example 2 of Bayesian One-sided Hypothesis Testing (repeated in range G3:G14 of Figure 1 below) using a scaled information prior with r = .707.

H0: μ = 0

H1: μ ≠ 0

The left side of Figure 1 shows the calculation of BF01, BF10, P(H0|X), and P(H1|X) based on the values n, t, and r shown in range B2:B4. If you supply these values, then you can perform the analysis.

The right side of Figure 1 shows how to calculate n and t based on the data in column G.

One sample hypothesis test

Figure 1 – One sample test using scaled information prior

We see that the alternative hypothesis is favored with BF10 = 2.09, which is barely worth mentioning. The probability that μ = 0 is about 1/3 and the probability that μ ≠ 0 is about 2/3.

Sensitivity Analysis

It is useful to see how sensitive these results are to the original choice of the scale parameter r. We can create a chart to evaluate this by using Excel’s Data Table capability. We enter values for r between 0 and 3 as shown in range N3:N33 of Figure 2 (the middle values are not displayed).

Data Table results

Figure 2 – Data Table

You then insert the formula =B15 in cell O2 (pointing to the BF10 cell in Figure 1), highlight the range N2:O33 and select Data > Forecast|What-If Analysis > DataTable… from the ribbon. You then insert B4 (which points to the r cell in Figure 1) in the Column input cell field of the dialog box that appears. Excel now fills in column O with the BF10 values that correspond to each of the r values.

Next, we highlight range N3:O33 and select Input > Charts|Scatter > Scatter with Smooth Lines and Markers from the ribbon. This results in a chart similar to that shown in Figure 3.

Sensitivity analysis

Figure 3 – Sensitivity Analysis

We see that the curve is fairly flat around r = .707, obtaining a peak at about r = .538 with BF10 = 2.175. We also see that if we make r too high then the test favors the null even though it is more likely that the alternative hypothesis holds.

Furthermore, we see that if we had selected r = 1, then BF10 = 1.79, which still favors the alternative hypothesis.

JZS Prior

Example 2: Repeat Example 1 using the JZS prior.

The results are shown in Figure 4.

One-sample test (JZS Prior)

Figure 4 – One sample test using JZS prior

The formula in cell B15 is

=1/SQRT(1+B$2*B$12*B14)*(1+B$7/(B$6*(1+B$2*B$12*B14)))^B$9*B$11*B14^(-3/2)*EXP(-1/(2*B14))

Once again, we can use Excel’s Data Table capability and a scatter chart to obtain the results in Figures 5 and 6.

Data Table

Figure 5 – Data Table (JZS Priors)

Sensitivity analysis

Figure 6 – Sensitivity Analysis (JZS Priors)

Paired-sample examples

Example 3: Test the null and alternative hypotheses for the paired data shown in range G2:H14 of Figure 7 using a JZS prior with r = .707.

As usual, this is equivalent to a one-sample test on the difference between the paired values. The analysis is shown in Figure 7 and is equivalent to the analysis shown in Figure 4.

Bayesian paired sample test

Figure 7 – Paired sample test using JZS prior

Two independent samples examples

Example 4: Test the null and alternative hypotheses for two independent samples using a scaled information prior with r = .707 based on the data in range G2:H14 of Figure 8.

H0: μ1 = μ2 (equivalent to δ = 0)

H1: μ1 ≠ μ2 (equivalent to δ ≠ 0)

where δ = (μ2 – μ1)/σ.

The analysis is shown in Figure 8.

Two-sample Bayesian test

Figure 8 – Two sample test using scaled information prior

Note that despite the wide difference between the means for X and Y, the BF10 = 2.018 is barely worth mentioning. In the frequentist approach, p-value = .06 which is not significant.

The sensitivity analysis is shown in Figure 9.

Bayesian sensitivity analysis

Figure 9 – Sensitivity Analysis

JZS Prior

Example 5: Repeat Example 4 using the JZS prior.

The results are shown in Figures 10 and 11.

Two-sample test JZS prior

Figure 10 – Two sample test using JZS prior

Sensitivity analysis JZS prior

Figure 11 – Sensitivity Analysis (JZS Prior)

References

Stack Exchange (2022) Understanding the Jeffreys-Zellner-Siow (JZS) prior in Bayesian t-tests
https://stats.stackexchange.com/questions/570605/understanding-the-jeffreys-zellner-siow-jzs-prior-in-bayesian-t-tests

Rouder, J. N. et al. (2009) Bayesian t tests for accepting and rejecting the null hypothesis
https://www.researchgate.net/publication/24207221_Bayesian_t_test_for_accepting_and_rejecting_the_null_hypothesis

Gonen, M., Lu, Y., Johnson, W. O., Westfall, P. H. (2005) The Bayesian two-sample t-test
https://core.ac.uk/download/pdf/61318655.pdf

Du, H., Edwards, M. C., Zhang, Z. (2019) Bayes factor in one-sample tests of means with a sensitivity analysis: a discussion of separate prior distributions
https://link.springer.com/content/pdf/10.3758/s13428-019-01262-w.pdf

Faukenberry, T. (2023) Bayesian t-tests
https://www.youtube.com/watch?v=D4RJk3SmhUY

Leave a Comment