Confidence intervals of effect size and power for regression

The same approach used to calculate a confidence interval for the effect size of a t-test (see Confidence Intervals for Effect Size and Power for t-Tests) can be employed to create a confidence interval for a noncentrality parameter, and in turn the R² effect size and statistical power, for multiple regression.

Example 1: Find the 95% confidence interval for R² and power of the multiple regression in Example 1 of Confidence and Prediction Intervals.

The analysis is shown in Figure 1 (with references to cells in Figure 2 of Confidence and Prediction Intervals).

Figure 1 – Confidence intervals for effect size and power

We see from Figure 1 that the 95% confidence interval for the noncentrality parameter is (5.30, 46.76). The corresponding confidence interval for R² of .338 is (.096, .483) and the confidence interval for power of 98.94% is (43.05%, 99.99%).

Even though this test results in a relatively high effect size (.338) and power (98.94%), because the sample is relatively small, the confidence intervals are relatively wide, and so if the test is repeated with similar-sized data it is not so unlikely that the resulting effect size can be near the lower bound of .096 (which is small) and power can be near the lower bound of 43.05%. This reinforces what we should already know, namely that caution needs to be applied when running statistical tests, especially with small sample sizes.