The same approach used to calculate a confidence interval for the effect size or statistical power of a t-test (see Confidence Intervals for Power and Effect Size of t Tests) can be employed to create a confidence interval for the noncentrality parameter and in turn the RMSSE effect size and statistical power, of one-factor ANOVA.
Example 1: Find the 95% confidence interval for RMSSE and power of the one-way ANOVA of Example 1 of Power of One-way ANOVA.
The analysis is shown in Figure 1 (with references to cells in Figure 1 of Power of One-way ANOVA).
Figure 1 – Confidence intervals for RMSSE and statistical power
We see from Figure 1 that the 95% confidence interval for the noncentrality parameter is (0, 23.68). The corresponding confidence interval for the RMSSE of .62 is (0, .99) and the confidence interval for power of 65.3% is (5%, 97.8%).
Even though this test results in a relatively high effect size (.62) and power of 65.3%, because the sample is very small, the confidence intervals are quite wide, and so if the test is repeated with similar-sized data it is not so unlikely that the resulting effect size and power can be near zero. This reinforces what we should already know, namely that caution needs to be applied when running statistical tests, especially with small sample sizes.
Hi Charles,
I have done meta-analyses. When I want to compare Cohen’s d from 2 metaanalyses, I have an Excel formula for the t-test based only on the effect sizes d1, d2 and n1, n2 (the number of participants in the 2 meta-analyses). I get a t-value which I can check for significance in an online source.
Now, what if I want to compare the d’s from 3 or more meta-analyses, I think I the t-test is not appropriate, and I have to do an ANOVA. Do you have an Excel formula that could be used for this problem, I mean, when I only have the d’s and the n’s ? Also, after doing the ANOVA, I will have to do post-hoc tests, to see which pairs of d’s differ significantly.
Kind regards, Borwin
Hello Borwin,
Sorry, but I don’t have such an Excel formula. I imagine that the approach is similar, namely that you need to calculate the F statistic in terms of d’s and n’s. I haven’t tried to do it, but you need to expand F to MSBet/MSW and then expand MSBet and MSW, which will principally depend on SSBet and SSW. Then hopefully you can find a way to express SSBet and SSW in terms of the d’s and n’s.
Charles