Basic Concepts
Another approach for determining which pairwise groups are significantly different following ANOVA is to use multiple t-tests followed by one of the following tests to deal with familywise error: Bonferroni, Dunn-Sidàk, Holm’s, Hochberg, Benjamini-Hochberg or Benjamini-Yekutieli. This topic is explored in Multiple Tests.
Example
Example 1: Determine which of the four teaching methods are significantly different based on the data on the left side of Figure 1.
Figure 1 – Testing differences in teaching methods
From the ANOVA you can see that there is a significant difference between the methods. Next, use the Single Factor Anova data analysis tool to determine which specific pairs of methods show a significant difference. In particular, when the dialog box shown in Figure 1 of ANOVA Analysis Tool appears, fill in the Input Range with A3:D11, make sure that the Column headings included with data option is checked, and choose the Pairwise t-tests option. After clicking on the OK button, the output shown on the left side of Figure 1 is displayed.
Figure 2 – Pairwise t-tests
Multiple Test Analysis
You can then use the Multiple Tests data analysis tool, as described in Multiple Tests Analysis Tool, to perform the Benjamini-Hochberg (or any of the other tests listed above).
More specifically, press Ctrl-m and choose the Multiple Tests option (found on the Misc tab when using the Multipage user interface). Fill in the dialog box that appears as shown in Figure 3. It is assumed that the data takes the form of a three-column range with the first and second columns consisting of group names and the third column the corresponding p-values.
Figure 3 – Multiple Tests dialog box
The result displayed on the right side of Figure 2 shows there is a significant difference between Method 1 and 2 as well as Method 1 and 3, but none of the others.
Examples Workbook
Click here to download the Excel workbook with the examples described on this webpage.
References
Howell, D. C. (2010) Statistical methods for psychology (7th ed.). Wadsworth, Cengage Learning.
https://labs.la.utexas.edu/gilden/files/2016/05/Statistics-Text.pdf
Wikipedia (2018) False discovery rate
https://en.wikipedia.org/wiki/False_discovery_rate