Introduction to Post-Hoc Tests in ANOVA

When an Analysis of Variance (ANOVA) reveals significant differences across groups for an explanatory variable with more than two categories, it does not specify which groups differ from each other. To determine the specific group differences, a post-hoc test is required. Post-hoc, meaning “after the fact,” involves paired comparisons of group means. These comparisons are conducted in a controlled manner to minimize the risk of Type I errors, which occur when the null hypothesis is incorrectly rejected.

Why Not Use Multiple ANOVAs?

Performing multiple ANOVAs to compare all possible group pairs might seem like an intuitive approach. However, each ANOVA carries a 5% chance of committing a Type I error if the significance level is set at p≤0.05p \leq 0.05p≤0.05. Conducting multiple tests inflates this error rate, known as the family-wise error rate, leading to a much higher overall likelihood of rejecting the null hypothesis when it is true. For instance, after 10 tests, the probability of committing a Type I error rises to 40%, making the results less reliable.

Purpose of Post-Hoc Tests

Post-hoc tests are specifically designed to compare pairs of means while controlling for the inflation of Type I errors. They adjust the error rate to ensure that the likelihood of incorrect conclusions remains within acceptable bounds. Several post hoc tests are available, including:

• Sidak Test
• Holm Test
• Fisher’s Least Significant Difference Test
• Tukey’s Honestly Significant Difference Test
• Scheffé Test
• Newman-Keuls Test
• Dunnett’s Multiple Comparison Test
• Duncan Multiple Range Test
• Bonferroni Procedure

While these tests vary in their level of conservatism (how strictly they control for Type I errors), the choice of test is often less critical than ensuring that a post hoc analysis is conducted.

Applying Post-Hoc Analysis: An Example

In examining the relationship between ethnicity and the number of cigarettes smoked per month among young adult smokers, the Duncan Multiple Range Test was used as the post hoc method. This test was applied following an ANOVA with a significant F statistic (F=24.4F = 24.4F=24.4) and p<0.0001p < 0.0001p<0.0001, indicating significant differences among groups.

Interpreting Duncan Post-Hoc Results

The results of the Duncan Test are displayed in a table where group means with the same capital letter next to them are not significantly different. For example:

• Ethnic groups 1 (White) and 3 (American Indian/Alaskan Native) share the letter “A,” indicating no significant difference between them.
• Groups 2 (Black), 4 (Asian), and 5 (Hispanic/Latino) share the letter “C,” meaning they are not significantly different from each other.
• Groups 3, 2, and 4 also share the letter “B,” showing no significant differences among these groups.

Significant differences were observed in the following cases:

• Group 1 (White) smoked significantly more cigarettes per month than groups 2 (Black), 4 (Asian), and 5 (Hispanic/Latino).
• Group 3 (American Indian/Alaskan Native) smoked significantly more per month than group 5 (Hispanic/Latino).

Important Considerations in Interpretation

Some means have more than one letter next to them, indicating overlapping groups. It is crucial to follow the rule that means sharing at least one letter are not significantly different from one another. This nuanced approach ensures accurate identification of significant differences while maintaining rigorous control over Type I error rates.