Choosing Between the Two-sample T Test and its Alternatives: A Practical Guideline

Abstract

The traditional guideline for choosing between the two-sample t test and its alternatives is primarily based on assessing assumptions. Many flaws of this approach have been documented. In this paper, I address those flaws briefly and propose a new guideline for choosing between the two-sample t test and its alternatives. I propose to select the hypothesis that operationalizes the research question best. This selection is carried out before data collection and entails identifying the hypotheses that in principle produce meaningful results, and among those, the most appropriate one. For testing the selected hypotheses, I recommend bootstrap and permutation tests instead of the traditionally used parametric tests. The role of assessing assumptions is downgraded to deciding whether the results of a test are reliable. An important implication of the proposed guideline is that a nonparametric permutation test should be used instead of the t test in most cases. While this new guideline focuses on hypothesis testing, large parts are also relevant for choosing a confidence interval method due to the close correspondence between confidence intervals and hypothesis tests.

Authors 1

1. Julian David Karch corresponding Aachen Methodology and Statistics Department Institute of Psychology, Leiden University

   Leiden University

   Affiliation as printed

   Methodology and Statistics Department , Institute of Psychology , Leiden University , PO Box 9555 , 2300 RB Leiden

   University