Establishing driving simulator validity: drawbacks of null-hypothesis significance testing when compared to equivalence tests and Bayes factors

Theoretical Issues in Ergonomics Science, vol. 25, pp. 546–566

Abstract

The validity of driving simulators is a topic that has been addressed ever since their first use. In practice, a comparison of field data and simulator data is often used for validation purposes. A systematic literature search was conducted which identified large heterogeneity in both study designs and statistical methods used in research on simulator validity. Most of the included studies concluded simulator validity (thus equivalence of field data and simulator data) from non-significant results in Null-Hypothesis Significance Testing (NHST), which is a well-known yet somewhat controversial practice. In this work, the consequences of both study design choice and the choice of the statistical method are outlined and alternatives such as using Bayes factor hypothesis testing and equivalence tests are discussed and applied to a real data example. The present paper is intended to shed light on the influence of the statistical method on conclusions drawn from experimental data. For research on driving simulator validity, we highly encourage the use of equivalence tests or Bayes factor hypothesis tests, in particular when evidence of the data suggests that an effect might be absent.

Authors 4

1. Chantal Himmels corresponding

   Johannes Kepler University of Linz · BMW (Germany) · Technische Hochschule Ingolstadt · BMW Group (Germany)

   Affiliation as printed

   BMW Group, Central Innovations, Munich, Germany

   Institute of Pervasive Computing, Johannes-Kepler University Linz, Linz, Austria

   Technische Hochschule Ingolstadt, Human-Computer Interaction Group, Ingolstadt, Germany

2. Klemens Weigl Aachen Department of Psychological Assessment and Intervention Institute of Psychology

   Catholic University of Eichstätt-Ingolstadt · RWTH Aachen University

   Affiliation as printed

   Department of Psychological Assessment and Intervention, Institute of Psychology, RWTH Aachen University, Aachen, Germany

   Department of Psychology, Catholic University Eichstätt-Ingolstadt, Eichstätt, Germany

3. Andreas Riener

   Technische Hochschule Ingolstadt

   Affiliation as printed

   Technische Hochschule Ingolstadt, Human-Computer Interaction Group, Ingolstadt, Germany

4. Joost Venrooij

   BMW (Germany) · BMW Group (Germany)

   Affiliation as printed

   BMW Group, Central Innovations, Munich, Germany