Are extensive simulation studies always necessary? A critical overview

Statistical Methods & Applications

Abstract

Abstract Simulation studies are often carried out to study theoretical properties of a statistical procedure that cannot be derived analytically. Oftentimes, simulation studies can be very extensive, including various factors with various levels. This makes the results of such studies often difficult to report, and difficult for readers to interpret the general picture in the overabundance of results. Additionally, in many cases such extensive studies may not even be necessary. This work aims to explore and discuss some general guidelines on when (not) to include a simulation factor. Depending on the research question, oftentimes varying only one or two factors may suffice for answering the research question. In the current paper two different types of simulation studies are defined, namely proof of principle studies and robustness studies. Next, for a number of quality measures it is argued which factors are and are not useful to vary, both in proof of principle studies and robustness studies. Finally, the results of a literature study are presented confirming the need for guidelines on what factors to vary in a simulation study. A general recommendation is to not vary too many factors, especially for proof of principle studies.

Authors 2

1. Joost R. van Ginkel corresponding Aachen Department of Psychology, Methodology and Statistics

   Leiden University

   Affiliation as printed

   Department of Psychology, Methodology and Statistics, Leiden University, PO Box 9500, Leiden, 2300 RB, The Netherlands

2. Ralph C. A. Rippe Aachen Department of Child and Family Studies

   Leiden University

   Affiliation as printed

   Department of Child and Family Studies, Leiden University, Leiden, The Netherlands