Accuracy in parameter estimation and simulation approaches for sample-size planning accounting for item effects

Behavior Research Methods, vol. 58, pp. 48

Abstract

The planning of sample size for research studies often focuses on obtaining a significant result given a specified level of power, significance, and an anticipated effect size. This planning requires prior knowledge of the study design and a statistical analysis to calculate the proposed sample size. However, there may not be one specific testable analysis from which to derive power (Silberzahn et al., Advances in Methods and Practices in Psychological Science, 1(3), 337356, 2018) or a hypothesis to test for the project (e.g., creation of a stimuli database). Modern power and sample size planning suggestions include accuracy in parameter estimation (AIPE, Kelley, Behavior Research Methods, 39(4), 755-766, 2007; Maxell et al., Annual Review of Psychology, 59, 537-563, 2008) and simulation of proposed analyses (Chalmers & Adkins, The Quantitative Methods for Psychology, 16(4), 248-280, 2020). These toolkits offer flexibility in traditional power analyses that focus on the if-this, then-that approach. However, both AIPE and simulation require either a specific parameter (e.g., mean, effect size, etc.) or a statistical test for planning sample size. In this tutorial, we explore how AIPE and simulation approaches can be combined to accommodate studies that may not have a specific hypothesis test or wish to account for the potential of a multiverse of analyses. Specifically, we focus on studies that use multiple items and suggest that sample sizes can be planned to measure those items adequately and precisely, regardless of the statistical test. This tutorial also provides multiple code vignettes and package functionality that researchers can adapt and apply to their own measures.

Authors 25

1. Erin M. Buchanan corresponding

   Harrisburg University of Science and Technology

   Affiliation as printed

   Analytics, Harrisburg University of Science and Technology, 326 Market St, Harrisburg, PA, 17101, USA. ebuchanan@harrisburgu.edu

2. Mahmoud Medhat Elsherif

   University of Leicester

   Affiliation as printed

   Department of Psychology and Vision Sciences, University of Leicester, Leicester, UK

3. Jason Geller

   Boston College

   Affiliation as printed

   Department of Psychology and Neuroscience, Boston College, Boston, USA

4. Christopher L. Aberson
   Affiliation as printed

   The Dissertation Coach, Raleigh, NC, USA

5. Necdet Gurkan

   University of Missouri–St. Louis

   Affiliation as printed

   University of Missouri-St. Louis, St. Louis, MO, USA

6. Ettore Ambrosini

   University of Padua

   Affiliation as printed

   Department of Neuroscience, University of Padova, Padua, Italy

7. Tom Heyman Aachen Methodology and Statistics Unit Institute of Psychology, Leiden University

   Leiden University

   Affiliation as printed

   Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, Netherlands

8. Maria Montefinese

   University of Padua

   Affiliation as printed

   Department of Developmental and Social Psychology, University of Padova, Padua, Italy

9. Wolf Vanpaemel

   KU Leuven

   Affiliation as printed

   University of Leuven, Leuven, Belgium

10. Krystian Barzykowski

    Jagiellonian University

    Affiliation as printed

    Applied Memory Research Laboratory, Institute of Psychology, Faculty of Philosophy, Kraków, Poland

11. Carlota Batres

    Franklin & Marshall College

    Affiliation as printed

    Franklin and Marshall College, Lancaster, PA, USA

12. Katharina Fellnhofer

    ETH Zurich

    Affiliation as printed

    ETH Zürich, Zürich, Switzerland

    Research and Innovation Management GmbH, Neumarkt an der Ybbs, Austria

13. Guanxiong Huang

    City University of Hong Kong

    Affiliation as printed

    Department of Media and Communication, City University of Hong Kong, Kowloon Tong, Hong Kong

14. Joseph Patrick McFall

    Children’s Institute · University of Rochester

    Affiliation as printed

    Children's Institute Inc., Los Angeles, CA, USA

    Department of Psychology, University of Rochester, Rochester, NY, USA

15. Gianni Ribeiro

    The University of Queensland

    Affiliation as printed

    School of Psychology, The University of Queensland, Queensland, Australia

16. Jan Philipp Röer

    Witten/Herdecke University

    Affiliation as printed

    Department of Psychology and Psychotherapy, Witten/Herdecke University, Witten, Germany

17. José Luis Ulloa

    University of Talca

    Affiliation as printed

    Programa de Investigación Asociativa (PIA) en Ciencias Cognitivas, Centro de Investigación en Ciencias Cognitivas (CICC), Facultad de Psicología, Universidad de Talca, Talca, Chile

18. Timo Benjamin Roettger

    University of Oslo

    Affiliation as printed

    University of Oslo, Oslo, Norway

19. Katherine D. Valentine

    Harvard University · Massachusetts General Hospital

    Affiliation as printed

    Harvard Medical School, Boston, MA, USA

    Massachusetts General Hospital, Boston, MA, USA

20. Antonino Visalli

    IRCCS San Camillo Hospital

    Affiliation as printed

    IRCCS San Camillo Hospital, Venice, Italy

21. Kathleen Schmidt

    Ashland University

    Affiliation as printed

    Ashland University, Ashland, OH, USA

22. Martin R. Vasilev

    Bournemouth University

    Affiliation as printed

    Bournemouth University, Poole, UK

23. Giada Viviani

    University of Padua

    Affiliation as printed

    Department of Developmental and Social Psychology, University of Padova, Padua, Italy

24. Jacob Francisco Miranda

    California State University, East Bay

    Affiliation as printed

    California State University East Bay, Hayward, CA, USA

25. Savannah C Lewis

    University of Alabama

    Affiliation as printed

    University of Alabama, Tuscaloosa, AL, USA