Power to the Stimuli: Not the Effect

Sample size planning for research studies often focuses on obtaining a significant result given a specified level of power, alpha, and proposed effect size. This planning generally requires prior knowledge of study design and a statistical analysis to calculate the proposed sample size. However, there may not be just one specific testable analysis from which to derive power (Silberzahn et al., 2018)(link is external) or even a hypothesis to test for the project (e.g., stimuli database creation). Newer power and sample size planning suggestions include Accuracy in Parameter Estimation (AIPE; Kelley, 2007; Maxwell et al., 2008)(link is external) and simulation of proposed analyses (Chalmers & Adkins, 2020)(link is external). These toolkits provide flexibility in traditional power analyses that focus on the if-this-then-that approach, yet, both AIPE and simulation require either a specific parameter (e.g., mean, effect size, etc.) or statistical test for planning sample size. In this talk, I will explore how these latter two approaches can be combined to accommodate studies that may not have a specific hypothesis test or wish to account for the potential of a multiverse of analyses. Specifically, the examples focus on studies that implement multiple items and suggest that sample sizes can be planned to measure those items adequately and accurately, regardless of statistical test. Results show that pilot data can be used to determine a sample size that represents well-measured data, and multiple code vignettes will be provided for researchers to adapt and apply to their own measures.