"Integrating collection, analysis, and interpretation of data in social and behavioral research"

Abstract
The replication crisis has made us increasingly aware of the flaws of conventional statistical reasoning based on hypothesis testing.  The problem is not just a technical issue with p-values, not can it be solved using preregistration or other purely procedural approaches.  Rather, appropriate solutions have three aspects.  First, in collecting your data there should be a concordance between theory and measurement:  for example, in studying the effect of an intervention applied to individuals, you should measure within-person comparisons.  Second, in analyzing your data, you should study all comparisons of potential interest, rather than selecting based on statistical significance or other inherently noisy measures.  Third, you should interpret your results in the context of theory, background knowledge, and the data collection and analysis you have performed.  We discuss these issues on a theoretical level and with examples in psychology, political science, and policy analysis.