Browse by author
Lookup NU author(s): Dr Chris MorehORCiD
This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
The same dataset can be analysed in different justifiable ways to answer the same researchquestion, potentially challenging the robustness of empirical science1–3. In this crowd initiative,we investigated the degree to which research findings in the social and behavioural sciences arecontingent on analysts’ choices. We examined a stratified random sample of 100 studiespublished between 2009 and 2018, where for one claim per study, at least five re-analystsindependently re-analysed the original data. The statistical appropriateness of the re-analyses wasassessed in peer evaluations, and the robustness indicators were inspected along a range ofresearch characteristics and study designs. We found that 34% of the independent re-analysesyielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report;with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusionsdrawn, 74% of analyses were reported to arrive at the same conclusion as in the originalinvestigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in theoriginal investigation. This exploratory study suggests that the common single-path analyses insocial and behavioural research should not simply be assumed to be robust to alternativeanalyses4. Therefore, we recommend the development and use of practices to explore andcommunicate this neglected source of uncertainty.
Author(s): Aczel B, Szaszi B, Clelland H, Haigh M, Rotella A, Moreh C, Et al
Publication type: Article
Publication status: Published
Journal: Nature
Year: 2026
Volume: 652
Pages: 135-142
Online publication date: 01/04/2026
Acceptance date: 31/10/2025
Date deposited: 03/02/2026
ISSN (print): 0028-0836
ISSN (electronic): 1476-4687
Publisher: Nature
URL: .10.1038/s41586-025-09844-9
DOI: 10.1038/s41586-025-09844-9
ePrints DOI: 10.57711/x84y-3143
Data Access Statement: Study data, materials and analysis code files will be available on the project OSF (https://osf.io/q5h2c/) and GitHub repositories (https://github.com/marton-balazs-kovacs/multi100/). Archived data include the original datasets or a description how to gain access to them.
Altmetrics provided by Altmetric
