University of Nottingham Drives Research Credibility Push in Landmark Nature Reproducibility Studies

Unveiling the Reproducibility Challenge in Social and Behavioural Sciences

The quest for reliable scientific knowledge has hit a significant roadblock in recent years, particularly within the social and behavioural sciences. These fields, which explore human behaviour, decision-making, economics, psychology, and sociology, have long grappled with questions about the stability of their findings. A reproducibility crisis—where original results fail to hold up under repeated testing—has sparked global debate, eroding confidence in published research and prompting calls for reform.

At the heart of this issue lies the core principle of science: findings should be verifiable. Reproducibility means obtaining the same result using the same data and analysis; replicability involves new data yielding similar outcomes; and analytical robustness checks if different methods on the same data confirm conclusions. A monumental effort, the SCORE (Systematising Confidence in Open Research and Evidence) project, has now provided unprecedented insights, analysing nearly 4,000 papers from 2009 to 2018 across 62 journals.

University of Nottingham's Pivotal Role in the SCORE Initiative

Researchers from the University of Nottingham played a key part in this landmark endeavour, contributing to three high-impact papers published in Nature in April 2026. Dr Joris Schroeder, Research Fellow in the School of Economics, and Dr Christopher Madan, Assistant Professor in the School of Psychology, joined 865 scientists from over 100 institutions worldwide. Funded by the US Defence Advanced Research Projects Agency (DARPA), SCORE evaluated research credibility through reproduction studies, replication experiments, expert assessments, and machine learning models.

"When we read a published study, how confident can we really be in its findings? SCORE takes on this question at an unprecedented scale," said Dr Schroeder. Dr Madan added, "Large field-wide collaborative projects like this are what truly move the field forward." As a Russell Group university ranked seventh for research power in the UK by REF 2021, Nottingham's involvement underscores its leadership in addressing methodological challenges.

Reproducibility Results: The Transparency Imperative

The first Nature paper, "Investigating the reproducibility of the social and behavioural sciences" (DOI: 10.1038/s41586-026-10203-5), revealed stark realities. Data was available for just 24% of the 600 sampled papers. Among testable cases, 72% approximately reproduced (effects within 15% or p-values within 0.05), but only 53% exactly matched originals.

Crucially, sharing data and code transformed outcomes: 88% approximate and 75% precise reproducibility. Without them, precise rates plummeted to 11% using reconstructed data. Fields like political science and economics fared better due to stronger open data norms, highlighting how journal policies—rising from 27% in 2009 to 52% by 2025—drive progress.

• Data availability: 24% of papers
• Approximate reproducibility: 72%
• Exact reproducibility: 53%
• With data/code: Up to 88% approximate

Replicability Findings: Effects Halve on Retest

The replicability paper (DOI: 10.1038/s41586-025-10078-y) tested 274 positive claims from 164 papers via new experiments. Success rate: 49-55%, with replicated effects averaging less than half the original size (correlation from 0.25 to 0.10). Economics showed lower rates, education higher, but no field excelled consistently.

This aligns with prior crises, like the 2015 Nature/Science replication where only 36% succeeded. SCORE's scale—largest ever—confirms persistent issues, urging caution with single studies.

Photo by BEN ELLIOTT on Unsplash

Analytical Robustness: Sensitivity to Methods

Robustness testing reanalysed 100 papers by five experts each using alternatives. Only 34% closely matched originals, but 74% drew similar conclusions; 24% failed to detect effects, 2% contradicted. The paper (DOI: 10.1038/s41586-025-09844-9) stresses credibility's multi-dimensionality—repro, robust, repl weakly linked.

Small effects, method variations, and analyst choices explain divergences, not fraud. UK implications: Aligns with UKRI's open research push, emphasising preregistration and sensitivity analyses.

Predicting Credibility: Humans Outperform Machines

SCORE explored forecasting replicability. Human experts predicted fairly accurately; machines less so. No strong predictors like citations emerged—only data sharing correlated robustly. Leader Brian Nosek noted, "A lot more evidence is needed before we would be confident in a valid, scalable solution."

For UK researchers, this validates peer review but calls for AI aids in triage.

UK Higher Education Context and Nottingham's Leadership

In the UK, where social sciences drive policy via ESRC funding (£500m+ annually), these findings resonate. Universities like Nottingham, Stirling (Dr Arran Reader contributed), and others face pressure to adopt open practices amid REF 2021's emphasis on rigour. UKRI mandates data management plans; journals like British Journal of Psychology require sharing.

Nottingham's Horizon Centre for Doctoral Training exemplifies proactive steps, training in reproducible workflows. Yet challenges persist: resource strains, career incentives favour novelty over replication.

Solutions: Embracing Open Science Practices

SCORE's open dataset empowers meta-research. Solutions include:

• Preregistration: Predefine analyses to curb p-hacking.
• Data/code sharing: Platforms like OSF.io boost repro by 3x.
• Replication incentives: Fund registered reports.
• Training: ReproducibiliTea clubs at Stirling/Nottingham.
• Journal reforms: Badges for openness.

UK's Plan S and UKRI Open Access Policy accelerate this, positioning British unis as leaders.

Photo by Winston Tjia on Unsplash

Case Studies: Lessons from Economics and Psychology

In economics, robustness low due to model choices; Nottingham's Schroeder highlights need for sensitivity tests. Psychology saw variable replicability, echoing 2015 crisis. Real-world: Policy reliant on fragile findings risks flawed interventions, e.g., behavioural nudges.

Future Outlook: Building a More Credible Research Ecosystem

SCORE charts a path: Multi-dimensional assessment over single metrics. For UK HE, expect REF 2029 to reward openness; AI tools may aid predictions. Nottingham's contributions signal commitment, fostering trust in social sciences for evidence-based policy.

Optimism prevails: Transparency gains since 2009 promise progress, ensuring research withstands scrutiny.