The reproducibility crisis in social sciences has long plagued researchers, policymakers, and funders, raising questions about the reliability of findings that shape everything from economic models to educational reforms. A groundbreaking study led by researchers at the University of Stirling has now pinpointed a straightforward solution: sharing data and code. This landmark Nature paper reveals that when these resources are available, over three-quarters of results can be precisely reproduced, offering a beacon of hope amid ongoing debates.
Social sciences, encompassing fields like psychology, economics, sociology, and political science, grapple with challenges unique to studying human behaviour. Unlike physics experiments with controlled conditions, social research often involves complex variables, small sample sizes, and subjective interpretations. The crisis erupted prominently around 2011 with high-profile failures to replicate seminal psychology studies, sparking a global reckoning.
🔬 The Stirling-Led SCORE Project: Methods and Scope
The study, titled "Investigating the reproducibility of the social and behavioural sciences," stems from the Systematizing Confidence in Open Research and Evidence (SCORE) program, funded by the US Defense Advanced Research Projects Agency. An international team from over 100 institutions, including Dr. Arran Reader from Stirling's Faculty of Natural Sciences, scrutinized 600 quantitative papers published between 2009 and 2018 across 62 top journals in business, economics, education, political science, psychology, sociology, and related areas.
Researchers first checked data and code availability—crucial for reproducibility, defined as re-running the original analyses on the same data to match reported results. They distinguished this from replicability (new data, same question) and robustness (alternative analyses). Where possible, independent analysts attempted reproduction: obtaining data from authors or reconstructing it from sources, then verifying statistical outputs.
Key Findings: Only Half Replicate, But Data Sharing Transforms Outcomes
Strikingly, just 24% of papers shared data, and 20% provided both data and code. Among 182 papers where reproduction was feasible, 72% were approximately reproducible (effects within 15% or p-values within 0.05), and 53% precisely so. However, when both data and code were shared, figures soared to 88% approximate and 75% precise reproducibility. Reconstructing datasets without access dropped precise matches to a mere 11%.
For replicability, 164 papers yielded a 49% success rate—consistent with prior efforts like psychology's 36% in 2015. Effect sizes halved from originals (0.25 to 0.10). Economics lagged in replication, while political science and economics excelled in reproducibility, likely due to stricter journal mandates.Read the full Nature paper here.
Field-by-Field Breakdown: Economics Struggles, Pol Sci Shines
- Psychology and Sociology: Lower data sharing (under 20%), reproducibility around 50% precise where possible.
- Economics: Better data/code availability but lowest replication (due to complex models?), highlighting analytical challenges.
- Political Science: Highest rates, with journals enforcing transparency pre-publication.
- Education: Poor reproducibility but stronger replication, suggesting robust core findings despite reporting issues.
Transparency trends are positive: journal data-sharing requirements rose from 27% in 2018 to 52% by 2025.
Why Data Sharing Matters: Quotes from Stirling Experts
Dr. Arran Reader emphasized: "Three-quarters of evaluated results could be precisely reproduced when data and code were available, highlighting the importance of sharing these resources." He added that sharing facilitates verification, error correction, and public confidence.
Dr. Gemma Learmonth, leading Stirling's Open Research Network, noted: "Reproducibility is about continually improving how we design, document, and share research." Olivia Miske, lead author, framed it as quality control: "Openness may be an effective tool for quality control."
Photo by Julia Taubitz on Unsplash
UK Response: UKRN and University Initiatives
The UK Reproducibility Network (UKRN), a consortium of universities including Stirling, Cambridge, and Manchester, drives change through training, senior reproducibility roles at 10 institutions, and open practices. Stirling launched the Stirling Open Research and Scholarship (SORS) Network and ReproducibiliTea journal clubs—student-led discussions on replication.
UK Research and Innovation (UKRI) mandates data management plans, with a 2025 draft policy harmonizing sharing expectations to boost reproducibility.Explore UKRI's open research guidance.
Implications for UK Higher Education and Research Funding
For UK universities, unverifiable results undermine REF assessments, grant applications, and policy influence. The Stirling findings validate UKRI's push: without data, independent checks are impossible, eroding trust. Social sciences, vital for UK policy on inequality, mental health, and economy, risk stalled progress. Funders like ESRC now require data archiving via UK Data Service, aligning with global open science.
Stakeholders—from VCs to PhD students—face pressure: poor reproducibility inflates Type I errors, wastes resources (e.g., £100m+ annually in failed replications), and fuels public skepticism post-COVID.
Real-World Case Studies: Lessons from Psychology and Economics
In psychology, the 2015 Open Science Collaboration replicated 100 studies: only 36% succeeded, sparking preregistration (pre-committing analyses). Economics saw the Many Analysts, One Dataset project reveal conclusion variability from same data.
Stirling's work echoes: a political science paper with shared code reproduced precisely; an economics model without faltered. UK example: Birmingham's replication of inequality studies via shared data confirmed core effects, informing policy.
Practical Solutions: Step-by-Step Guide to Open Practices
- Preregister: Outline hypotheses, analyses on OSF.io before data collection.
- Share Data/Code: Use repositories like Zenodo, Figshare; anonymize sensitive info.
- Transparent Reporting: Follow TOP Guidelines (Transparency and Openness Promotion).
- ReproducibiliTea Clubs: Join or start at your uni for peer review.
- Tools: R Markdown, Jupyter for executable documents.
UK unis like Edinburgh mandate these for grants, yielding 20-30% reproducibility gains.
Challenges and Criticisms: Not All Failures Are Crises
Critics argue low rates reflect small effects, not fraud—social phenomena are noisy. Failures may stem from analyst errors or software diffs (e.g., R vs. Stata). Yet, Stirling stresses: non-reproducibility flags ambiguities, not falsity. Economics' complexity demands nuanced views.
Photo by Huichao Ji on Unsplash
Future Outlook: Towards a Credible Social Science Ecosystem
With journal mandates rising and AI aiding code checks, 80%+ reproducibility seems feasible by 2030. UKRN's expansion, UKRI's policies, and Stirling-like collaborations promise progress. Researchers embracing openness will lead, bolstering careers via robust impacts. For UK higher ed, this crisis is an opportunity: prioritize verification to reclaim prestige.
