About the Project
Clinical prediction models are developed to help healthcare providers estimate the probability of a patient health outcome and are widely used in many clinical areas [1]. They calculate outcome probability using pre-selected clinical variables to inform treatment decisions. Once a prediction model has been developed, it then needs to be externally evaluated in data that has not been used to develop it to understand its likely use and performance in clinical practice. Often evaluation data has a…
different patient case mix (distribution of patient characteristics) that can affect how the model performs. Matters are further complicated when a developed prediction model is repurposed to clinical areas, patient populations or outcomes that they have not been developed.
In these cases, understanding the degree of similarity/dissimilarity between the development and evaluation data (relatedness) becomes important, as prediction models often demonstrate reduced performance or miscalibrated predictions when applied to new populations or clinical settings [2,3], including new data from the same population used develop the model. We should not assume that prediction models can be transferred directly from one setting or target population to another [4–7]. Instead, further understanding is needed when and where a developed model does and does not work well.
Therefore, when externally evaluating a prediction model the relatedness of development and evaluation data can be used understand with the evaluation study assesses model transportability (ability of a model to perform well [accurately] on new patient data with dissimilar case mix to the development sample) or reproducibility (ability of a model to perform well [accurately] on new patient data with similar case mix to the development sample). Variations in predictive performance can be attributed to differences in patient characteristics, clinical settings, or time periods between the development and evaluation data [8–10].
The degree of similarity between development and validation populations and data can help assess whether an external validation study assesses reproducibility (model stability in a similar context) or transportability (predictive accuracy in a genuinely different population or setting) [8]. Although this distinction has been proposed to improve the interpretation of external validation studies [5,6,8,11,12], population similarity is rarely quantified or explicitly reported, limiting the interpretation of transportability across studies. Methodological guidance is needed to implement and extend current model evaluation practice to produce evidence and understanding how far away from the development case mix a dataset can go without affecting the model performance, especially when transporting a model to a new clinical setting.
Addressing these uncertainties would not only improve the interpretation of external evaluation studies, but also represent a more efficient use of research resources by enabling the optimisation of existing models to new populations and settings [13–17], as well as generating more information about the developed model and when and where it does and does not work well, including distribution of its clinical predictors.
This DPhil will involve real-world scenarios (including rheumatology) to address this methodological gap and develop guidance to assess and understand the extent to which the case mix for a prediction model can vary without affecting its model performance (performance decay), thus informing need and methods for model updating in new clinical settings.
Research objectives
- Systematically review published external validation studies of clinical prediction models to characterise how the reproducibility–transportability distinction is currently evaluated and reported, and to quantify what proportion of existing validation evidence genuinely assesses transportability.
- Explore the relationship between the degree of population divergence and observed changes in predictive performance, including discrimination and calibration, to identify thresholds at which transportability meaningfully deteriorates and determine whether this relationship varies by clinical domain or model type.
- Apply developed methods to real world clinical examples, including rheumatology.
Supervisors:
- Associate Professor Paula Dhiman (primary supervisor), NDORMS, CSM, University of Oxford. paula.dhiman@csm.ox.ac.uk
- Associate Professor Laura Coates, NDORMS, University of Oxford, laura.coates@ndorms.ox.ac.uk
- Professor Gary Collins, Applied Health, University of Birmingham, g.s.collins@bham.ac.uk
Training
The Botnar Research Centre plays host to the University of Oxford's Institute of Musculoskeletal Sciences and the Centre for Statistics in Medicine (CSM), which enables and encourages research and education into the causes of musculoskeletal disease and their treatment and medical statistics. Training will be provided in techniques including medical statistics, prediction model research, simulations studies and clinical data management.
A core curriculum of lectures will be taken in the first term to provide a solid foundation in a broad range of subjects including musculoskeletal biology, inflammation, epigenetics, translational immunology, data analysis and the microbiome. Students will also be required to attend regular seminars within the Department and those relevant in the wider University.
Students will be expected to present data regularly in Departmental and CSM seminars, and to attend external conferences to present their research globally, with limited financial support from the Department.
Students will also have the opportunity to work closely with the Oxford Psoriatic Arthritis Centre (Ox-PACE, NDORMS) and the Biostatistics, Evidence Synthesis, Test Evaluation and prediction Models (BESTEAM, Applied Health, University of Birmingham).
Students will have access to various courses run by the Medical Sciences Division Skills Training Team and other Departments. All students are required to attend a 2-day Statistical and Experimental Design course at NDORMS (information will be provided once accepted to the programme).
How to Apply
Please contact the relevant supervisor(s), to register your interest in the project, and, if required, the departmental Education Team (graduate.studies@ndorms.ox.ac.uk), who will be able to advise you of the essential requirements for the programme and provide further information on how to make an official application.
Interested applicants should have, or expect to obtain, a first or upper second-class BSc degree or equivalent in a relevant subject and will also need to provide evidence of English language competence (where applicable). The application guide and form is found online and the DPhil will commence in October 2027.
Applications should be made to one of the following programmes using the specified course code.
- D.Phil in Clinical Epidemiology and Medical Statistics (course code: RD_NNRA1)
For further information, please visit http://www.ox.ac.uk/admissions/graduate/applying-to-oxford
Applications open mid-September
Application deadline: 12:00 on 1st December
References
- Van Smeden M, Reitsma JB, Riley RD, et al. Clinical prediction models: diagnosis versus prognosis. Journal of Clinical Epidemiology. 2021;132:142–5. doi: 10.1016/j.jclinepi.2021.01.009
- Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17:230. doi: 10.1186/s12916-019-1466-7
- Siontis GCM, Tzoulaki I, Castaldi PJ, et al. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. Journal of Clinical Epidemiology. 2015;68:25–34. doi: 10.1016/j.jclinepi.2014.09.007
- Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. doi: 10.1136/bmj.b605
- Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19:453–73. doi: 10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
- Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130:515–24. doi: 10.7326/0003-4819-130-6-199903160-00016
- Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med. 2006;144:201–9. doi: 10.7326/0003-4819-144-3-200602070-00009
- Debray TPA, Vergouwe Y, Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol. 2015;68:279–89. doi: 10.1016/j.jclinepi.2014.06.018
- Vergouwe Y, Moons KGM, Steyerberg EW. External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-Mix Effect From Incorrect Coefficients. Am J Epidemiol. 2010;172:971–80. doi: 10.1093/aje/kwq223
- Usher-Smith JA, Stephen J Sharp, Griffin SJ. The spectrum effect in tests for risk prediction, screening, and diagnosis. BMJ. 2016;353:i3139. doi: 10.1136/bmj.i3139
- Knottnerus JA. Prediction Rules: Statistical Reproducibility and Clinical Similarity. Med Decis Making. 1992;12:286–7. doi: 10.1177/0272989X9201200407
- Knottnerus JA. Diagnostic prediction rules: principles, requirements and pitfalls. Prim Care. 1995;22:341–63.
- Damen JAAG, Hooft L, Schuit E, et al. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ. 2016;353:i2416. doi: 10.1136/bmj.i2416
- Phung MT, Tin Tin S, Elwood JM. Prognostic models for breast cancer: a systematic review. BMC Cancer. 2019;19:230. doi: 10.1186/s12885-019-5442-6
- van den Boorn HG, Engelhardt EG, van Kleef J, et al. Prediction models for patients with esophageal or gastric cancer: A systematic review and meta-analysis. PLoS One. 2018;13:e0192310. doi: 10.1371/journal.pone.0192310
- Lamain-de Ruiter M, Kwee A, Naaktgeboren CA, et al. Prediction models for the risk of gestational diabetes: a systematic review. Diagn Progn Res. 2017;1:3. doi: 10.1186/s41512-016-0005-7
- Van Calster B, Steyerberg EW, Wynants L, et al. There is no such thing as a validated prediction model. BMC Med. 2023;21:70. doi: 10.1186/s12916-023-02779-w
This is a Preview Listing…
You must sign in to see the full job description, and to apply.
Manage / Upgrade this job to a Full Job Listing.
Find Your Best Opportunity
Tell them AcademicJobs.com sent you!