Designing Benchmarks that Reflect Real-World NLP Use

About the Project

Project Description

NLP benchmarks strongly influence research directions, yet many benchmarks prioritise convenience and leaderboard performance over realism, usability, and societal relevance. As a result, benchmark scores often provide limited insight into how NLP systems behave in real-world settings. This PhD project rethinks how benchmarks are designed, interpreted, and used, with the goal of better reflecting real-world language use and decision-making contexts.

Rather than proposing a single new benchmark, the project takes a critical and constructive approach to benchmark design as a research problem in its own right. It examines how task formulations, datasets, and evaluation metrics shape research conclusions, system rankings, and claims about progress.

Aims and Methods

The student will analyse existing benchmarks in areas such as translation, summarisation, educational NLP, or other language technologies, identifying implicit assumptions about users, domains, communicative goals, and acceptable errors. Based on this analysis, the project will design alternative benchmark formulations that incorporate more realistic inputs, constraints, and evaluation criteria.

Methods will include comparative evaluation, stress testing, and meta-evaluation, examining how model rankings and conclusions change under different benchmark designs. The project may explore benchmarks that explicitly model domain variation, audience differences, uncertainty, or risk-sensitive errors, as well as the interaction between automatic metrics and human judgments.

Potential application domains include, but are not limited to, education, translation, summarisation, public-sector communication, or social care, and may involve high-resource or low-resource languages. While specific domains or languages may be used as case studies, the project aims to derive general principles for responsible and informative benchmark design across NLP.

Deliverables (indicative)

• Novel benchmark designs grounded in real-world use scenarios
• Meta-evaluations showing how benchmark assumptions affect system rankings
• Publicly released datasets and evaluation protocols
• Position papers and shared-task-style resources

Keywords

Benchmark design, evaluation methodology, applied NLP, robustness, responsible AI

How to Apply

This project is accepting applications all year round, for self-funded candidates.

Mode of Study: Full-time or part-time

Please submit your application via Computer Science and Informatics - Study - Cardiff University

In the funding field of your application, indicate “I am applying for a self-funded PhD in Computer Science and Informatics”, and specify the project title and supervisors of this project in the text box provided.

Academic criteria: A 2:1 Honours undergraduate degree or a master's degree, in computing or a related subject. Applicants with appropriate professional experience are also considered. Degree-level mathematics (or equivalent) is required for research in some project areas.

Applicants must demonstrate English language proficiency. Students who do not have English as a first language must prove this by obtaining an IELTS score of at least 6.5 overall, with a minimum of 6.0 in each skills component. A full list of accepted qualifications is available here: https://www.cardiff.ac.uk/study/international/english-language-requirements/postgraduate

If you are interested, please contact Dr Fernando Alva Manchego (alvamanchegof@cardiff.ac.uk) sending your CV in the first instance. The application process requires you to develop an individual research proposal jointly with the supervision team, which builds on the information provided in this advert.

Once you have developed the proposal with support from the supervisors, please submit your application following the instructions provided below.

Please submit your application via Computer Science and Informatics - Study - Cardiff University

In order to be considered candidates must submit the following information:

• In the ‘Research Proposal’ section of the application enter the name of the project you are applying to and upload your Individual research proposal. Your research proposal should not exceed 2000 words, including references and bibliography.
• A personal statement (as part of the university application form, or as a separate attachment, if you prefer).
• A CV. Guidance on CVs for a PhD position can be found on the FindAPhD website.
• Qualification certificates and Transcripts - original and English translation, if applicable.
• References x 2 which should be academic references. Please note you need to provide the reference documents as part of your application.
• Proof of English language (if applicable).

Interview– If the application meets all of the entrance requirements listed above, you will be invited to an interview.