Query Performance Prediction in Context of Conversational Search

About the Project

Query performance prediction (QPP) is a crucial task in information retrieval (IR) that estimates the effectiveness of a search system for a given query without requiring human relevance judgments. QPP has various applications, such as helping an IR system determine whether a specific query is likely to be effective. Based on the assessment, the search system can either apply query reformulation, adjust the retrieval configuration [1,4], or engage in an interactive session with the user (known as conversational search [5]) to better understand the underlying search intent and improve the overall search experience.

Accurately estimating query performance presents a significant challenge. Existing QPP models have typically relied on pre-retrieval features derived from collection statistics and/or post-retrieval features based on the top-retrieved documents [2, 3]. These models are often evaluated using sparse retrieval models (e.g., BM25 or DFR) as a reference system. With the rise of large models such as BERT [6], neural retrieval models have emerged and demonstrated improved retrieval effectiveness [7]. Despite extensive research on QPP for ad hoc search tasks that utilise pre-retrieval and post-retrieval features with traditional and neural models, there has been limited exploration of QPP in the context of Conversational search [5].

This Ph.D. project aims to develop query performance predictors specifically for conversational search, focusing on ambiguity detection and estimation of generated responses. Experiments will be conducted on standard TREC collections (e.g., MS MARCO, TREC Deep learning, and TREC CIS tracks) to demonstrate the effectiveness of the developed QPPs and compare them with the state-of-the-art approaches [8].

Academic qualifications

Have, or expect to achieve by the time of start of the studentship a first-class honours degree, or a distinction at master level, ideally in Computer Science, Data Science or Artificial Intelligence, with a good fundamental knowledge of Information retrieval (IR), Natural language processing (NLP), Machine learning/Deep learning, Large language models (LLM)

English language requirement

IELTS score must be at least 6.5 (with not less than 6.0 in each of the four components). Other, equivalent qualifications will be accepted. Full details of the University’s policy are available online.

Essential attributes:

• Experience of fundamental knowledge of IR, NLP, LLMs, and Machine learning/Deep learning.
• Competent in Shell scripting, R, Python, Java, and PyTorch,
• Knowledge of Information Retrieval and Deep Learning.
• Good written and oral communication skills
• Strong motivation, with evidence of independent research skills relevant to the project
• Good time management

Desirable attributes:

• Experience of IR Tools, such as Terrier IR/Lemur Indri/Anserini/Lucene.

APPLICATION CHECKLIST

• Completed application form
• CV
• 2 academic references, using the Postgraduate Educational Reference Form (download)
• Research project outline of 2 pages (list of references excluded). The outline may provide details about:
  1. Background and motivation of the project. The motivation, explaining the importance of the project, should be supported also by relevant literature. You can also discuss the applications you expect for the project results.
  2. Research questions or objectives.
  3. Methodology: types of data to be used, approach to data collection, and data analysis methods.
  4. List of references.

The outline must be created solely by the applicant. Supervisors can only offer general discussions about the project idea without providing any additional support.

• Statement no longer than 1 page describing your motivations and fit with the project.
• Evidence of proficiency in English (if appropriate)

To be considered, the application must use

• the advertised title as project title

For informal enquiries about this PhD project, please contact Dr Md Zia Ullah - M.Ullah@napier.ac.uk