Reinforcement Learning from Human and AI Feedback

A fully funded PhD opportunity to participate in the world-leading research undertaken by the EPSRC Doctoral Landscape Award at the University of Sheffield.

Supervised by Dr Bei Peng and Dr Zheng Yuan.

The University of Sheffield is one of the leading Russell Group universities in the UK. We carry out cutting-edge research with strong links to industry. When you enrol to do a PhD with us, you will be working with world-leading academics and have access to top of the range facilities. As a PhD student you will have the opportunity to gain skills not only to conduct research, but also to take your career to the next level, whether you want to stay in academia, go into industry or the public sector, or set up your own company. You will have access to a range of training and support services to help you excel in your studies and beyond.

Project Description

In the future, autonomous learning agents are likely to take an active role in human society, engaging in daily interaction and collaboration with humans. Developing learning algorithms that enable these agents to produce behaviours aligned with human values and preferences will therefore be of critical importance. Reinforcement learning from human feedback (RLHF) has recently emerged as the central technique for training AI systems to align with complex human objectives and preferences. For example, it has shown remarkable success in language domains [Ziegler et. al. 2019, Ouyang et. al. 2022], playing a decisive role in training state-of-the-art large language models, such as GPT-4 [OpenAI 2023], to generate outputs that are preferable, helpful, and harmless to humans.

However, state-of-the-art RLHF methods typically require tens of thousands of high-quality human preference labels, which can be expensive or even infeasible to collect. Humans can also make mistakes when providing feedback, which can dramatically degrade the performance of RLHF methods. To address this, reinforcement learning from AI feedback (RLAIF) [Bai et al. 2022] has recently been proposed as a promising alternative that leverages AI to supervise AI—that is, using an off-the-shelf large language model rather than humans to generate feedback labels. This technique holds great potential for reducing the amount of human feedback required for efficient learning. However, feedback labels generated by large language models can be unreliable or biased, posing risks to the trained AI system.

Learning from a combination of human and AI feedback can potentially provide the best of both worlds, reducing the amount of human feedback necessary for learning while improving the reliability of AI feedback. This PhD project aims to develop new RLHF methods that can leverage both human and AI feedback to train AI agents to solve a wide range of complex decision-making tasks more effectively, efficiently, and reliably.

We will first focus on gaining a better theoretical and empirical understanding of feedback generated by AI, such as large language models and vision language models. Fundamental questions that we will consider include: 1) how do different AI behaviours tend to generalise and interfere? and 2) what is the efficiency and reliability of human vs. AI feedback across various decision-making tasks? Then, we will develop novel RLHF algorithms that leverage both human and AI feedback to train AI agents to solve complex sequential decision-making tasks. The goal is to propose a new framework that best combine human and AI feedback to enable more efficient and reliable learning than current RLHF methods. The algorithms will be tested and validated though experiments with both simulations and real human subjects. Finally, we will investigate how to improve the robustness and interpretability of AI decision-making using methods such as chain-of-thought reasoning.

In completing this project, the student will gain hands-on experience working with several technologies that are now widely used in both industry and in academic research, including:

• Deep reinforcement learning, reinforcement learning from human feedback
• Large language models, vision language models
• Hardware accelerated deep learning frameworks such as Torch and TensorFlow
• Simulation platforms for complex control and manipulation tasks, such as OpenAI Gym, DeepMind Control Suite, MetaWorld, and Soft-Gym

How to apply

Interested candidates are strongly encouraged to contact the project supervisors to discuss your interest in and suitability for the project prior to submitting your application.

Please refer to the EPSRC DLA webpage for detailed information about the EPSRC DLA and how to apply.

Applicants must ensure that they use the correct application form linked to the EPSRC DLA webpage and must submit an application for October 2027 entry. The form is unique to the EPSRC DLA. It contains the list of advertised projects from which to select, and three questions specific to the DLA.

The application deadline is 17:00 GMT 13 January 2027.

The majority of our DLA studentships will be allocated to candidates who qualify for Home fee status, in accordance with EPSRC funding regulations which place strict limits on international recruitment.

Funding Notes

The award will fund the full tuition fee and a maintenance stipend at the UKRI rate (currently £21,805 per annum) for 3.5 years, as well as a research grant to support costs associated with the project.