PhD Research Fellow in Structured Vision-Language Models

seeking a motivated PhD candidate to join our interdisciplinary research centre at the University of Oslo, Norway. You will be part of a dynamic group of early career researchers, supervised by senior experts, collaborating across disciplines to tackle fundamental challenges through innovative methods, theory and critical analysis. The fellowship period is 3 years. A fourth year may be considered, and it will involve 25 % of other career-promoting work. Other career-promoting work may consist of teaching, supervision, and/or research assistance. This is dependent upon the qualifications of the applicant and the current needs of the Department of Informatics Starting date from January 2027 and upon individual agreement thereafter. A candidate may be appointed to only one PhD Research Fellowship at the University of Oslo. About Integreat See the video for more information about Integreat. About Integreat Integreat - Norwegian Centre for Knowledge-driven Machine Learning is a Centre of Excellence, funded by the Research Council of Norway (2023-33). Integreat has two branches, one in Oslo (University of Oslo, UiO) and one in Tromsø (UiT The Arctic University of Norway). Machine learning is the mathematical and computational engine of Artificial Intelligence (AI), and therefore it is a fundamental force of technological progress in our increasingly digital, data- and algorithm-driven world. Integreat develops theories, methods, models, and algorithms that integrate general and domain-specific knowledge with data, laying the foundations of next generation machine learning. We do this by combining the mathematical and computational cultures, and the methodologies and theories, of statistics, logic, language technology, ethics and machine learning, in new and unique ways. Focus of Integreat is to develop groundbreaking methods and theories, and therefore solving fundamental problems in science, technology, health and society. Integreat draws on the research strengths of researchers and students from the departments of Mathematics, Informatics, Philosophy, and the Oslo Centre for Biostatistics and Epidemiology at UiO, the Norwegian Computing Centre (NR) and the ML group at UiT, with members from the departments of Physics and Technology, Mathematics and Statistics, and Computer Science. Project description Vision-language models (VLMs) can name what is in a picture, but they often miss what is happening in it. Shown a park scene, a model may list "a dog", "a Frisbee" and "a park" and still fail to say that the dog is leaping to catch the Frisbee mid-air. The same gap shows up when a model misreads a symbolic painting or gives a vague description of a medical image. More training data does not close this gap, because the models have no explicit representation of how the objects in a scene relate to each other. This project gives VLMs that representation. You will develop panoptic scene graphs, in which every object and background region in an image becomes a node and the edges encode spatial, semantic and relational context. The graphs are built from superpixels rather than the fixed square patches used in standard vision transformers. Superpixels follow object boundaries, so the nodes of the graph are meaningful from the start. The project builds directly on our recent work on superpixel tokenization for vision transformers (ECCV Workshops 2024, NeurIPS 2025). The scene graph is then fused with a large language model (LLM), and the two parts inform each other. The LLM guides which objects and relations the vision model should focus on, and the graph grounds the LLM's answers in what is actually in the image, which reduces hallucination. Because a scene graph has the same form as a knowledge graph, it also opens the way to graph-based visual retrieval-augmented generation (graph visual RAG), where a model links what it sees to external knowledge before it answers. The aim is a new class of VLMs that reason over images instead of only describing them. Such models would be more accurate, more explainable (each answer can be traced to nodes and edges in the graph) and more useful where deep understanding matters. Examples include a robot that follows an instruction such as "place the red mug next to the laptop onto the top shelf", a medical system that reasons about the relations between structures in an X-ray, and a self-driving car that anticipates how pedestrians and vehicles will interact. As a PhD fellow on the project, you will contribute to: develop superpixel-based panoptic segmentation models built on modern transformer architectures design methods that predict relations between segmented regions and generate panoptic scene graphs fuse scene graphs with LLMs for visual question answering and grounded reasoning, including graph visual RAG publish and present your results at leading machine learning and computer vision venues Project supervisors: Professor Adín Ramírez Rivera at the Department of Informatics, UiO, in collaboration with Benjamin Ricaud and Filippo Maria Bianchi at UiT The Arctic University of Norway. Working language: English. What skills are important in this role? The Faculty of Mathematics and Natural Sciences has a strategic ambition to be among Europe’s leading communities for research, education and innovation. Candidates for these fellowships will be selected in accordance with this, and expected to be in the upper segment of their class with respect to academic credentials. Qualification requirements: Master’s degree or equivalent in computer science, machine learning, data science, mathematics, statistics, electrical engineering, or a related field. Foreign completed degree (M.Sc.-level) corresponding to a minimum of four years in the Norwegian educational system. Experience in scientific programming. Desired qualifications: Proficiency in scientific programming (e.g., Python) Proficiency in deep learning frameworks (e.g., PyTorch) Background in machine learning, mathematics, linear algebra, and statistics Experience in creating, adapting, or training modern, token-based transformer architectures Experience working with image / vision models Experience in creating or adapting large language models (LLMs) and interfacing them with other models Familiarity with image segmentation Familiarity with Scene Graph Generation Familiarity with LLM fine-tuning and alignment Knowledge of graph-based retrieval-augmented generation (Graph RAG) Understanding of probabilistic modelling and information-based regularizers Strong scientific writing skills Language requirement: Good oral and written communication skills in English English requirements for applicants from outside of EU/ EEA countries and exemptions from the requirements: https://www.mn.uio.no/english/research/phd/regulations/regulations.html#toc8 Grade requirements: The norm is as follows: the average grade point for courses included in the Bachelor’s degree must be C or better in the Norwegian educational system the average grade point for courses included in the Master’s degree must be B or better in the Norwegian educational system the Master’s thesis must have the grade B or better in the Norwegian educational system The purpose of the fellowship is research training leading to the successful completion of a PhD degree. For more information see: Supplementary Regulations at the Faculty of Mathematics and Natural Sciences to the Regulations for the Degree of Philosophiae Doctor (PhD) at the University of Oslo - The Faculty of Mathematics and Natural Sciences Personal skills We need different perspectives in our work Integreat is committed to equity, diversity, inclusion, and belonging, guided by our INTEGREAT principles (Integrity, Non discrimination, Tact, Environment, Gratitude, Respect, Empathy, Accountability, Transparency). We embed these values in practice through inclusive hiring and structured evaluations, targeted mentoring and career development, and flexible work arrangements and accommodations to ensure everyone can thrive and contribute. We particularly welcome applications from women and gender minorities who are underrepresented in STEM, as well as applicants with immigrant backgrounds, disabilities, non linear career paths, or career breaks. We will take career interruptions and diverse career trajectories into account during evaluation. If you have an immigrant background, a disability, or CV gap s, we encourage you to indicate this in the job application portal. We always invite at least one qualified candidate from each group for an interview. In this context, disability is defined as an applicant who identifies as having a disability that requires workplace or employment-related accommodations. For more details about the requirements, please refer to the Employer portal (Norwegian). The selections made in the job application portal are used for anonymized statistics that all state employers include in their annual reports. More information about gender equality initiatives at UiO can be found here. We offer Salary in position as PhD Research Fellow, position code 1017 in salary range NOK 550 800 – 595 000, depending on competence and experience. From the salary, 2% is deducted in statutory contributions to the State Pension Fund A unique research environment with multiple opportunities to develop research themes at the forefront of modern science A friendly, inclusive, and collaborative international working environment that values diverse perspectives Access to a strong network of top-level national and international collaborators A reliable and generous pension agreement, along with strong public benefits Comprehensive welfare schemes supporting both personal and professional well-being Full access to public health services through membership in the National Insurance Scheme A vibrant academic environment with an active and supportive research community Structured career development programmes at the faculty level and an individual professional development plan throughout the postdoctoral period Mentoring and support structures tailored to early-career researchers Flexible working conditions, with understanding for different life situations and family responsibilities Research mobility funds supporting short research stays and international collaboration Family-friendly surroundings in Oslo and Tromsø, with rich opportunities for culture, nature, and outdoor activities A clear institutional commitment to gender equality and diversity, with dedicated initiatives and networks for women in science Integreat provides a researcher training programme, INTREF - the Integreat Young Researchers’ Forum - a fellow driven initiative offering scientific training, transferable skills workshops, mentorship, research visits, and career development support. Application Your application should include: Cover letter – describe your motivation for pursuing a PhD at Integreat and in Norway, with particular emphasis on your research interests and academic achievements to date relevant for this position and the project CV (summarizing education, positions and academic work - scientific publications) Copies of the original bachelor’s and master’s degree diploma, and transcripts of records Documentation of English proficiency when required List of publications and academic work that the applicant wishes to be considered by the evaluation committee, if any The master thesis or at least some finished chapters of the master thesis Names and contact details of 2-3 references (name, relation to candidate, e-mail and telephone number) For applicants who have not yet completed their master’s degree: a draft of the master’s thesis and a statement from their supervisor or institution confirming the expected date of degree completion. Application with attachments must be submitted via our recruitment system Jobbnorge, click "Apply for the position". Foreign applicants should attach an official explanation of their University's grading system. When applying for the position, we ask you to retrieve your education results from Vitnemålsportalen.no . If your education results are not available through Vitnemålsportalen, we ask you to upload copies of your transcripts or grades. Please note that all documentation must be in English or a Scandinavian language. General information The best qualified candidates will invited for interviews. Please note that we are unable to add documentation after the application deadline. As the deadline applies equally to all applicants, this policy ensures equal treatment throughout the recruitment process. If you experience difficulties uploading documents in Jobbnorge, please contact their customer service: Phone: +47 75 54 22 20 or Email: support@grade.com The University of Oslo has a transfer agreement with all employees to secure the rights to all research results and related intellectual property. Please refer to Regulations for the Act on universities and colleges chapter 3 (Norwegian), Guidelines concerning appointment to recruitment positions at UiO and Regulations for the degree of Philosophiae Doctor (PhD) at the University of Oslo. Applicant lists can be published in accordance with Norwegian Freedom of Information Act § 25. When you apply for a position with us, your name will appear on the public applicant list. It is possible to request to be excluded from this list. You must justify why you want an exemption from publication and we will then decide whether we can grant your request. If we cannot, you will hear from us. University of Oslo The University of Oslo is Norway’s oldest and highest ranked educational and research institution, with 26 500 students and 7 200 employees. With its broad range of academic disciplines and internationally recognised research communities, UiO is an important contributor to society. Integreat – Norwegian Centre for Knowledge-driven Machine Learning - Integreat is a Centre of Excellence, funded by the Research council of Norway. Integreat has two branches, one in Oslo (University of Oslo, UiO) and one in Tromsø (UiT The Arctic University of Norway). Machine learning is the mathematical and computational engine of Artificial Intelligence (AI), and therefore a fundamental force of technological progress in our increasingly digital, data- and algorithm-driven world. Integreat develops theories, methods, models and algorithms that integrate general and domain-specific knowledge with data, laying the foundations of next generation machine learning. This will be done by combining the mathematical and computational cultures, and the methodologies and theories, of statistics, logic, language technologies, ethics and machine learning, in new and unique ways. Focus of Integreat is to develop ground-breaking methods and theories, and by this solving fundamental problems in science, technology, health and society. Integreat draws on the research strengths of researchers and students from the departments of Mathematics, Informatics, Philosophy, and the Oslo Centre for Biostatistics and Epidemiology at UiO, the Norwegian Computing Centre (NR) and the ML group at UiT, with members from the departments of Physics and Technology, Mathematics and Statistics, and Computer Science.