"PhD Position on Perceptual Foundation Models"

Do you have a background in deep learning and computer vision? Are you independent, creative and eager to take initiatives? Do you enjoy working in an international research group and interacting with colleagues with diverse backgrounds? The Informatics Institute is looking for an ambitious PhD student on perceptual foundation models. Your research is part of the Video & Image Sense Lab.

Recent breakthroughs in Artificial Intelligence resulted in the emergence of the first generation of foundation models that are capable of transfer across conditions and tasks. This opened entirely new ways to solve domain-specific problems by training transferable models and adapting those in a data- and compute-efficient manner to various problem settings. However, the perceptual possibilities are critically hindered by the language-focussed optimization of current foundation models, fundamentally limiting their ability for spatial reasoning, temporal logic, and operating in low-resource scenarios, which leads to shortcut learning and hallucination at test-time. This PhD project focuses on a new generation of perceptual foundation models by contributing advanced perceptual pre-training and fine-tuning algorithms.

What you will do: You will carry out research and development in the areas of perceptual foundation models, using advances in deep machine learning and computer vision. The goal is to invent, develop and evaluate novel methods for pre-training and fine-tuning of perceptual foundation models, that expand their sensing abilities, generalize at deployment, and do so in an efficient and sustainable way. Importantly, the models should transfer to specialist industrial use cases. The research is embedded in the Video & Image Sense lab at the University of Amsterdam, and you will actively collaborate with our partner TNO within the NWO Perspectief Foundation for Industry (FIND) project.

Tasks and responsibilities:

• Develop new deep learning, computer vision and multimodal learning methods for pre-training and fine-tuning perceptual foundation models.
• Actively collaborate within the FIND project and contribute to its use cases.
• Regularly present internally on your progress.
• Regularly present intermediate research at international conferences and workshops, publish them in proceedings and journals.
• Assist in teaching activities such as lab assistance and student supervision.
• Complete and defend a PhD thesis within the official appointment duration of four years.

Your profile: An MSc degree in Artificial Intelligence, (Applied) Mathematics/Physics, Computer Science, Engineering or related field. A strong background/knowledge in machine learning and computer vision, natural language processing is a plus. Solid mathematics foundations, especially statistics, calculus and linear algebra; Excellent programming skills, preferably in Python. Experience with AI/HPC supercomputing and running software on scale. You are highly motivated, independent, and creative. Strong communication, presentation and writing skills and excellent command of English. Prior publications in relevant machine learning or computer vision conferences or journals are advantageous.

This is what we offer: A temporary contract for 38 hours per week for the duration of 4 years (the initial contract will be for a period of 18 months and after satisfactory evaluation it will be extended for a total duration of 4 years). The preferred starting date is December 1st 2025. This should lead to a dissertation (PhD thesis). We will draft an educational plan that includes attendance of courses and (international) meetings. We also expect you to assist in teaching undergraduates and master students. The gross monthly salary, based on 38 hours per week and dependent on relevant experience, ranges between € 3,059 to € 3,881 (scale P). This does not include 8% holiday allowance and 8,3% year-end allowance. The UFO profile Promovendus is applicable. A favourable tax agreement, the ‘30% ruling’, may apply to non-Dutch applicants. The Collective Labour Agreement of Universities of the Netherlands is applicable. Besides the salary and a vibrant and challenging environment at Science Park we offer you multiple fringe benefits: 232 holiday hours per year (based on fulltime) and extra holidays between Christmas and 1 January; multiple courses to follow from our Teaching and Learning Centre; a complete educational program for PhD students; multiple courses on topics such as leadership for academic staff; multiple courses on topics such as time management, handling stress and an online learning platform with 100+ different courses; 7 weeks birth leave (partner leave) with 100% salary; partly paid parental leave; the possibility to set up a workplace at home; a pension at ABP for which UvA pays two third part of the contribution; the possibility to follow courses to learn Dutch; help with housing for a studio or small apartment when you’re moving from abroad.

Where you will work: The Faculty of Science has a student body of around 8,000, as well as 1,800 members of staff working in education, research or support services. Researchers and students at the Faculty of Science are fascinated by every aspect of how the world works, be it elementary particles, the birth of the universe or the functioning of the brain. The mission of the Informatics Institute (IvI) is to perform curiosity-driven and use-inspired fundamental research in Computer Science. The main research themes are Artificial Intelligence, Computational Science and Systems and Network Engineering. Our research involves complex information systems at large, with a focus on collaborative, data driven, computational and intelligent systems, all with a strong interactive component. The position is with Prof. dr. Cees Snoek, Professor, head of the Video & Image Sense lab (VIS lab), at the University of Amsterdam. VIS lab is a world-leading lab on Computer Vision and Machine Learning, and has over 30 PhD students, postdoctoral researchers and faculty members working on a broad variety of deep learning, computer vision, and foundation model subjects, like self-supervised learning, diffusion models, and test-time generalization for perception tasks like object detection, instance segmentation and activity recognition. The position is also embedded in the European ELLIS Network of Excellence in AI. We also anticipate regular visits to project partner TNO in the Hague.