AI Auditing Engineer (Temporary) - Penn Center for Media, Technology, and Democracy - Penn Engineering

University Overview

The University of Pennsylvania, the largest private employer in Philadelphia, is a world-renowned leader in education, research, and innovation. This historic, Ivy League school consistently ranks among the top 10 universities in the annual U.S. News & World Report survey. Penn has 12 highly-regarded schools that provide opportunities for undergraduate, graduate and continuing education, all influenced by Penn's distinctive interdisciplinary approach to scholarship and learning. As an employer Penn has been ranked nationally on many occasions with the most recent award from Forbes who named Penn one of America's Best Large Employers in 2023.

Penn offers a unique working environment within the city of Philadelphia. The University is situated on a beautiful urban campus, with easy access to a range of educational, cultural, and recreational activities. With its historical significance and landmarks, lively cultural offerings, and wide variety of atmospheres, Philadelphia is the perfect place to call home for work and play.

Posted Job Title

AI Auditing Engineer (Temporary) - Penn Center for Media, Technology, and Democracy - Penn Engineering

Job Profile Title

Temporary Employee - Non-Exempt

Job Description Summary

The Penn Center for Media, Technology, and Democracy (Penn MEDIATED) works to advance the scientific understanding of the information ecosystem and leverage that research to strengthen the foundations of democracy.

Penn MEDIATED seeks a part-time AI Auditing Engineer to develop and productionize auditing and monitoring of large language models (LLMs). The AI Auditing Engineer will unify existing research prototypes from several codebases in different academic projects to create an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs. The tooling will be used to automatically and longitudinally monitor rates of refusal or soft censorship, political lean, sourcing information, and other features of LLM responses on political topics.

We seek an AI Auditing Engineer experienced in developing efficient and scalable data pipelines in Python, designing and managing NoSQL databases, working with cloud computing platforms, and in working with LLM APIs. Experience with and knowledge with natural language processing methods for LLM evaluations is also beneficial.

The AI Auditing Engineer will work closely with Danae Metaxa, Raj & Neera Singh Assistant Professor in the Department of Computer and Information Sciences. The position will be jointly supervised by Danae Metaxa and Alex Engler, Executive Director of the Penn Center on Media, Technology, and Democracy. This role can be fully remote or hybrid. This is a temporary role, and may be either part-time or full-time, depending on the applicants, with the expected term during summer 2026 at between 20-40 hours per week.

Job Description

Develop and deploy an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs.

Design and manage appropriate databases for LLM monitoring.

Advance research methods for LLM monitoring of political topics.

Qualifications

• Bachelor's degree in a relevant field of study
• Required proficiency in developing efficient and scalable data pipelines in Python.
• Required proficiency in designing and managing NoSQL databases.
• Required proficiency in working with cloud computing platforms.
• Preferred understanding or prior experience working with LLM APIs.
• Preferred understanding or experience with natural language processing methods for LLM evaluation.
• Preferred understanding or experience with cloud cost engineering and other aspects of DevOps, especially cost efficient use of LLMs.
• Preferred experience with data visualization in interactive dashboards.
• Interest in the political outputs and impacts of large language models (LLMs) is a bonus.

To Apply: Please Submit a Resume and Cover Letter