"Principal Engineer"

The Institutional Data Initiative (IDI) is a new research center working to advance society’s relationship with knowledge by expanding access to, and deepening our understanding of, the data that underpins AI. By collaborating with library, government, and academic institutions to publish their knowledge collections as AI training sets, IDI seeks to empower those institutions and the cultures they represent, build a foundational pipeline for academic inquiry of AI, and advance the state of the art for all builders of AI systems.

IDI’s work spans the AI data ecosystem—from digitization, data structuring, and metadata synthesis, to safety and security analysis, all the way through to benchmarking and the development of ethical and governance frameworks. Institutional collaboration forms the gateway to this work and IDI places a particular emphasis on opportunities with institutions that expand the cultural breadth of knowledge represented in the building blocks of AI.

At its core, IDI is a data practice around which other interdisciplinary work is convened. While theory and analysis are critical components of IDI’s work, our impact is a direct factor of our ability to ship novel data. As such, IDI’s workflows resemble those of a product studio. Our projects are time-bound and scopes are driven by ambition within time constraints. Our team structure is relatively flat and each member is expected to bring vision for their work and drive it through to completion. We prioritize interdisciplinary collaboration with academic contributors, both internal and external, as essential work that prevents the commodification of the data we help to publish.

The technical capabilities of our Principal Engineers define the depth of analysis and inquiry at IDI while developing and deploying repeatable methods and pipelines. The person in this role will have an ability to think creatively about extracting and manipulating data to unlock knowledge collections that have been stubbornly inaccessible, sometimes for centuries. Their understanding of machine learning and AI fundamentals will help identify areas of high impact and utilize models to facilitate this work. Each Principal Engineer must bring a unique set of skills and approaches to a team of engineers whose distinct capabilities complement the whole. This team works together to build an action plan for each corpus that takes it from uncharted territory to a well-defined map that others can traverse.

Beyond data, Principal Engineers also contribute to the building of community around IDI’s work to enable outside collaborators—fellow technologists, academics, students, cultural stakeholders—to expand our capabilities, capacity, and perspectives. IDI operates within Harvard and alongside the Library Innovation Lab, the Berkman Klein Center, and the Applied Social Media Lab; engaging these communities, among others, is critical to delivering on our mission.

As a Principal Engineer, you will:

• Develop, refine and evaluate methods for analyzing and augmenting corpora.
• Research, train and evaluate machine learning models.
• Research, adjust and evaluate natural language processing techniques.
• Write and contribute to open-source software.
• Conduct an ongoing technology and research watch in areas pertinent to IDI’s focus, including: generative AI, natural language processing, digital preservation and open knowledge.
• Provide technical leadership and guidance to both your team members and your project peers.
• Help build and lead development of multiple discrete projects at once.
• Draft technical and scientific communications outlining datasets and novel methodologies developed in the course of your work.
• Be a technological ambassador for the research center. Help build and engage with the broader academic and AI communities including the Harvard student population.
• Engage with partners to both share our work and explore new opportunities.