Agentic AI evaluation: Build computational infrastructure and approaches to evaluate agentic AI systems, emphasizing single-agent systems at first. This involves capability detection, identifying and performing appropriate evaluations, handling benchmark and metric administration, managing controlled experiments, and reviewing performance across agent, model, and system layers.

This project will develop mathematical models, optimization algorithms, and a pipeline. It will evaluate how workflows and roles change when AI is deployed. Across both projects, the successful candidate will provide technical contributions and play a key collaboration role.

The candidate will drive research in partnership with campus faculty and other technical researchers from the CMU ecosystem.

Core responsibilities include:

• Develop parts of a capability-aware evaluation framework for agentic AI systems, including capability detection, test recommendation, controlled execution, scoring, and reporting.

• Develop and maintain a versioned registry of AI evaluation metrics, benchmarks, datasets, scoring procedures, and reference results.

• Implement interfaces and research infrastructure for accessing and delivering evaluations, including MCP-based services and integration with established AI evaluation and sandboxing tools.

• Conduct benchmark experiments at the component, model, and end-to-end system levels, including controlled comparisons in which models are varied while the agent harness is held fixed.

• Assess the consistency and repeatability of evaluation methods by reproducing benchmarks, conducting empirical validation, and analyzing experimental outcomes.

Adaptability, excellence, and passion are vital qualities within Carnegie Mellon University. We are in search of a team member who can effectively interact with a varied population of internal and external partners at a high level of integrity. We are looking for someone who shares our values and who will support the mission of the university through their work.

You should demonstrate:

• Works well in a team with students and faculty

• Ability to conduct work independently and jointly with faculty supervisors

• Strong project management skills

• Strong problem-solving, reasoning, and analytical skills

• Strong verbal and written communication skills

Qualifications:

Master's degree required

A combination of education and relevant experience from which comparable knowledge is demonstrated may be considered.

Graduate-level education in computer science, artificial intelligence, machine learning, operations research, statistics, applied mathematics, engineering, or a related field. Foundational understanding of LLMs and agentic AI systems, optimization, statistics, and machine learning, together with strong Python engineering skills.

At least one year of experience in any of these areas is preferred: LLM or agentic AI systems, AI evaluation and benchmarking, optimization modeling, simulation, statistical analysis, or AI-assisted software development. Graduate research, internships, or significant project work can meet this criterion.

Demonstrated experience developing computational models, research software, or experimental pipelines using Python.

Requirements:

• Successful completion of a pre-employment background check

Additional Information

Sponsorship: Applicants for this position must be currently legally authorized to work for CMU in the United States. CMU will not sponsor or take over the sponsorship of an employment visa for this opportunity. Carnegie Mellon is not a qualifying employer for the STEM OPT benefit: only the 12-month OPT may be used to work at Carnegie Mellon.

Fixed Term: This is a fixed-term position with an estimated duration of approximately seven (7) months ending on May 1, 2027.

Typical Work Schedule: Monday-Friday from 8:30-5:00pm. An alternate work schedule allowing for evening and weekend hours will be required on occasion.

Joining the CMU team opens the door to an array of exceptional benefits.

Benefits eligible employees enjoy a wide array of benefits including comprehensive medical, prescription, dental, and vision insurance as well as a generous retirement savings program with employer contributions. Unlock your potential with tuition benefits, take well-deserved breaks with ample paid time off and observed holidays, and rest easy with life and accidental death and disability insurance.

Additional perks include a free Pittsburgh Regional Transit bus pass, access to our Family Concierge Team to help navigate childcare needs, fitness center access, and much more!

For a comprehensive overview of the benefits available, explore our Benefits page.

At Carnegie Mellon, we value the whole package when extending offers of employment. Beyond credentials, we evaluate the role and responsibilities, your valuable work experience, and the knowledge gained through education and training. We appreciate your unique skills and the perspective you bring. Your journey with us is about more than just a job; it's about finding the perfect fit for your professional growth and personal aspirations.

Are you interested in an exciting opportunity with an exceptional organization?! Apply today!

Location

Pittsburgh, PA

Job Function

Researchers

Position Type

Staff - Fixed Term (Fixed Term)

Full Time/Part time

Full time

Pay Basis

Salary

More Information:

Agentic AI evaluation: Build computational infrastructure and approaches to evaluate agentic AI systems, emphasizing single-agent systems at first. This involves capability detection, identifying and performing appropriate evaluations, handling benchmark and metric administration, managing controlled experiments, and reviewing performance across agent, model, and system layers.

This project will develop mathematical models, optimization algorithms, and a pipeline. It will evaluate how workflows and roles change when AI is deployed. Across both projects, the successful candidate will provide technical contributions and play a key collaboration role.

The candidate will drive research in partnership with campus faculty and other technical researchers from the CMU ecosystem.

Core responsibilities include:

• Develop parts of a capability-aware evaluation framework for agentic AI systems, including capability detection, test recommendation, controlled execution, scoring, and reporting.

• Develop and maintain a versioned registry of AI evaluation metrics, benchmarks, datasets, scoring procedures, and reference results.

• Implement interfaces and research infrastructure for accessing and delivering evaluations, including MCP-based services and integration with established AI evaluation and sandboxing tools.

• Conduct benchmark experiments at the component, model, and end-to-end system levels, including controlled comparisons in which models are varied while the agent harness is held fixed.

• Assess the consistency and repeatability of evaluation methods by reproducing benchmarks, conducting empirical validation, and analyzing experimental outcomes.

Adaptability, excellence, and passion are vital qualities within Carnegie Mellon University. We are in search of a team member who can effectively interact with a varied population of internal and external partners at a high level of integrity. We are looking for someone who shares our values and who will support the mission of the university through their work.

You should demonstrate:

• Works well in a team with students and faculty

• Ability to conduct work independently and jointly with faculty supervisors

• Strong project management skills

• Strong problem-solving, reasoning, and analytical skills

• Strong verbal and written communication skills

Qualifications:

Master's degree required

A combination of education and relevant experience from which comparable knowledge is demonstrated may be considered.

Graduate-level education in computer science, artificial intelligence, machine learning, operations research, statistics, applied mathematics, engineering, or a related field. Foundational understanding of LLMs and agentic AI systems, optimization, statistics, and machine learning, together with strong Python engineering skills.

At least one year of experience in any of these areas is preferred: LLM or agentic AI systems, AI evaluation and benchmarking, optimization modeling, simulation, statistical analysis, or AI-assisted software development. Graduate research, internships, or significant project work can meet this criterion.

Demonstrated experience developing computational models, research software, or experimental pipelines using Python.

Requirements:

• Successful completion of a pre-employment background check

Additional Information

Sponsorship: Applicants for this position must be currently legally authorized to work for CMU in the United States. CMU will not sponsor or take over the sponsorship of an employment visa for this opportunity. Carnegie Mellon is not a qualifying employer for the STEM OPT benefit: only the 12-month OPT may be used to work at Carnegie Mellon.

Fixed Term: This is a fixed-term position with an estimated duration of approximately seven (7) months ending on May 1, 2027.

Typical Work Schedule: Monday-Friday from 8:30-5:00pm. An alternate work schedule allowing for evening and weekend hours will be required on occasion.

Joining the CMU team opens the door to an array of exceptional benefits.

Benefits eligible employees enjoy a wide array of benefits including comprehensive medical, prescription, dental, and vision insurance as well as a generous retirement savings program with employer contributions. Unlock your potential with tuition benefits, take well-deserved breaks with ample paid time off and observed holidays, and rest easy with life and accidental death and disability insurance.

Additional perks include a free Pittsburgh Regional Transit bus pass, access to our Family Concierge Team to help navigate childcare needs, fitness center access, and much more!

For a comprehensive overview of the benefits available, explore our Benefits page.

At Carnegie Mellon, we value the whole package when extending offers of employment. Beyond credentials, we evaluate the role and responsibilities, your valuable work experience, and the knowledge gained through education and training. We appreciate your unique skills and the perspective you bring. Your journey with us is about more than just a job; it's about finding the perfect fit for your professional growth and personal aspirations.

Are you interested in an exciting opportunity with an exceptional organization?! Apply today!

Location

Pittsburgh, PA

Job Function

Researchers

Position Type

Staff - Fixed Term (Fixed Term)

Full Time/Part time

Full time

Pay Basis

Salary

More Information: