Manager, ML Agent Evaluation

Make your mark at Comcast -- a Fortune 30 global media and technology company. From the connectivity and platforms we provide, to the content and experiences we create, we reach hundreds of millions of customers, viewers, and guests worldwide. Become part of our award-winning technology team that turns big ideas into cutting-edge products, platforms, and solutions that our customers love. We create space to innovate, and we recognize, reward, and invest in your ideas, while ensuring you can proudly bring your authentic self to the workplace. Join us. You'll do the best work of your career right here at Comcast. (In most cases, Comcast prefers to have employees on-site collaborating unless the team has been designated as virtual due to the nature of their work. If a position is listed with both office locations and virtual offerings, Comcast may be willing to consider candidates who live greater than 100 miles from the office for the remote option.)
Job Summary
The Agent Evaluation team is responsible for testing whether AI agents return the correct and expected responses. We build the framework, metrics, and test cases that validate agent behavior, accuracy, and reliability before release. Our goal is to ensure agents perform consistently and meet product and user expectations.
Job Description
Role Summary:
The Manager, Agent Evaluation will lead the team responsible for building and scaling the evaluation framework that tests whether AI agents return accurate, reliable, and expected responses across real-world scenarios.
Key Responsibilities:

• Lead and grow a team focused on agent and model evaluation
• Define the strategy, roadmap, and standards for agent testing and validation
• Oversee development of metrics, benchmarks, and testing frameworks to measure response quality, accuracy, safety, and performance
• Ensure evaluation coverage aligns with product, UX, and business requirements
• Partner closely with Product, Engineering, Research, and Platform teams to integrate evaluation into the development lifecycle
• Drive experimentation and continuous improvement of evaluation methodologies
• Establish reporting mechanisms to clearly communicate evaluation results and trade-offs to leadership
• Implement best practices for model versioning, monitoring, and release validation
• Stay current with advancements in LLMs, AI agents, and evaluation techniques

Required Skills:

• Strong foundation in machine learning fundamentals and applied ML systems
• Hands-on experience with model and agent evaluation methodologies
• Familiarity with LLMs, AI agents, and prompt-driven systems
• Proficiency in Python and modern ML frameworks (e.g., PyTorch, TensorFlow)
• Experience defining metrics, benchmarks, and experimentation frameworks
• Solid understanding of MLOps practices, including model versioning, monitoring, and CI/CD
• Ability to collaborate effectively with product, platform, and research teams
• Clear communicator of technical trade-offs, evaluation insights, and results

Disclaimer:

• This information has been designed to indicate the general nature and level of work performed by employees in this role. It is not designed to contain or be interpreted as a comprehensive inventory of all duties, responsibilities and qualifications.

Comcast is an equal opportunity workplace. We will consider all qualified applicants for employment without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability, veteran status, genetic information, or any other basis protected by applicable law.
Skills:
Metrics Reporting; Machine Learning (ML); Natural Language Processing (NLP); Cross-Functional Collaboration; Large Language Models (LLMs); Python (Programming Language); AI Frameworks
Salary:
Primary Location Pay Range: $183,063.62 - $274,595.42
Comcast intends to offer the selected candidate base pay within this range, dependent on job-related, non-discriminatory factors such as experience. The application window is 30 days from the date job is posted, unless the number of applicants requires it to close sooner or later.
Base pay is one part of the Total Rewards that Comcast provides to compensate and recognize employees for their work. Most sales positions are eligible for a Commission under the terms of an applicable plan, while most non-sales positions are eligible for a Bonus. Additionally, Comcast provides best-in-class Benefits to eligible employees. We believe that benefits should connect you to the support you need when it matters most, and should help you care for those who matter most. That?s why we provide an array of options, expert guidance and always-on tools, that are personalized to meet the needs of your reality ? to help support you physically, financially and emotionally through the big milestones and in your everyday life. Please visit the compensation and benefits summary on our careers site for more details.
Education
Master's Degree
While possessing the stated degree is preferred, Comcast also may consider applicants who hold some combination of coursework and experience, or who have extensive related professional experience.
Relevant Work Experience
5-7 Years