Agentic AI Evaluation Engineer

Make your mark at Comcast -- a Fortune 30 global media and technology company. From the connectivity and platforms we provide, to the content and experiences we create, we reach hundreds of millions of customers, viewers, and guests worldwide. Become part of our award-winning technology team that turns big ideas into cutting-edge products, platforms, and solutions that our customers love. We create space to innovate, and we recognize, reward, and invest in your ideas, while ensuring you can proudly bring your authentic self to the workplace. Join us. You'll do the best work of your career right here at Comcast. (In most cases, Comcast prefers to have employees on-site collaborating unless the team has been designated as virtual due to the nature of their work. If a position is listed with both office locations and virtual offerings, Comcast may be willing to consider candidates who live greater than 100 miles from the office for the remote option.)
Job Summary
The Agent Evaluation team is responsible for testing whether AI agents return the correct and expected responses. We build the framework, metrics, and test cases that validate agent behavior, accuracy, and reliability before release. Our goal is to ensure agents perform consistently and meet product and user expectations.
Job Description
Job Description:

• Design and develop agent evaluation pipelines across development, staging, and production environments
• Define and standardize evaluation metrics and benchmarks for conversational AI quality (accuracy, relevance, CX, safety)
• Build automated and human-in-the-loop evaluation systems to assess agent performance
• Manage and curate evaluation datasets, test sets, and annotation workflows
• Enable continuous evaluation and monitoring of agents in production
• Integrate evaluation into CI/CD pipelines to support safe and efficient releases
• Conduct experiments, A/B testing, and case studies to drive improvements in agent quality
• Partner with engineering, and product teams to deliver high-quality AI solutions
• Create technical documentation and drive best practices across teams
• Mentor junior engineers and contribute to team growth

Preferred skills:

• Experience in customer support AI or chatbot platforms
• Understanding of responsible AI (bias, fairness, hallucination mitigation)

Disclaimer: This information has been designed to indicate the general nature and level of work performed by employees in this role. It is not designed to contain or be interpreted as a comprehensive inventory of all duties, responsibilities and qualifications.
Comcast is an equal opportunity workplace. We will consider all qualified applicants for employment without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability, veteran status, genetic information, or any other basis protected by applicable law.
Skills:
CI/CD; Machine Learning (ML); Large Language Models (LLMs); Curious Mindset; AI Agents; Benchmarking; Evaluation Metrics
Salary:
Primary Location Pay Range: $142,651.46 - $213,977.19
Comcast intends to offer the selected candidate base pay within this range, dependent on job-related, non-discriminatory factors such as experience. The application window is 30 days from the date job is posted, unless the number of applicants requires it to close sooner or later.
Base pay is one part of the Total Rewards that Comcast provides to compensate and recognize employees for their work. Most sales positions are eligible for a Commission under the terms of an applicable plan, while most non-sales positions are eligible for a Bonus. Additionally, Comcast provides best-in-class Benefits to eligible employees. We believe that benefits should connect you to the support you need when it matters most, and should help you care for those who matter most. That's why we provide an array of options, expert guidance and always-on tools, that are personalized to meet the needs of your reality - to help support you physically, financially and emotionally through the big milestones and in your everyday life. Please visit the compensation and benefits summary on our careers site for more details.
Education
Bachelor's Degree
While possessing the stated degree is preferred, Comcast also may consider applicants who hold some combination of coursework and experience, or who have extensive related professional experience.
Relevant Work Experience
5-7 Years