Sr. Product Mgr, AI & Human Eval Prog

At eBay, we're more than a global ecommerce leader — we’re changing the way the world shops and sells. Our platform empowers millions of buyers and sellers in more than 190 markets around the world. We’re committed to pushing boundaries and leaving our mark as we reinvent the future of ecommerce for enthusiasts.

Our customers are our compass, authenticity thrives, bold ideas are welcome, and everyone can bring their unique selves to work — every day. We're in this together, sustaining the future of our customers, our company, and our planet.

Join a team of passionate thinkers, innovators, and dreamers — and help us connect people and build communities to create economic opportunity for all.

About the Team and the Role:

The Senior Product Manager will be the connective tissue between category managers, merchandisers, and domain guides on one side, and the scientists and engineers building AI models on the other. This role requires a deep understanding of how AI models are built and trained — including how data curation, human judgment, and iterative feedback shape model behavior — and the ability to actively contribute to that process. Your impact depends on execution, trust, and the ability to coordinate complex cross-functional work across a matrix organization.

What you will accomplish:

Run Human Evaluation and Annotation Programs

• Design and lead structured programs to evaluate AI model outputs across ranking, classification, generation, and extraction tasks

• Coordinate distributed annotation teams across timezones — running workflows, maintaining quality consistency, and ensuring continuous coverage and on-time delivery. Handle relationships with internal and external evaluator pools

• Translate evaluation results into clear, actionable requirements for model and product teams
   

Lead AI Quality Monitoring Programs

• Track quality metrics, surface issues proactively, and run structured resolution processes to ensure AI-driven experiences consistently meet the bar

• Ensure quality signals from the live site feed back into model development priorities.

• Review what ships, ask hard questions, and hold teams accountable when AI-driven experiences fall short

• Help define what “good data” looks like for each focus category: coverage, freshness, labeling schema, and quality thresholds

• Build the end-to-end processes to meet those standards — from scoping and sourcing through annotation, QA, and delivery — in partnership with science teams and business partners

• Define quality standards for training and evaluation data in partnership with business partners and science teams — and build the processes to meet those standards consistently.​

What you will bring:

• 5+ years of PM experience with deep exposure to AI/ML or technical platform products

• Experience leading annotation or evaluation programs, including working with distributed annotator pools, external vendors, or crowdsourced labeling at scale

• Analytical and data science fluency — comfortable writing SQL queries to explore datasets, track program metrics, and validate quality signals independently

• Ability to work alongside Applied Scientists — understands how DS workflows operate, can scope analytical work clearly, and knows when to roll up their sleeves. 

• Working knowledge of how AI models are built and trained — including the role of data, human judgment, and evaluation in shaping model behavior

• Direct exposure to LLM evaluation — preference data, benchmarking, red-teaming, or human feedback collection is a plus

• Experience with annotation vendors or crowdsourced labeling programs is a plus

• Familiarity with AI observability, quality monitoring, or anomaly detection in production is a plus

Additional Details

The base pay range for this position is expected in the range below:

Base pay offered may vary depending on multiple individualized factors, including location, skills, and experience. The total compensation package for this position may also include other elements, including a target bonus and restricted stock units (as applicable) in addition to a full range of medical, financial, and/or other benefits (including RRSP eligibility, various paid time off benefits, such as PTO and parental leave). Details of participation in these benefit plans will be provided if an employee receives an offer of employment.

This job posting relates to an existing vacancy within eBay.

eBay is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, sex, sexual orientation, gender identity, and disability, or other legally protected status. If you have a need that requires accommodation, please contact us at [email protected]. We will make every effort to respond to your request for accommodation as soon as possible. View our accessibility statement to learn more about eBay's commitment to ensuring digital accessibility.

We use cookies to enhance your experience and may use AI tools for administrative tasks in the hiring process. To learn how we handle your personal data and use AI responsibly, please visit our Talent Privacy Notice, Privacy Center and AI Hiring Guidelines.