About the company
Our client is a fast-growing software company.
The role
Raydar is recruiting for this role on behalf of our client. Design and build the measurement systems used to judge the quality of machine learning outputs. You will define the scoring approach, assemble labeled reference data and run evaluation pipelines, then feed the findings back into model and product decisions.
What you'll do
- Define quality measures that capture what a good result looks like across a wide variety of inputs and edge cases.
- Build evaluation pipelines and harnesses that score output quality reliably and at scale.
- Create and curate labeled reference data so that evaluation results can be trusted.
- Connect evaluation findings back to model and product decisions.
- Present results in a way that gives the whole team clear, actionable answers.
Requirements
What we're looking for
- Proven experience building evaluations for LLM-based data extraction or data quality at scale.
- Experience creating the scoring approach and the evaluation system yourself, rather than only running an existing benchmark.
- Experience assembling human-labeled or verified reference data to compare model outputs against.
- Evaluation work that clearly influenced model or product decisions, visible in your work history.
- Strong Python skills.
- A solid grounding in statistics for choosing and interpreting quality measures.
- Confidence working on open-ended problems where there is no established way to measure success.
- A bias toward shipping, measuring and iterating quickly.
- Willingness to work on-site five days a week, now or after relocating before the start date.
Benefits
Compensation and benefits
- Base salary: USD 250,000 to 290,000 per year
- Equity
- Health, dental and vision coverage
- Life and disability insurance
- Retirement plan
- Paid time off
- Paid parental leave
Location and work model
- San Francisco, CA, United States or Toronto, ON, Canada
- On-site, five days a week
- Full-time
Skills Required
- Proven experience building evaluations for LLM-based data extraction or data quality at scale
- Experience creating scoring approaches and evaluation systems independently
- Experience assembling human-labeled or verified reference data for model-output comparison
- Evaluation work that influenced model or product decisions
- Strong Python skills
- Solid grounding in statistics for selecting and interpreting quality measures
- Confidence working on open-ended problems without established success measures
- Willingness to work on-site five days per week in San Francisco or Toronto, either currently or after relocating
What We Do
Raydar is a talent acquisition and business consulting firm that connects world-class and emerging talent with growing organizations. It supports companies through team development, strategic hiring, and customized growth solutions, helping clients recruit roles such as engineers, product managers, executives, legal counsel, and quantitative traders. Raydar focuses on understanding each organization’s needs, culture, and long-term goals to build high-impact teams.
.png)
