About the company
Our client is an artificial intelligence company.
The role
Raydar is recruiting for this role on behalf of our client. Design and build test environments, evaluation harnesses and labeled datasets that measure how AI systems behave across multi-step tasks. The work combines reviewing system logs, spotting recurring errors and turning them into reusable test material. The role suits an engineer who wants to own problems from diagnosis through delivery.
What you'll do
- Construct simulated software environments that mimic the tools and workflows automated systems use.
- Build grading logic and evaluation systems that hold up when systems look for shortcuts.
- Create benchmarks whose scores track real-world results.
- Review interaction logs to find and measure previously unrecorded failure patterns.
- Turn findings into reusable datasets and test suites for internal improvement and external partners.
- Support data and training agreements alongside the founders while refining the evaluation tooling.
Requirements
What we're looking for
- One to seven years of relevant experience.
- Hands-on work with training data, post-training, benchmarks or evaluation for machine learning models.
- Judgment to tell which failures or data points are meaningful and valuable.
- A track record of building or contributing to benchmarks, datasets or environments used to train or test models.
- Skill in replicating common business software well enough that automated agents can interact with it reliably.
- Understanding of grader design, evaluation pitfalls and the ways systems game metrics.
- Prior work at a data vendor is relevant if it produced training material or model assessments, not expert labeling alone.
- A concrete example of building a benchmark, dataset, environment or evaluation system.
- Commercial awareness of what data and environments are worth.
- Ability to explain technical decisions and trade-offs in an interview.
Bonus points
- Found a real flaw in a widely used benchmark.
- Caught a model gaming a grader and fixed the grader.
- Built environments or benchmarks a leading AI lab used for training or evaluation.
- Startup founding experience or participation in a selective fellowship.
- High-level competitive achievement in mathematics, programming, debate or games.
- Understanding of how benchmarks and datasets support commercial model evaluation.
Benefits
Compensation and benefits
- Base salary: USD 150,000 to 300,000
- Equity
- Health, dental and vision coverage
- Retirement plan
- Flexible time off
Location and work model
- San Francisco, CA, United States
- On-site (in office)
- Full-time
Skills Required
- One to seven years of relevant experience
- Hands-on experience with training data, post-training, benchmarks, or machine learning model evaluation
- Ability to identify meaningful and valuable failures or data points
- Track record building or contributing to benchmarks, datasets, or environments used to train or test models
- Ability to replicate common business software for reliable automated-agent interaction
- Understanding of grader design, evaluation pitfalls, and metric gaming
- Relevant data-vendor experience producing training material or model assessments
- Concrete example of building a benchmark, dataset, environment, or evaluation system
- Commercial awareness of the value of data and environments
- Ability to explain technical decisions and trade-offs in an interview
- Identified a flaw in a widely used benchmark
- Detected and fixed a model gaming a grader
- Built environments or benchmarks used by a leading AI lab
- Startup founding experience or participation in a selective fellowship
- High-level competitive achievement in mathematics, programming, debate, or games
- Understanding of how benchmarks and datasets support commercial model evaluation
What We Do
Raydar is a talent acquisition and business consulting firm that connects world-class and emerging talent with growing organizations. It supports companies through team development, strategic hiring, and customized growth solutions, helping clients recruit roles such as engineers, product managers, executives, legal counsel, and quantitative traders. Raydar focuses on understanding each organization’s needs, culture, and long-term goals to build high-impact teams.
