Research Scientist - Evaluations

Reposted 3 Months Ago
San Francisco, CA, USA
In-Office
310K-500K Annually
Mid level
Artificial Intelligence • Software
The Role
The AI Behavior Engineer will measure and shape AI model behaviors, collaborating with stakeholders to conduct evaluations and adapt methods for various contexts.
Summary Generated by Built In
Salary range: $310,000 - $500,000/year + benefits

Description: Transluce is a fast-moving nonprofit research lab building the public tech stack for AI evaluation and oversight. We partner with frontier AI developers, governments, and civil society organizations to evaluate how AI systems behave, from misaligned agent swarms to child safety and mental health. Our research has uncovered surprising behaviors in frontier models, such as pre-release models fabricating actions they never took. Our evaluations directly underpin policy, including through collaborations with the EU AI Office and Common Sense Media.

About the role: We’re looking for research scientists to work on measuring AI model behaviors and developing new methodologies for evaluating AI systems. You will build independent evaluations of AI models and do research into key questions that guide that work: how can we discover problems with AI models before they cause negative impacts in the world? How can we leverage AI assistance to automate parts of the evaluation process?

You’ll help to set standards for our evaluation work across the board and work with engineers to translate research insights into reusable tools. For example, you might develop a new methodology for discovering interesting LLM behaviors from in-the-wild transcripts; collaborate with other teams to build a scalable pipeline for behavior discovery; and, publish your findings and tools to advance evaluation work here and in other organizations.

This is a high-autonomy role with a clear line of sight from your work to real-world impact. As part of a mission-focused non-profit, you’ll contribute to strengthening independent oversight of AI and advancing the science of AI evaluation.

Core responsibilities: Develop evaluation methods for measuring evolving AI model behaviors and put those methods into practice by building public evaluations. This includes:
  • Researching techniques for building realistic simulations or reproductions of AI deployment environments.
  • Developing high quality public evaluations to raise the bar for evaluation work across the independent oversight ecosystem.
  • Informing and contributing to the development of tools for automating parts of the evaluation process.
  • Shaping Transluce’s approach to evaluations and investigating fundamental questions in evaluation science.

Qualities of a strong candidate:
  • Hands-on experience designing and running AI evaluations, particularly behavioral or interactive evaluations (multi-turn, agentic, or red-teaming contexts).
  • Research track record that includes careful empirical or methods work, ideally in evaluation science.
  • Experience working in cross functional teams of researchers, engineers, and domain experts.
  • Strong communication skills, low ego, openness to giving and receiving feedback.

Preferred qualifications (nice to have):
  • Engineering skills: implementing well-designed software, user testing, and iterating on feedback.
  • Past work on building agent harnesses or LLM-powered automated research systems.

We are located in San Francisco and enthusiastic to work together in-person. We are open to sponsoring international visas.

Skills Required

  • Experience designing and running AI evaluations, especially behavioral or interactive evaluations
  • Strong engineering instincts and judgment for deliverables
  • Experience in customer-facing, consulting, or forward-deployed roles
  • Experience running evaluations at scale or in a production context
  • Ability to balance needs of researchers and domain experts
  • Strong communication skills and openness to feedback
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: San Francisco, California
20 Employees
Year Founded: 2024

What We Do

Transluce is an independent research lab that builds open, scalable technology for understanding AI systems and steering them in the public interest. Transluce means to shine light through something to reveal its structure. Today’s complex AI systems are difficult to understand—not even experts can reliably predict their behavior once deployed. Given AI's extraordinary consequences on society, we need scalable and open analyses of the capabilities and risks of AI systems. We are building open source, AI-driven tools to understand and analyze AI systems. We will apply these tools to open-weight models, so the world can vet our analyses and improve their reliability. Once our technology has been vetted, we will work with frontier AI labs and governments to ensure that internal assessments reach the same standards as our publicly vetted procedures. Email: [email protected]

Similar Jobs

Lila Sciences Logo Lila Sciences

Scientist

Artificial Intelligence • Software
In-Office
3 Locations
224 Employees
176K-304K Annually

Apollo Research Logo Apollo Research

Scientist

Artificial Intelligence • Machine Learning • Security • Software
In-Office
2 Locations
160K-590K Annually

Scale AI Logo Scale AI

Scientist

Artificial Intelligence • Big Data • Machine Learning
In-Office
3 Locations
523 Employees
166K-207K Annually

Epsilon Health Logo Epsilon Health

Scientist

Healthtech • Software
In-Office
San Francisco, CA, USA
6 Employees

Similar Companies Hiring

Revel Thumbnail
Aerospace • Hardware • Robotics • Software
Marina Del Rey, California
70 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
Vega Thumbnail
Artificial Intelligence • Automotive • Insurance • Transportation
US
65 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account