The Role
Investigate honesty, alignment, and unexpected behaviors in frontier AI systems. Develop automated evaluations, analysis workflows, and LLM-as-a-judge pipelines; analyze agent transcripts and large datasets; identify anomalies; conduct rapid investigations; and communicate findings through rigorous reports for technical, policy, government, and AI lab audiences.
Summary Generated by Built In
Salary range: $250,000 - $500,000/year + benefits
Description: Transluce is a fast-moving nonprofit research lab building the public tech stack for AI evaluation and oversight. We have contributed foundational research to the study of AI agents and their behaviors, and we put the results where they can change decisions: in the hands of labs, policymakers, and the public.
About the role: We are looking for a frontline evaluator to investigate the honesty and alignment of frontier AI systems. You will develop and run rigorous automated evaluations, conduct novel analyses of massive datasets, and surface behaviors of interest, writing up your results for technical, policy, and lab audiences. Example behaviors of interest include misreporting results, falsely claiming success, evaluation awareness, and memetic effects within AI swarms. Your work will uncover risks that might otherwise go unnoticed, turning observations into evidence for the public to decide how AI is built, deployed, and governed.
As an early member of a highly collaborative team, you will learn and grow quickly, and work with our governance and infrastructure teams to scale your impact and technical reach. Your work will be with frontier labs, with governments, and on key topics of public interest — for example, rapid-response investigations of major incidents, public reports on frontier model behavior, and serving as an independent evaluator for governments such as the EU. It may include embedded evals within frontier AI labs as those opportunities arise. As we further develop this approach to evaluating frontier AI systems, we expect the role to evolve.
Core responsibilities:
- Develop and conduct evaluations to investigate misalignment and unexpected behaviors in AI systems
- Write code to quickly build and run evaluation and analysis workflows, including data-science tools, LLM-as-a-judge pipelines, and evaluation environments
- Analyze agent transcripts, datasets, and evaluation results, finding notable patterns or unexpected behaviors to investigate
- Perform investigations under time and access constraints, iterating quickly while validating conclusions
- Translate findings into clear, rigorous written reports
- Collaborate with teammates and external technical stakeholders to conduct evaluations, communicate progress, and share relevant findings
What we're looking for:
- Strong empirical judgment to extract meaningful findings from data, design follow-up experiments, and identify anomalies.
- Proficiency in Python to implement data analysis, experiments, and evaluation tooling.
- Ability to turn an ambiguous concern into a testable question.
- Operational resourcefulness, adaptability, and sound prioritization in the face of incomplete information.
- Ability to iterate quickly and balance between scrappiness and thoroughness based on the impact needs of a project.
- Strong communication skills, including the ability to clearly explain technical findings to various readers.
- Collaborative orientation, low ego, and openness to both giving and receiving feedback.
Preferred qualifications (nice to have):
- Experience with evaluation (e.g. model-based judges or agent evaluations), red-teaming, or rapid-response and incident-analysis work on frontier systems
- A track record of insightful empirical investigations shared through reports, blog posts, research, or independent projects.
- Practical understanding of how model training and deployment can affect behavior and evaluation results.
- Experience in customer-facing, consulting, or forward-deployed roles translating ambiguous partner needs into concrete deliverables.
- Experience delivering technical work under tight deadlines or in unfamiliar or constrained environments.
We are hiring at all levels of experience and would encourage those enthusiastic about the role who do not meet all of the qualifications to apply. We are located in San Francisco and excited to work together in-person. We are open to sponsoring international visas.
Skills Required
- Strong empirical judgment to extract meaningful findings from data, design follow-up experiments, and identify anomalies.
- Proficiency in Python for data analysis, experiments, and evaluation tooling.
- Ability to turn ambiguous concerns into testable questions.
- Operational resourcefulness, adaptability, and sound prioritization with incomplete information.
- Ability to iterate quickly and balance scrappiness with thoroughness based on project impact.
- Strong communication skills for explaining technical findings to varied audiences.
- Collaborative orientation, low ego, and openness to giving and receiving feedback.
- Experience with model-based judges, agent evaluations, red-teaming, or rapid-response and incident-analysis work on frontier systems.
- Track record of empirical investigations shared through reports, blog posts, research, or independent projects.
- Practical understanding of how model training and deployment affect behavior and evaluation results.
- Experience in customer-facing, consulting, or forward-deployed roles translating ambiguous partner needs into deliverables.
- Experience delivering technical work under tight deadlines or in unfamiliar or constrained environments.
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Transluce is an independent research lab that builds open, scalable technology for understanding AI systems and steering them in the public interest. Transluce means to shine light through something to reveal its structure. Today’s complex AI systems are difficult to understand—not even experts can reliably predict their behavior once deployed. Given AI's extraordinary consequences on society, we need scalable and open analyses of the capabilities and risks of AI systems. We are building open source, AI-driven tools to understand and analyze AI systems. We will apply these tools to open-weight models, so the world can vet our analyses and improve their reliability. Once our technology has been vetted, we will work with frontier AI labs and governments to ensure that internal assessments reach the same standards as our publicly vetted procedures. Email: [email protected]
.png){kind=logo}
.jpg)
