Research Scientist

FAR.AI is a non-profit AI research institute dedicated to ensuring advanced AI is safe and beneficial for everyone. Our mission is to facilitate breakthrough AI safety research, advance global understanding of AI risks and solutions, and foster a coordinated global response.

Since our founding in July 2022, we've grown quickly to 20+ staff, producing 30 influential academic papers, and established the leading AI Safety events for research, and international cooperation. Our work is recognized globally, with publications at premier venues such as NeurIPS, ICML, and ICLR, and features in the Financial Times, Nature News, and MIT Technology Review.

We drive practical change through red-teaming with frontier model developers and government institutes. Additionally, we help steer and grow the AI safety field through developing research roadmaps with renowned researchers such as Yoshua Bengio, running FAR.Labs, an AI safety-focused co-working space in Berkeley housing 40 members, and supporting the community through targeted grants to technical researchers.

Our research team likes to move fast. We explore promising research directions in AI safety and scale up only those showing a high potential for impact. Unlike other AI safety labs that take a bet on a single research direction, FAR.AI aims to pursue a diverse portfolio of projects.

Our current focus areas include:

• Building a science of robustness (e.g. finding vulnerabilities in superhuman Go AIs)

• Finding more effective approaches to value alignment (e.g. training from language feedback)

• Advancing model evaluation techniques (e.g. inverse scaling and codebook features, and learned planning).

We also put our research into practice through red-teaming engagements with frontier AI developers, and collaborations with government institutes.

To build a flourishing field of AI safety research, we host targeted workshops and events, and operate a co-working space in Berkeley, called FAR.Labs. Our previous events include the International Dialogue for AI Safety that brought together prominent scientists (including 2 Turing Award winners) from around the globe, culminating in a public statement calling for global action on AI safety research and governance. We also host the semiannual Alignment Workshop with 150 researchers from academia, industry and government to learn about the latest developments in AI safety and find collaborators. For more information on FAR.AI’s activities, please visit our recent post.

We are seeking applications from potential Research Scientists who can:

• Take ownership of and accelerate existing AI Alignment research agendas.

• Develop their own exciting AI Alignment research agendas.

• Lead novel research projects where there may be unclear markers of progress and/or success.

• Contribute to the development of best practices for AI safety research at FAR.AI and in the broader community.

• Publish research findings and engage with the AI safety community.

We are excited by unconventional backgrounds.

You may have the following:

• New and under-explored AI Alignment idea(s).

• Experience leading and/or playing a senior role in research projects related to machine learning.

• Ability to effectively communicate novel methods and solutions to both technical and non-technical audiences.

• PhD or several years research experience in computer science, artificial intelligence, machine learning or statistics.

As Research Scientist you would lead AI safety research projects or make essential contributions to existing projects. Examples of ongoing projects at FAR.AI include:

• Scaling laws for prompt injections. Will advances in capabilities from increasing model and data scale help resolve prompt injections or “jailbreaks” in language models, or is progress in average-case performance orthogonal to worst-case robustness?

• Robustness of advanced AI systems. Explore adversarial training, architectural improvements and other changes to deep learning systems to improve their robustness. We are exploring this both in zero-sum board games and language models.

• Mechanistic interpretability for mesa-optimization. Develop techniques to identify internal planning in models to effectively audit the “goals” of models in addition to their external behavior.

• Red-teaming of frontier models. Apply our research insights to test for vulnerabilities and limitations of frontier AI models prior to deployment.

You could be an employee or an independent Contractor for FAR.AI, a 501(c)(3) research non-profit.

• Location: Both remote and in-person (Berkeley, CA) are possible. We sponsor visas for in-person employees, and can also hire remotely in most countries.

• Hours: Full-time (40 hours/week).

• Compensation: $100,000-$175,000/year depending on experience and location. We will also pay for work-related travel and equipment expenses. We offer catered lunch and dinner at our offices in Berkeley.

• Application process: A 72-minute programming assessment, a short screening call, two 1-hour interviews, and a 1-2 week paid work trial. If you are not available for a work trial we may be able to find alternative ways of testing your fit.

If you have any questions about the role, please do get in touch at [email protected].