AI Researcher

How do you know if a candidate is doing well when there is no right answer?
For twenty years, code evaluation was deterministic. A solution passed a test case or it did not. That world is over. Candidates today work alongside AI assistants, engage in multi-turn conversations, and produce outputs that blend human and machine reasoning. The evaluation signal is now qualitative, contextual, and deeply subjective.
What Goldman Sachs considers a strong candidate looks different from what a Series B startup considers one. What looks like a correct answer in a Python interview looks different when a transcript is in the mix. And without a rigorous methodology for defining "good," any evaluation system we build risks embedding bias we cannot see and cannot defend.
This role exists to solve that. Not as a support function. As the foundational research layer that everything else depends on.

• Design and execute research studies to define what "good performance" means across different job contexts, industries, and candidate types.
• Build the datasets, annotation frameworks, and ground truth labels that ML models on the Evaluation team learn from.
• Own the bias audit process: identify where our evaluation systems treat groups differently, understand why, and recommend what to change.
• Conduct customer and industry interviews to understand how enterprise clients define skilled performance, then translate those signals into measurable constructs.
• Write and maintain documentation on research methodology that product, ML, and commercial teams can reference and trust.
• Stay current on relevant literature in educational measurement, psychometrics, algorithmic fairness, and LLM evaluation, and bring applicable ideas into the work.

• You have a Master's or PhD in a research-intensive discipline: cognitive science, computational social science, NLP, educational measurement, human-computer interaction, or a closely adjacent field.
• You know how to design a methodology from scratch. You do not need a playbook. You can structure a research question, identify what data would answer it, and execute through to a defensible conclusion.
• You are digitally self-sufficient. You can write your own scripts, scrape data, run your own analyses, and work without a dedicated data engineer holding your hand.
• You think carefully about what fairness actually means in a measurement context, not just as a talking point but as a design constraint.
• You communicate findings to non-researchers without oversimplifying. You know how to make a methodology legible to a PM, a customer, and an engineer.

• Familiarity with psychometric frameworks, IRT, or educational assessment standards.
• Published work or thesis research touching on algorithmic fairness, evaluation design, or human judgment in AI systems.
• Experience running user research or qualitative studies alongside quantitative modeling.
• Prior exposure to B2B or enterprise product environments where research outputs feed directly into shipped product.

• You are energized by the fact that there is no prior art for this problem. 
• You find defining the measurement framework as intellectually interesting as building the model. 
• You are the kind of researcher who talks to customers, gets their hands in messy data, and does not wait to be assigned a hypothesis. You want your research to ship, not just get published.