AI Engineer - Evaluations

Hippocratic AI is the leading generative AI company in healthcare. We have the only system that can have safe, autonomous, clinical conversations with patients. We have trained our own LLMs as part of our Polaris constellation, resulting in a system with over 99.9% accuracy.

Reinvent healthcare with AI that puts safety first. We’re building the world’s first healthcare‑only, safety‑focused LLM — a breakthrough platform designed to transform patient outcomes at a global scale. This is category creation.

Work with the people shaping the future. Hippocratic AI was co‑founded by CEO Munjal Shah and a team of physicians, hospital leaders, AI pioneers, and researchers from institutions like El Camino Health, Johns Hopkins, Washington University in St. Louis, Stanford, Google, Meta, Microsoft, and NVIDIA.

Backed by the world’s leading healthcare and AI investors. We recently raised a $126M Series C at a $3.5B valuation, led by Avenir Growth, bringing total funding to $404M with participation from CapitalG, General Catalyst, a16z, Kleiner Perkins, Premji Invest, UHS, Cincinnati Children’s, WellSpan Health, John Doerr, Rick Klausner, and others.

Build alongside the best in healthcare and AI. Join experts who’ve spent their careers improving care, advancing science, and building world‑changing technologies — ensuring our platform is powerful, trusted, and truly transformative.

Location Requirement

We believe the best ideas happen together. To support fast collaboration and a strong team culture, this role is expected to be in our Palo Alto office five days a week, unless otherwise specified.

As an AI Engineer – Evaluations at Hippocratic AI, you’ll define and build the systems that measure, validate, and improve the intelligence, safety, and empathy of our voice-based generative healthcare agents.

Evaluation sits at the heart of our model improvement loop — it informs architecture choices, training priorities, and launch decisions for every patient-facing agent. You’ll design LLM-based auto-evaluators, agent harnesses, and feedback pipelines that ensure each model interaction is clinically safe, contextually aware, and grounded in healthcare best practices.

You’ll collaborate closely with research, product, and clinical teams, working across the stack — from backend data pipelines and evaluation frameworks to tooling that surfaces insights for model iteration. Your work will directly shape how our agents behave, accelerating both their reliability and their real-world impact.

• Design and build evaluation frameworks and harnesses that measure the performance, safety, and trustworthiness of Hippocratic AI’s generative voice agents.

• Prototype and deploy LLM-based evaluators to assess reasoning quality, empathy, factual correctness, and adherence to clinical safety standards.

• Build feedback pipelines that connect evaluation signals directly to model improvement and retraining loops.

• Partner with AI researchers and product teams to turn qualitative gaps into clear, defensible, and reproducible metrics.

• Develop reusable systems and tooling that enable contributions from across the company, steadily raising the quality bar for model behavior and user experience.

Must-Have:

• 3+ years of software or ML engineering experience with a track record of shipping production systems end-to-end.

• Proficiency in Python and experience building data pipelines, evaluation frameworks, or ML infrastructure.

• Familiarity with LLM evaluation techniques — including prompt testing, multi-agent workflows, and tool-using systems.

• Understanding of deep learning fundamentals and how offline datasets, evaluation data, and experiments drive model reliability.

• Excellent communication skills with the ability to partner effectively across engineering, research, and clinical domains.

• Passion for safety, quality, and real-world impact in AI-driven healthcare products.

Nice-to-Have:

• Experience developing agent harnesses or simulation environments for model testing.

• Background in AI safety, healthcare QA, or human feedback evaluation (RLHF).

• Familiarity with reinforcement learning, retrieval-augmented evaluation, or long-context model testing.

If you’re excited by the challenge of building trusted, production-grade evaluation systems that directly shape how AI behaves in the real world, we’d love to hear from you.

Join Hippocratic AI and help define the standard for clinically safe, high-quality AI evaluation in healthcare.

Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.