Be an Early Applicant
The Role
Design, implement, and scale evaluation frameworks and automated pipelines to monitor safety, reliability, and alignment of agentic LLM systems. Define metrics, run experiments assessing reasoning, factuality and robustness, collaborate with model developers and clinicians, and drive iterative improvement and internal knowledge sharing.
Summary Generated by Built In
Set a new pulse for healthcare!
We are looking for a Senior Machine Learning Engineer to join the Orchestration team in AI Health Assistant.
Your mission will be to ensure our AI Health Companion behaves safely, reliably, and helpfully — for millions of patients and practitioners. You will work in a cross-functional team building robust evaluation pipelines for agentic AI systems, contributing directly to improving the quality and safety of AI-powered healthcare.
Working in the tech team at Doctolib means building innovative products and features to improve the daily lives of care teams and patients.
What you'll doYour responsibilities include but are not limited to:
- Define and own the evaluation strategy for our AI agentic system — metrics, protocols, datasets, and tooling
- Implement and maintain automated evaluation pipelines to monitor model quality, safety, and alignment across iterations
- Run systematic experiments to assess reasoning, factuality, robustness, and user experience
- Collaborate closely with model developers and research scientists to provide insights and drive iterative improvement
- Contribute to research and internal knowledge sharing on LLM evaluation methodologies and best practices
- Our solutions are built on a single fully cloud-native platform that supports web and mobile app interfaces, multiple languages, and is adapted to country and healthcare specialty requirements.
- Our stack is composed of Rails, TypeScript, Java, Python, Kotlin, Swift, and React Native.
- We leverage AI ethically across our products to empower patients and health professionals. Discover our AI vision here.
Want to learn more about our tech culture and environment? Visit the Doctolib Tech site.
Who you areBefore you read on: if you don't have the exact profile described below, but you feel this job description matches your skill set, we still encourage you to apply.
You'll be a great fit if you:
- Hold an MSc or PhD in Computer Science, Machine Learning, Data Science, or a related field
- Have 7+ years of hands-on experience working with large language models (e.g., GPT, Claude, Llama, or BERT-like architectures)
- Have proven experience evaluating agentic or reasoning systems (e.g., autonomous agents, tool-using LLMs, dialogue systems, or task-oriented assistants)
- Have a strong track record in experiment design, metric definition, and evaluation automation
- Are able to bridge research and production, influencing both modeling and product decisions
- Have excellent communication skills, a collaborative mindset, and are fluent in English
It would be fantastic if you:
- Have experience in the clinical or medical domain and sensitivity to ethical or regulatory challenges in healthcare AI
- Free comprehensive health insurance (basic package) for you and your children
- 25 days of paid vacation per year, plus up to 14 days of RTT
- Free mental health and coaching services through our partner Moka.care
- Work from abroad for up to 10 days per year thanks to our flexibility days policy
- Lunch vouchers (Swile card) worth €8.50 per working day, with €4.50 covered by Doctolib
- A subsidy from the work council to refund part of the membership to a sport club or a creative class
- 50% reimbursement of your public transport subscription
- ParentCare Program: Enjoy full salary coverage (100%) during your first month of birth leave, and 75% during the second, covered by Doctolib
- Enrollment in Doctolib's long-term employee value sharing plan called DoctoGrowth
- For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
- Relocation support in case of international mobility
- Access to the best AI tools for coding, development and dedicated training
- Recruiter Interview
- Technical Deep Dive
- Data System Design
- Behavioral Interview
- At least one reference check
We want your experience to be clear, respectful, and transparent. Learn more about our hiring process on our candidate experience page.
Job details- Permanent position
- Full-time
- Hybrid in our Levallois-Perret office (up to 2 remote days per week)
- Start date: as soon as possible
At Doctolib, we are committed to improving access to healthcare for everyone. This translates into our recruitment process. We evaluate candidates based solely on qualifications and motivation, without any form of discrimination.
The more diverse ideas are heard, the more our product will truly improve healthcare for all. You are welcome to apply to Doctolib, regardless of your gender, religion, age, sexual orientation, ethnicity, or disability.
To ensure equal opportunities, we invite you to exclude personal information (e.g., pictures, age) from your applications. If you require any accommodation, please let us know for support during the hiring process.
Join us in building the healthcare we all dream of!
Your data privacyAll information provided is processed by Doctolib for application management. For data processing details, click here: France. Please contact hr.dataprivacy(at)doctolib.com for inquiries or to exercise your rights.
Skills Required
- MSc or PhD in Computer Science, Machine Learning, Data Science, or related field
- 7+ years of hands-on experience working with large language models (e.g., GPT, Claude, Llama, or BERT-like architectures)
- Proven experience in evaluating agentic or reasoning systems (autonomous agents, tool-using LLMs, dialogue systems, task-oriented assistants)
- Strong track record in experiment design, metric definition, and evaluation automation
- Ability to bridge research and production and influence modeling and product decisions
- Excellent communication skills and a collaborative mindset
- Experience in the clinical or medical domain and sensitivity to ethical or regulatory challenges in healthcare AI
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Since Doctolib's creation in 2013, we have had one purpose: strive for a healthier world. 1. We aim to improve the daily lives of care teams by providing them with a new generation of technologies and services. 2. We aim to improve health for all, by offering a fast and frictionless journey for all care episodes, creating new ways for people to receive care and empowering them to become actors of their health. At Doctolib, we are honored to work in the healthcare field and we believe that innovation in healthcare should be handled differently. We apply 4 guiding principles in everything we do: 1. We create helpful solutions for care teams and people. 2. We serve everyone equally and create well-designed and accessible technologies. 3. We team up with our users to strive for a healthier world and act as one team. 4. We protect our users' privacy. It’s their health, their data. To achieve our purpose, we are assembling a team dedicated to improving healthcare, with a human-centric approach and an entrepreneurial mindset. www.doctolib.com
