Applied Research Scientist

Posted 19 Days Ago
Be an Early Applicant
Hiring Remotely in US
Remote
Senior level
Artificial Intelligence • Healthtech
The Role
Design and build clinical-grade automated evaluation pipelines (LLM-as-judge + human review), define benchmarks, run rigorous experiments, partner with engineering to implement CI gates for accuracy, factuality, and safety, and publish findings to improve model robustness and reduce hallucinations.
Summary Generated by Built In

About Us
At Sully.ai, We’re Building the Most Impactful Healthcare Company on Earth

We believe that access to a great doctor is a basic human right. Today, that’s not a reality. Delays, misdiagnoses, administrative chaos, and burnout plague the system.

Our Mission
One Human, One Doctor.
We enable our customers to staff 30% of their workforce with AI by creating a shared agent architecture for scale and efficiency. All powered by our own patented, world-class models and deployed in real-world care.

Key Responsibilities
  • Build and scale automated evaluation pipelines (LLM-as-judge + human review) with clinical-grade benchmarks.

Hard Requirements
  • Proven experience designing agentic processes and LLM evaluation/benchmarking frameworks.

  • Strong Python and ML background (PyTorch/TensorFlow, Hugging Face, LangChain/LlamaIndex).

  • Demonstrated ability to design rigorous experiments and translate findings into production.

  • Track record of published research or deep applied work in LLMs and agent evaluation.

  • Strong communication and technical writing skills to articulate complex findings clearly.

  • First-Month Focus

  • Audit existing evaluation approaches for clinical and agentic tasks.

  • Define initial benchmarks and build early automated pipelines.

  • Partner with engineering to land first set of CI gates for accuracy, factuality, and safety.

  • 90 Days

  • Deliver a repeatable evaluation framework with automated pipelines in production.

  • Demonstrate measurable improvements in robustness, hallucination reduction, or safety.

  • Publish or present internal research findings that directly shape product reliability.

  • If you’ve ever said, “I want to do work that actually matters”, this is it. Let’s build something life-changing, together.

Key Results (First 90 Days)
  • Deliver a repeatable evaluation framework with automated pipelines in production.

  • Demonstrate measurable improvements in robustness, hallucination reduction, or safety.

  • Publish or present internal research findings that directly shape product reliability.

Why Join Sully.ai?
🔥 Revolutionizing the antiquated $800B+ Healthcare market

🧠 60%+ Ex-founders who have built, scaled, exited. We hire A-players

⚡️ Speed matters: we operate with urgency, autonomy, and ownership

🧪 You’ll work on real, first-of-their-kind problems at the edge of AI and medicine

❤️ Your work directly unlocks doctors to reclaim their time, and patients get better, faster care

Sully.ai is an equal opportunity employer. In addition to EEO being the law, it is a policy that is fully consistent with our principles. All qualified applicants will receive consideration for employment without regard to status as a protected veteran or a qualified individual with a disability, or other protected status such as race, religion, color, national origin, sex, sexual orientation, gender identity, genetic information, pregnancy or age. Sully.ai prohibits any form of workplace harassment. 

Top Skills

Hugging Face
Langchain
Llamaindex
Python
PyTorch
TensorFlow
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Mountain View, California
60 Employees
Year Founded: 2023

What We Do

Superhuman medical staff that’s 10x better, 20x cheaper, and 100x faster.

Similar Jobs

Tensorlake Logo Tensorlake

Scientist

Generative AI
In-Office or Remote
San Francisco, CA, USA
13 Employees

Nourish Logo Nourish

Staff Software Engineer

Healthtech • Software • Telehealth
Easy Apply
In-Office or Remote
2 Locations
215 Employees

Mastercard Logo Mastercard

Director, Site Reliability Engineering

Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Remote or Hybrid
O'Fallon, MO, USA
38800 Employees
152K-258K Annually

Mastercard Logo Mastercard

Sr Specialist, Merchant Offer Sales - Mastercard Commerce Media

Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Remote or Hybrid
New York, NY, USA
38800 Employees
104K-177K Annually

Similar Companies Hiring

Milestone Systems Thumbnail
Software • Security • Other • Big Data Analytics • Artificial Intelligence • Analytics
Lake Oswego, OR
1500 Employees
Idler Thumbnail
Artificial Intelligence
San Francisco, California
6 Employees
Bellagent Thumbnail
Artificial Intelligence • Machine Learning • Business Intelligence • Generative AI
Chicago, IL
20 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account