Staff Data Scientist, Agent Improvement Loop

Posted Yesterday
Be an Early Applicant
11 Locations
Remote
Expert/Leader
Machine Learning • Software
The Role
Owns the evaluation and continuous improvement loop for Yalo’s conversational AI agents. Responsibilities include designing agent evaluations, analyzing production traces, identifying failure patterns, calibrating LLM judges, implementing evidence-based fixes, and measuring outcomes with causal methods. The role also develops production Python and TypeScript tooling, partners with customer-facing teams, mentors technical staff, and contributes to agent-quality planning.
Summary Generated by Built In
Yalo

Hi! This is Yalo! We are on a mission to bring conversational commerce to the world...

Remember how it used to be to interact with businesses that knew and understood you, that could recommend exactly what you needed, and that with a simple message could get you what you wanted??? Yep... neither do we. That is why at Yalo we are marrying the scale of digital commerce with the personalization and simplicity of conversations to help companies delight their users.

We know that traditional SAAS companies focus on first-world problems... we don't! Having started in Latin America, our roots are in Emerging Markets and therefore we care about bringing amazing experiences to a population that traditionally has been underserved, such as the small shop owner in Brazil who is ordering online for the first time.

If you're looking for a place to make things happen, learn fast, and impact emerging markets in a way that hasn't been done before, look no further. 💫

Come Join us in our mission of improving billions of lives through the power of conversational commerce!

Your mission 🚀

As a Staff Data Scientist, you make our AI agents measurably better, and you prove it. You own how Yalo evaluates agents: evals, trace and error analysis, and the improvement loop that turns what we find in production into fixes to our agent skills, harness, CLI and customers' conversational flows. You work with Product, Engineering, Delivery and Customer Success. Your job is to show what moved the metric, not to guess.

What are the responsibilities for this role?
  • Own the evals strategy for our agents: pre-release behavioral testing, production monitoring, and the judges that score conversations.
  • Run continuous error analysis on agent traces at scale. Find failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.
  • Close the loop. Turn findings into fixes and new evals, ship them, and measure the result with sound before-and-after and causal methods.
  • Improve our agent skills, harness and CLI using trace evidence. Work through the owning teams' repos and review process.
  • Partner with delivery and customer teams to take real customer flows from underperforming to reliably healthy, and make the process repeatable for others.
  • Calibrate and validate LLM judges against human review. Measure their stability, report their limits, and set the bar for what counts as evidence at Yalo.
  • Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI, so it can live in a shared codebase.
  • Mentor data scientists and engineers on measurement, experimentation and evaluation practice.
  • Help shape quarterly and annual planning for agent quality.

Job Requirements (Must have)

  • BS or MS in a technical field (Computer Science, Statistics, Applied Math, Operations Research, Economics, etc.).
  • 8+ years delivering data science solutions with measurable business impact in cross-functional teams.
  • Hands-on experience evaluating LLM-based systems or agents: eval design, LLM-as-judge, rubric design, and judge calibration.
  • Expert in Python and SQL. Comfortable in TypeScript, or ready to be.
  • Strong production engineering habits: version control, code review, testing, CI/CD, and shipping tools others use.
  • Experience with GenAI systems in production: prompt and tool design, agent frameworks (e.g., LangGraph), and LLM observability tooling (e.g., LangSmith).
  • Skilled at turning messy trace and conversation data into clear findings for technical and non-technical audiences.
  • Strong product and business sense. You frame quality work in terms of customer outcomes.
Nice to have:
  • Experience with conversational AI, customer-facing agents or tool-using agents.
  • NLP and clustering for failure-pattern discovery.
  • Causal inference and experimentation skills: A/B tests, difference-in-differences, pre/post analysis, and the judgment to say what the data can't show.
  • Exposure to MLOps or LLMOps platforms (Vertex, SageMaker, MLflow, or similar) and big data tools.
  • Familiarity with AI safety and responsible AI practices.
  • Comfort in an ambiguous, fast-changing environment.
  • Experience mentoring other data scientists.

What do we offer? 

  • Unlimited PTO policy
  • Competitive rewards on the market range
  • Remote working is available (-+3 hours CT)
  • Flexible time (driven by results)
  • Start-up environment
  • International teamwork
  • You and nothing else limit your career here

We care,

We keep it simple,

We make it happen,

We strive for excellence. 

At Yalo, we are dedicated to creating a workplace that embodies our core values: caring, initiative, excellence, and simplicity. We believe in the power of diversity and inclusivity, where everyone's unique perspectives, experiences, and talents contribute to our collective success. As we embrace and respect our differences, we strive to create something extraordinary for the benefit of all.
We are proud to be an Equal Opportunity Employer, providing equal opportunities to individuals regardless of race, color, religion, national or ethnic origin, gender, sexual orientation, gender identity or expression, age, disability, protected veteran status, or any other legally protected characteristic. Our commitment to fairness and equality is a fundamental pillar of our company.

At Yalo, we uphold a culture of excellence. We constantly challenge ourselves to go above and beyond, delivering remarkable results and driving innovation. We encourage each team member to take initiative and make things happen, empowering them to bring their best ideas forward and contribute to our shared goals.

Skills Required

  • BS or MS degree in a technical field such as Computer Science, Statistics, Applied Math, Operations Research, or Economics
  • 8+ years delivering data science solutions with measurable business impact in cross-functional teams
  • Hands-on experience evaluating LLM-based systems or agents, including eval design, LLM-as-judge, rubric design, and judge calibration
  • Expertise in Python and SQL
  • Comfort with TypeScript or willingness to learn it
  • Production engineering practices including version control, code review, testing, CI/CD, and shipping shared tools
  • Experience with production GenAI systems, prompt and tool design, agent frameworks such as LangGraph, and LLM observability tools such as LangSmith
  • Ability to analyze trace and conversation data and communicate findings to technical and non-technical audiences
  • Strong product and business sense focused on customer outcomes
  • Experience with conversational AI, customer-facing agents, or tool-using agents
  • NLP and clustering experience for failure-pattern discovery
  • Causal inference and experimentation skills, including A/B tests, difference-in-differences, and pre/post analysis
  • Exposure to MLOps or LLMOps platforms such as Vertex, SageMaker, MLflow, or similar, and big data tools
  • Familiarity with AI safety and responsible AI practices
  • Comfort working in an ambiguous, fast-changing environment
  • Experience mentoring other data scientists
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Ciudad de Mexico
288 Employees
Year Founded: 2015

What We Do

Yalo helps businesses offer a delightful customer experience on their customers'​ favorite apps – Facebook Messenger, WhatsApp, and WeChat. Every business on Yalo has a virtual assistant with an optimized flow that handles customer requests to the point of payment and fulfillment. If at any point the customer need is beyond what the chatflow is capable of doing, the request gets seamlessly transferred to a human agent to provide resolution. Through the Yalo platform, businesses get an artificial intelligence powered CRM that allows them to handle sales and build personal relationships, at scale, on Facebook Messenger, WhatsApp, and WeChat. Yalo uses deep learning (a machine learning technique) to have chatbots learn from their human counterparts to continuously improve the experience. The team behind Yalo is composed of Columbia University and Salamanca engineers, former PayPal technical leads, and is supported by a group of investors from New York, Mexico and Latin America.

Similar Jobs

Atlassian Logo Atlassian

Consultant

Cloud • Information Technology • Productivity • Security • Software • App development • Automation
In-Office or Remote
16 Locations
11000 Employees

Atlassian Logo Atlassian

Senior Account Executive

Cloud • Information Technology • Productivity • Security • Software • App development • Automation
In-Office or Remote
17 Locations
11000 Employees

Atlassian Logo Atlassian

Head Of Product

Cloud • Information Technology • Productivity • Security • Software • App development • Automation
In-Office or Remote
19 Locations
11000 Employees

Luxury Presence Logo Luxury Presence

Manager, Production Quality

Marketing Tech • Real Estate • Software • PropTech • SEO
Easy Apply
Remote or Hybrid
12 Locations
500 Employees

Similar Companies Hiring

Ford Energy Thumbnail
Automotive • Software • Energy • Utilities • Manufacturing • Renewable Energy
US
Revel Thumbnail
Aerospace • Hardware • Robotics • Software
Marina Del Rey, California
70 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account