Senior AI Engineer (LLM & Agent Systems) — Platform

Posted Yesterday
Be an Early Applicant
New York, NY, USA
In-Office
Senior level
Agency • Information Technology • Professional Services
The Role
Design and scale reusable LLM and agent workflows, orchestration patterns, evaluation pipelines, monitoring systems, APIs, and platform components. Improve AI system reliability, accuracy, latency, robustness, observability, and cost efficiency. Select models and tools, implement safeguards and fallbacks, and collaborate with platform and backend engineers to deliver scalable, reusable AI capabilities.
Summary Generated by Built In


Role Summary
This role focuses on designing and scaling reusable AI-driven workflows, particularly those powered by large language models and autonomous agents. You will build foundational components and abstractions that enable multiple internal teams to rapidly develop and deploy intelligent systems. The position emphasizes system reliability, evaluation rigor, and thoughtful tradeoffs in model and tooling selection.

Core Ownership Areas

  • Develop reusable agent-based workflows to accelerate delivery across multiple projects.

  • Define and maintain evaluation standards to ensure consistent model performance over time.

  • Improve system reliability across key dimensions such as accuracy, latency, and robustness.

  • Build shared APIs and platform components used broadly across engineering teams.

Key Responsibilities

  • Design and implement orchestration patterns for LLM-powered agents.

  • Evaluate and select models, tools, and providers based on performance, cost, and reliability.

  • Build testing frameworks, evaluation pipelines, and monitoring systems for AI outputs.

  • Implement safeguards, fallback mechanisms, and cost optimization strategies.

  • Collaborate with platform and backend engineers to integrate AI capabilities into scalable services.

  • Identify repeatable patterns across projects and convert them into reusable platform features.



Requirements

Required Experience

  • Strong background in building production-grade distributed systems or platform infrastructure.

  • Practical experience developing and deploying LLM-based or agent-driven systems.

  • Demonstrated ability to design for reliability, observability, and cost efficiency.

  • High standards for code quality and system design.

Nice-to-Have Experience

  • Familiarity with retrieval systems, embeddings, or context management pipelines.

  • Experience working within regulated or security-conscious environments.

Approach to Work

  • Prioritizes measurable quality through structured evaluation and testing.

  • Designs systems for reuse, scalability, and clean abstraction layers.

  • Focuses on building solutions that generalize beyond a single use case or team.



Benefits
- Hybrid onsite.
- Incredible perks and comp package. 

Skills Required

  • Strong background building production-grade distributed systems or platform infrastructure
  • Practical experience developing and deploying LLM-based or agent-driven systems
  • Experience designing for reliability, observability, and cost efficiency
  • High standards for code quality and system design
  • Familiarity with retrieval systems, embeddings, or context management pipelines
  • Experience working within regulated or security-conscious environments
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
3 Employees
Year Founded: 2008

What We Do

Calliere is a boutique professional and executive recruitment and advisory firm that helps organizations build high-performance teams. It specializes in sourcing leadership, engineering, data science and AI, healthcare technology, infrastructure, telecom, sales, and business-development talent. The firm serves clients across technology, fintech, healthcare, life sciences, infrastructure, and other sectors through a personalized, thorough, results-driven approach tailored to each client's specific hiring and organizational needs.

Similar Jobs

Hilton Logo Hilton

Assistant Chef Concierge/Assistant Director of Guest Relations - Waldorf Astoria New York

Events • Food • Travel • Hospitality • Hotels & Resorts
In-Office
New York, NY, USA
500000 Employees

Cloudflare Logo Cloudflare

Commercial Account Executive

Cloud • Information Technology • Security • Software • Cybersecurity
Hybrid
5 Locations
4400 Employees
234K-321K Annually

Cloudflare Logo Cloudflare

Commercial Account Executive

Cloud • Information Technology • Security • Software • Cybersecurity
Hybrid
5 Locations
4400 Employees
234K-321K Annually

New York Life Logo New York Life

Events Team Coordinator

Artificial Intelligence • Cloud • Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Hybrid
New York, NY, USA
12000 Employees
65K-93K Annually

Similar Companies Hiring

Cravath, Swaine & Moore LLP Thumbnail
Legal Tech • Other • Professional Services
New York, NY
1200 Employees
VCA Animal Hospitals Thumbnail
Pet • Professional Services
Los Angeles, CA
35000 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account