Agent Felet & Multi Agent Systems (Horizon AI)

Posted 8 Hours Ago
Be an Early Applicant
Hiring Remotely in USA
Remote
Expert/Leader
Cloud • Information Technology • Cybersecurity • Infrastructure as a Service (IaaS)
The Role
Own the reasoning and autonomy layer for production AI agents managing GPU infrastructure. Build agent-behavior evaluation harnesses, define autonomy SLOs, establish evidence-based action thresholds, and lead orchestration, tool dispatch, task decomposition, escalation, and governance. Improve agent explanations and encode operational runbooks into executable reasoning while addressing reliability issues such as loops, malformed tool outputs, stuck states, and context exhaustion.
Summary Generated by Built In
Agent Fleet & Multi Agent Systems

Level: Principal to Distinguished (MTS 5 to 6) · Reports to: Chief AI Officer

Horizon AI is building the operating system for how organizations run themselves. The first step is autonomous data centers. A fleet of AI agents senses the state of STN’s GPU infrastructure, reasons about what it finds, and increasingly acts on it. This is running in production against real fleet data, not a prototype. Agents already take action where the risk is bounded and the outcome is verifiable. Everything riskier arrives as a recommendation for a human, and the work is moving decisions from the second category into the first, one class at a time, on evidence rather than optimism.

Data centers are the proving ground, not the destination. We start here because infrastructure is measurable and unforgiving: an agent is either right, or the fleet tells you it was wrong. What we build here generalizes, and we know where it goes next.

The role. Own the reasoning layer: dozens of agents that diagnose fleet problems, decide what should happen, and explain why. You also own the thing that does not exist yet and gates everything, which is a way to prove those agents are right often enough to be trusted with more action.

The problems you inherit

  • There is no agent-behaviour evaluation. Our existing replay harness tests code-fix regression, not operational decision quality. Building evaluation against real fleet scenarios is greenfield.

  • There is no autonomy SLO. No defined, measurable bar an agent must clear before it is trusted with a new class of action. Without one, every expansion of autonomy is a judgment call rather than a decision backed by evidence.

  • Multi-agent orchestration is real but fragile. Dispatch and meta-audit exist; reliability does not yet match the ambition.

  • The actuation ladder is incomplete. Higher-risk remediations are specified but unbuilt. Designing what agents are permitted to do, and proving they should be, is your work with the Chief Architect.

You will

  • Build the evaluation harness for agent behaviour: scenario suites drawn from real incidents, regression gating in CI, and metrics that catch an agent being confidently wrong

  • Define the autonomy SLO with the Chief Architect, and own the evidence presented when we propose moving a decision from recommendation to action

  • Own the agent architecture: orchestration, tool dispatch, task decomposition, escalation, and the governance boundaries that separate read from write

  • Improve how agents explain their reasoning, because an unexplained recommendation does not get acted on

  • Continue encoding operational runbooks into agent-executable reasoning

You have

  • Built and shipped LLM-based agentic systems in production, not prototypes, including the failure modes: malformed tool output, loops, stuck states, context exhaustion

  • Strong and specific opinions about evaluation. This is the part of the job we most need and least have

  • Experience with tool and function calling at scale, and with LLM tracing and online evaluation

  • Strong systems engineering fundamentals in Python, since agents that touch infrastructure are distributed systems first

  • Experience with multi-agent orchestration, handoff, and context management

Nice to have: operational or SRE background, human-in-the-loop system design, retrieval over operational data, experience defining what an autonomous system is allowed to do

Skills Required

  • Production experience building and shipping LLM-based agentic systems
  • Experience handling agent failure modes including malformed tool output, loops, stuck states, and context exhaustion
  • Strong expertise and opinions regarding evaluation of agent behavior
  • Experience with tool and function calling at scale
  • Experience with LLM tracing and online evaluation
  • Strong systems engineering fundamentals in Python
  • Experience with multi-agent orchestration, handoff, and context management
  • Operational or SRE background
  • Experience designing human-in-the-loop systems
  • Experience with retrieval over operational data
  • Experience defining permissions and boundaries for autonomous systems
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
99 Employees
Year Founded: 2016

What We Do

STN, Inc. is a managed technology and infrastructure provider serving enterprise, regulated, and AI-driven organizations. It designs, operates, and supports secure, scalable systems, including managed IT, cloud and platform services, cybersecurity, data management, compliance engineering, enterprise hardware and software, and GPU One, its GPU-as-a-Service platform for AI training, tuning, inference, and other high-performance workloads. STN emphasizes reliability, audit readiness, and ongoing human support.

Similar Jobs

Hilton Logo Hilton

Senior Manager, Owner Relations

Events • Food • Travel • Hospitality • Hotels & Resorts
In-Office or Remote
2 Locations
500000 Employees
40K-120K Annually

Zscaler Logo Zscaler

Product Specialist

Cloud • Information Technology • Security • Software • Cybersecurity
Easy Apply
Remote or Hybrid
USA
8697 Employees
172K-245K Annually
Remote
United States
550 Employees
150K-235K Annually

Pomelo Care Logo Pomelo Care

Consultant

Healthtech • Telehealth
Remote
United States
550 Employees
135K-200K Annually

Similar Companies Hiring

NODA AI Thumbnail
Artificial Intelligence • Information Technology • Software • Cybersecurity • Defense • Defense Technology
Austin, TX
54 Employees
Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
567 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account