Principal AI Engineer

Posted 2 Days Ago
Easy Apply
Be an Early Applicant
San Francisco, CA
Hybrid
215K-282K Annually
Senior level
Artificial Intelligence • Cloud • Information Technology • Machine Learning • Software
Hybrid Observability powered by AI
The Role
The Principal AI Engineer will lead the design and implementation of a multi-tenant data platform and multi-agent systems, focusing on AI and machine learning integration to enhance operational visibility and predictability for IT environments.
Summary Generated by Built In

About Us:  

We love going to work and think you should too. Our team is dedicated to trust, customer obsession, agility, and striving to be better everyday. These values serve as the foundation of our culture, guiding our actions and driving us towards excellence. We foster a culture of performance and recognition, allowing us to transform growth as we enable our employees to do the best work of their careers.

This role is open to candidates based in or near San Francisco, CA. At LogicMonitor, we hire within our Centers of Energy—vibrant locations where our teams connect, collaborate, and innovate.

To learn more about life at LogicMonitor, check out our Careers Page.

What You'll Do:

LM Envision, LogicMonitor's leading hybrid observability platform powered by AI, helps modern enterprises gain operational visibility into and predictability across their IT stacks, so they can continue to deliver extraordinary employee and customer experiences. LogicMonitor has a layered approach to intelligence, where AI and Machine Learning is baked into every facet of the LM Envision platform to help IT teams improve efficiency, minimize alert fatigue, proactively predict trends, and maximize enterprise growth and transformation. 

Our customers love LogicMonitor's ability to bring cloud and traditional IT together into one view, as seen in minimal churn rates, expansion business, and exciting new customer references. In fact, LogicMonitor has received the highest Net Promoter Score of any IT Infrastructure Management provider. LogicMonitor also boasts high employee satisfaction. We have been certified as a Great Place To Work®, and named one of BuiltIn's Best Places to Work for the seventh year in a row! 

We are looking for a Principal AI Engineer- Multi Agent Systems that will own the multi-tenant Data platform (ingestion, semantics, governance, APIs) and the MCP-enabled multi-agent runtime (planning, routing, memory, safety, evaluation). You’ll define the north-star architecture and ship a reliable, cost-aware platform that lets teams compose production agents over trusted data.

Here's a closer look at this key role:

  • Connectors & Ingestion: ERP/CRM/DB/files; Kafka/Pulsar; schema evolution & backfills.
  • Semantic Layer & Contracts: Pydantic/JSON Schema/OpenAPI; business semantics, field descriptions, policy enforcement points.
  • APIs & Access: REST/GraphQL + MCP tool endpoints with consistent pagination, filtering, row/column-level security.
  • Retrieval Layer: embeddings service, hybrid dense/sparse, re-rankers, graph joins; deterministic citation/attribution.
  • Multi-Agent Runtime: supervisor-decomposer, router-selector, tool-using workers, reflection/verifier/judge agents; capability negotiation; blackboard/event bus patterns.
  • State & Checkpointing: step store, durable checkpoints, resume/replay; concurrency controls and backpressure.
  • HITL: interruptible workflows, approval gates, timeouts/escalations, and UX hooks (SSE/WebSocket) for human feedback.
What You'll Need:
  • 8+ years backend/data platform engineering
  • 3 - 5+ years shipping production AI systems (RAG/agents).
  • Strong Python (or Go/Java) and distributed systems: queues, concurrency, idempotency, backpressure.
  • Hands-on with FastAPI/gRPC, GraphQL, OpenAPI/Pydantic, and schema governance.
  • Practical MCP experience (servers/clients, tool schemas, versioning, auth).
  • Retrieval depth: embeddings, hybrid search, vector/graph DBs, chunking, evals.
  • Kubernetes/Helm, IaC, autoscaling, secrets, blue/green & canaries for platform services.
  • Security/tenancy: OIDC/OAuth2, RBAC/ABAC, audit, DLP/PII, per-tenant isolation.

Residents of California, click Here to view our California Applicant Privacy Notice.

Anticipated Application Close Date: 12/19/25

LogicMonitor is an Equal Opportunity Employer
At LogicMonitor, we believe that innovation thrives when every voice is heard and each individual is empowered to bring their unique perspective. We’re committed to creating a workplace where diversity is celebrated, and all employees feel inspired and supported to contribute their best.

For us, equal opportunity means fostering a truly inclusive culture where everyone has the chance to grow and succeed. We don’t just open doors; we invite you to step through and be part of something bigger. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

#LI-JP1  #LI-Hybrid #BI-Hybrid 

LogicMonitor is dedicated to fostering a culture of transparency and fairness, including our commitment to pay transparency. We provide the base salary ranges for all positions posted within the United States. 

Compensation packages at LogicMonitor for eligible roles include base salary, a variable plan depending on role, along with comprehensive benefits. The range displayed on each job posting reflects the minimum and maximum base salary target for new hires in the position, determined by work location and additional factors, including job-related skills, experience, interview performance, and relevant education or training. LogicMonitor employees in eligible roles are also granted equity based compensation, subject to Board of Director approval. As part of our holistic compensation philosophy, your package will also include, but is not limited to: Comprehensive health, dental and vision coverage, generous parental leave policies, access to our Employee Assistance Program and various Wellness programs, a 401K with company matching, a learning and development stipend, and an unlimited vacation policy. For more information on our benefits, see our careers page.

The Base Salary range for this role is:
$215,160$282,425 USD

                                               

Our goal is to ensure an accessible and inclusive experience for every candidate.

If you need a reasonable accommodation during the application or interview process under applicable local law, please submit a request via this Accommodation Request Form.

Know your rights: workplace discrimination is illegal. Please click here to review LogicMonitor’s U.S. Pay Transparency Nondiscrimination Provision.

Top Skills

Fastapi
Go
GraphQL
Grpc
Helm
Java
Kubernetes
Openapi
Pydantic
Python

What the Team is Saying

Jude
Kenyon
Maddie
Franky
David
Kwame
Crystal
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Santa Barbara, CA
1,100 Employees
Year Founded: 2007

What We Do

LogicMonitor® offers hybrid observability powered by AI. The company’s SaaS-based platform, LM Envision, enables observability across on-prem and multi-cloud environments. We provide IT and business teams operational visibility and predictability across their technologies and applications to focus less on troubleshooting and more on delivering extraordinary employee and customer experiences. For more information, visit www.logicmonitor.com.

Why Work With Us

We love going to work and think you should too. We are customer-obsessed, work as one agile team, and strive to be better every day while building trust. These are our core values. So it's no surprise that we work hard and genuinely have fun working with each other as we expand our global presence and achieve record-breaking success.

Gallery

Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery

LogicMonitor Offices

Hybrid Workspace

Employees engage in a combination of remote and on-site work.

We call our offices Centers of Energy, because they’re where we accelerate work, spark creativity, and ignite our culture of connection and celebration. Our teams coordinate their time in Centers of Energy to reflect how they work best.

Typical time on-site: Flexible
Company Office Image
HQSanta Barbara, CA
Company Office Image
Singapore
Company Office Image
Austin, TX
Company Office Image
Boston, MA
Company Office Image
London, UK
Company Office Image
Pune, IN
Company Office Image
San Francisco
Company Office Image
Sydney, Australia
Learn more

Similar Jobs

LogicMonitor Logo LogicMonitor

Manager, Financial Planning & Analysis

Artificial Intelligence • Cloud • Information Technology • Machine Learning • Software
Easy Apply
Hybrid
San Francisco, CA, USA
131K-164K Annually

LogicMonitor Logo LogicMonitor

Public Relations Manager

Artificial Intelligence • Cloud • Information Technology • Machine Learning • Software
Easy Apply
Hybrid
San Francisco, CA, USA
131K-145K Annually

LogicMonitor Logo LogicMonitor

Executive Assistant

Artificial Intelligence • Cloud • Information Technology • Machine Learning • Software
Easy Apply
Hybrid
San Francisco, CA, USA
90K-112K Annually

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account