Principal AI Agent Engineer – Agentic AI & LLM Applications

Posted Yesterday
Be an Early Applicant
4 Locations
In-Office
155K-261K Annually
Senior level
Internet of Things • Mobile • Retail
The Role
Design, build, deploy, and scale production AI agents, agentic workflows, and LLM applications. Partner with business and product teams to identify opportunities, implement tools, APIs, retrieval, orchestration, evaluation, observability, guardrails, security, and Responsible AI controls. Integrate solutions with enterprise data and systems, establish reusable engineering patterns, and optimize models for accuracy, latency, cost, reliability, and measurable business impact.
Summary Generated by Built In

This is a full-time onsite role based in the location(s) posted and is performed from the designated work location five days per week. No relocation is offered.


The Business Agent Foundry is seeking a Principal AI Agent Engineer to design, build, and scale production-grade AI agents and LLM-powered applications that solve high-value business problems.


This is not a traditional software engineering role with AI layered on top. You will work directly with business partners to turn complex, sometimes ambiguous problems into working AI-enabled solutions, moving quickly from problem definition and prototype to production while maintaining enterprise standards for security, reliability, observability, Responsible AI, and maintainability.


We’re looking for someone who has already shipped AI agents, LLM-powered workflow automation, or applied GenAI solutions into production and can clearly explain how those systems worked, how they were adopted, and the measurable impact they delivered.


What You'll Do:

  • Design, build, and deploy production-grade AI agents and agentic workflows that improve customer experience, employee productivity, operational efficiency, quality, or cost.
  • Partner directly with business leaders, product teams, and process owners to identify high-value AI opportunities and translate ambiguous requirements into scalable solutions.
  • Build conversational, workflow, and task-oriented agents capable of using tools, APIs, enterprise data, retrieval, structured reasoning, and human-in-the-loop approvals.
  • Develop agent orchestration using LangGraph, Semantic Kernel, AutoGen, CrewAI, Bedrock Agents, or similar production-capable frameworks.
  • Engineer prompts, system instructions, context-management strategies, state, memory, and retrieval flows to improve agent accuracy and reliability.
  • Integrate AI solutions with enterprise APIs, databases, knowledge repositories, systems of record, vector stores, and other data sources.
  • Implement RAG and advanced retrieval patterns, including enterprise search, semantic retrieval, and context engineering.
  • Establish evaluation frameworks using golden datasets, scenario testing, regression testing, human review, and production feedback loops.
  • Build observability across prompts, model calls, retrieval, tool execution, latency, errors, and user interactions.
  • Design guardrails, escalation paths, human oversight, and secure access patterns for sensitive or higher-risk workflows.
  • Apply Responsible AI, privacy, security, and governance principles throughout solution design and deployment.
  • Establish reusable engineering patterns, templates, and components that other teams can leverage across the AI platform.

What You'll Bring:

  • Demonstrated experience building and shipping AI agents, LLM-powered applications, or GenAI workflow automation into production for real users or business teams.
  • The ability to explain the scope and impact of previously deployed AI systems, including architecture, users, workflows, data sources, integrations, adoption, operating model, and measurable business outcomes.
  • Deep hands-on experience with Python and backend/full-stack development using technologies such as FastAPI, Flask, or Django.
  • Strong understanding of LLM application architecture, including prompt engineering, context engineering, RAG, tool/function calling, session state, structured outputs, and response validation.
  • Hands-on experience with agent orchestration frameworks such as LangGraph, Semantic Kernel, AutoGen, CrewAI, or comparable platforms.
  • Experience designing APIs, service integrations, authentication, database interactions, and deployment-ready application components.
  • Experience connecting AI applications to enterprise data, APIs, knowledge repositories, vector databases, or systems of record.
  • Strong software engineering discipline across automated testing, code review, CI/CD, logging, monitoring, observability, and production support.
  • The ability to evaluate tradeoffs among AI models based on accuracy, latency, cost, reliability, risk, and business requirements.
  • Strong communication skills and the ability to explain complex AI and engineering decisions to both technical and non-technical stakeholders.

What You'll Need:

  • Typically 8+ years of software engineering experience, including experience designing and operating complex production systems.
  • Significant hands-on experience developing and deploying LLM, GenAI, or agentic AI applications.
  • Strong proficiency in Python, APIs, distributed application architecture, and enterprise integration.
  • Experience with at least one production-capable agent or AI orchestration framework.
  • Experience implementing evaluation, monitoring, observability, and reliability controls for AI-enabled applications.
  • Understanding of Responsible AI, data privacy, security, access control, auditability, and AI guardrails.
  • Ability to work in an onsite, highly collaborative environment with business and technology partners.

Preferred Qualifications:

  • Technical leadership experience establishing architecture, engineering standards, or reusable patterns across multiple teams.
  • Experience applying AI within enterprise functions such as customer care, operations, network, sales, finance, service delivery, or employee productivity.
  • Experience with advanced RAG, knowledge graphs, enterprise search, ontology-driven retrieval, or semantic data modeling.
  • Experience evaluating or serving LLMs at scale, including quality, latency, token/cost, and reliability optimization.
  • Front-end development, data visualization, or rapid product prototyping experience.
  • Open-source contributions or reusable frameworks related to AI agents, LLM applications, AI evaluation, or GenAI engineering.
Our Principal Software Engineering jobs earn between $155,400.00 - $261,100.00 USD Annual. Not to mention all the other amazing rewards that working at AT&T offers. Individual starting salary within this range may depend on geography, experience, expertise, and education/training.

Joining our team comes with amazing perks and benefits:

  • Medical/Dental/Vision coverage
  • 401(k) plan
  • Tuition reimbursement program
  • Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
  • Paid Parental Leave
  • Paid Caregiver Leave
  • Additional sick leave beyond what state and local law require may be available but is unprotected
  • Adoption Reimbursement
  • Disability Benefits (short term and long term)
  • Life and Accidental Death Insurance
  • Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
  • Employee Assistance Programs (EAP)
  • Extensive employee wellness programs
  • Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone

Weekly Hours:

40

Time Type:

Regular

Location:

Alpharetta, Georgia, Dallas, Texas, Plano, Texas, USA:GA:Atlanta / 1057 Lenox Park Blvd Ne - Adm:1057 Lenox Park Blvd Ne

Salary Range:

$155,400.00 - $261,100.00

AT&T and its subsidiaries are committed to equal employment opportunity. All hiring, promotion, and other employment decisions remain merit-based and free from discrimination on the basis of race, color, religion, religious creed, national origin, ancestry, age, sex, sexual orientation, gender, gender identity, gender expression, physical disability, mental disability, pregnancy, medical condition, genetic information, marital status, citizenship status, military status, veteran status, or any other characteristic protected by federal, state, or local laws. In addition, AT&T will provide reasonable accommodations to qualified individuals with disabilities. AT&T is a fair chance employer and does not initiate a background check until an offer is made. Click here to learn more or request an application accommodation here.

Skills Required

  • 8+ years of software engineering experience, including designing and operating complex production systems
  • Production experience building and deploying AI agents, LLM-powered applications, or GenAI workflow automation
  • Strong proficiency in Python, APIs, distributed application architecture, and enterprise integration
  • Hands-on backend or full-stack development experience with FastAPI, Flask, Django, or comparable frameworks
  • Experience with at least one production-capable agent or AI orchestration framework
  • Understanding of prompt engineering, context engineering, RAG, tool or function calling, session state, structured outputs, and response validation
  • Experience designing APIs, service integrations, authentication, database interactions, and deployment-ready application components
  • Experience connecting AI applications to enterprise data, APIs, knowledge repositories, vector databases, or systems of record
  • Experience implementing automated testing, code review, CI/CD, logging, monitoring, observability, and production support
  • Experience implementing evaluation, monitoring, observability, and reliability controls for AI-enabled applications
  • Understanding of Responsible AI, data privacy, security, access control, auditability, and AI guardrails
  • Ability to evaluate AI model tradeoffs involving accuracy, latency, cost, reliability, risk, and business requirements
  • Strong communication skills with technical and non-technical stakeholders
  • Technical leadership experience establishing architecture, engineering standards, or reusable patterns across multiple teams
  • Experience applying AI to customer care, operations, network, sales, finance, service delivery, or employee productivity
  • Experience with advanced RAG, knowledge graphs, enterprise search, ontology-driven retrieval, or semantic data modeling
  • Experience evaluating or serving LLMs at scale, including quality, latency, token or cost, and reliability optimization
  • Front-end development, data visualization, or rapid product prototyping experience
  • Open-source contributions or reusable frameworks related to AI agents, LLM applications, AI evaluation, or GenAI engineering

AT&T Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about AT&T and has not been reviewed or approved by AT&T.

  • Healthcare Strength — Coverage includes multiple medical-plan options, dental and vision insurance, mental-health resources, virtual care, and expanded services such as fertility support, caregiver resources, and doula or respite care. The breadth of offerings is positioned as a core benefit for full-time employees.
  • Retirement Support — A 401(k) with company match is available for many full-time U.S. employees, and some groups may also have access to defined-benefit plans. Retirement-planning resources and company contributions are emphasized.
  • Parental & Family Support — Programs include paid parental leave, caregiver leave, and resources for fertility, adoption, and elder care. These offerings complement standard time off and contribute to a well-rounded family benefits package.

AT&T Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Dallas, TX
150,000 Employees

What We Do

Bring us your biggest career aspirations. Share your boldest dreams. This is a moment to get energized. Through 5G and Fiber, AT&T provides connectivity that leads to smarter homes, safter communities, higher quality health care and more life-changing innovations. With AT&T, Connecting Changes Everything.

Gallery

Gallery

Similar Jobs

Hybrid
2 Locations
289097 Employees
Hybrid
San Antonio, TX, USA
289097 Employees
Hybrid
Plano, TX, USA
289097 Employees
Hybrid
Plano, TX, USA
289097 Employees

Similar Companies Hiring

Granted Thumbnail
Artificial Intelligence • Healthtech • Insurance • Mobile • Financial Services
New York, New York
23 Employees
Scotch Thumbnail
Artificial Intelligence • eCommerce • Fintech • Payments • Retail • Software • Analytics
US
35 Employees
Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
567 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account