Applied AI Engineer

Posted 15 Days Ago
Be an Early Applicant
Austin, TX, USA
In-Office
Senior level
Artificial Intelligence • Edtech • Gaming • Software
The Role
Build and operate production agentic AI systems, including prompts, tools, retrieval, guardrails, evaluations, observability, and rollout. Automate expert workflows, ensure compliance and safety, and establish human-in-the-loop controls. Deliver documentation, runbooks, dashboards, and evaluation harnesses while enabling teams to adopt AI safely. The role requires substantial software engineering experience, production LLM deployments, retrieval expertise, structured outputs, local model experimentation, and strong communication.
Summary Generated by Built In

Applied AI Engineer
What You'll Do:

  • Ship production agentic systems- Design and build agents and agentic workflows that solve a defined problem end to end. You own prompts, tools, retrieval, guardrails, observability, cost and latency budgets, and rollout.

  • Provide governance and security guidance surrounding AI designs, strategy, and usage.

  • Automate SME workflows- Identify high-leverage operational toil (review pipelines, content QA, labeling and ops loops, support triage, internal copilots) and partner with the SMEs running those workflows to define success criteria, validate outputs, and replace meaningful chunks of that work with AI systems they trust. SMEs are co-owners of quality from the start of the project.

  • Make AI features safe- Treat what you deliver as a regulated product. Design for relevant compliance frameworks (e.g., COPPA, FERPA, NIST CSF 2.0) from day one, run safety and bias evals before launch and continuously after, and build the human-in-the-loop and content-filtering controls AI features need before they reach end users.

  • Hand off what you ship- Production features leave your hands with documentation, runbooks, an eval harness, and dashboards.

  • Make AI features easier for the teams to utilize- foster a culture of safe AI adoption with manageable ecosystems that integrate and enable cross-functional teams to create and enhance their own workflows and automations.

What You Need:

  • 7+ years professional experience as an Engineer with at least 1+ years of hands-on experience building agentic systems on at least one modern stack (LangGraph, the Anthropic SDK / Claude Agent SDK, OpenAI Agents SDK, Pydantic-AI, Mastra, LlamaIndex, CrewAI, or a homegrown stack). We care that you have built and operated agentic systems in production, not which framework.

  • Strong Python familiarity plus one typed language for production services (TypeScript, Go, or similar). Cloud experience (AWS or GCP) and containerized deployment.

  • Software engineering foundation (formal or autodidact), with several years of production environment experience and a track record of leading systems to launch.

  • Multiple shipped LLM-powered features in a production environment, with concrete stories about what broke, how you fixed it, and what you would do differently.

  • Practical knowledge of common agentic patterns: ReAct, tool use with structured schemas, prompt chaining, routing, orchestrator-workers, evaluator-optimizer / reflection, and human-in-the-loop. You can decide when a deterministic workflow is the right answer instead of an autonomous agent.

  • Hands-on experience in a production environment with retrieval: chunking, embeddings, hybrid search, re-ranking, metadata filtering, and the failure modes of each. Working knowledge of grounding techniques that anchor generated answers in retrieved evidence (citation and quote extraction, faithfulness and refusal evals, post-hoc consistency checks).

  • Strong prompt-engineering practice: zero-shot, few-shot, and many-shot patterns; example selection and ordering; in-context learning and chain-of-thought.

  • Comfort with structured output and validation in production (provider-native structured outputs, Instructor, Pydantic-AI, Outlines, or a comparable approach).

  • Disciplined evaluation practice. You do not rely on subjective review to decide whether a system is ready.

  • Experience in standing up local models and creating and testing closed local model solutions.

  • Strong written and verbal communication. You can explain an architectural trade-off to an executive and to a junior engineer in the same week.

  • You are comfortable using AI coding tools heavily in your implementation workflow while you own problem framing, design choices, and verification. We measure your output by working systems delivered, not lines of code written.

Nice to Have:

  • Advanced retrieval experience: GraphRAG, agentic retrieval, evaluation-driven retrieval tuning, and hybrid retrieval at scale.

  • Direct or transferable experience with safety, privacy, and policy constraints in user-facing AI. K-12 or other regulated-domain experience is a strong plus.

  • Experience with prompt-optimization frameworks (DSPy, TEXTGRAD, AdalFlow) where they paid off in production.

  • A public repo, package, gist, or technical write-up of meaningful AI work, or a representative project you can describe in detail under your confidentiality constraints.

  • Open-source contributions to AI tooling (frameworks, agents, evals, MCP servers).

Skills Required

  • 7+ years of professional experience as an engineer
  • At least 1 year of hands-on experience building production agentic systems
  • Experience with a modern agentic systems stack such as LangGraph, Anthropic SDK, OpenAI Agents SDK, Pydantic-AI, Mastra, LlamaIndex, CrewAI, or an equivalent homegrown stack
  • Strong Python familiarity
  • Experience with at least one typed language for production services, such as TypeScript or Go
  • Cloud experience with AWS or GCP
  • Experience with containerized deployment
  • Several years of production software engineering experience
  • Track record of leading systems to launch
  • Multiple LLM-powered features shipped in production
  • Knowledge of agentic patterns including ReAct, tool use, prompt chaining, routing, orchestrator-workers, evaluator-optimizer, reflection, and human-in-the-loop
  • Production retrieval experience including chunking, embeddings, hybrid search, reranking, metadata filtering, grounding, citations, and evaluation
  • Strong prompt-engineering experience with zero-shot, few-shot, many-shot, example selection, in-context learning, and chain-of-thought
  • Production experience with structured output and validation using provider-native outputs, Instructor, Pydantic-AI, Outlines, or comparable tools
  • Disciplined evaluation practice for determining system readiness
  • Experience standing up and testing local models
  • Strong written and verbal communication skills
  • Comfort using AI coding tools heavily while owning problem framing, design, and verification
  • Advanced retrieval experience including GraphRAG, agentic retrieval, evaluation-driven tuning, or hybrid retrieval at scale
  • Experience with safety, privacy, and policy constraints in user-facing AI
  • K-12 or other regulated-domain experience
  • Experience with DSPy, TEXTGRAD, AdalFlow, or similar prompt-optimization frameworks
  • Public repository, package, gist, or technical write-up demonstrating meaningful AI work
  • Open-source contributions to AI tooling, frameworks, agents, evaluations, or MCP servers
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
218 Employees
Year Founded: 2018

What We Do

Level is an educational technology company that develops learning science products and academic content, focusing on ELA and Math. The company integrates game systems engineering, animation, and AI into its tools, employing a multidisciplinary team of designers, engineers, and content specialists to create its educational platform.

Similar Jobs

Hybrid
Plano, TX, USA
289097 Employees
2K-2K Annually

PwC Logo PwC

Applied AI Health Data System Engineer-Senior Manager

Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Hybrid
65 Locations
370000 Employees
124K-280K Annually

Qvest US Logo Qvest US

Staff Engineer

Digital Media • Information Technology • Professional Services • Consulting
In-Office or Remote
2 Locations
1300 Employees
80K-400K Annually

Qvest US Logo Qvest US

Senior Engineer

Digital Media • Information Technology • Professional Services • Consulting
In-Office or Remote
2 Locations
1300 Employees

Similar Companies Hiring

Revel Thumbnail
Aerospace • Hardware • Robotics • Software
Marina Del Rey, California
70 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
Vega Thumbnail
Artificial Intelligence • Automotive • Insurance • Transportation
US
65 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account