Software Principal Engineer - AI Quality

As a Principal AI Quality Lead, you will define and drive the quality engineering strategy for our production Generative AI and Agentic systems. You will establish automated evaluation frameworks, quality standards, and testing infrastructure that ensure our AI agents operate reliably, safely, and efficiently at scale. This is a high-impact technical leadership role where you'll build the foundation for trustworthy AI deployment, bridging AI/ML engineering expertise with quality engineering discipline. You'll architect the systems and practices that transform our approach from manual spot-checking to continuous, automated evaluation of complex agentic workflows.
 

Quality Infrastructure & Automation

• Architect and build automated evaluation frameworks for agentic workflows that assess behavior across effectiveness, efficiency, robustness, and safety dimensions.
• Design continuous evaluation pipelines that automatically test multi-step reasoning, tool selection patterns, error handling, and behavioral regressions across diverse scenarios.
• Establish observability requirements for AI agents including structured logging, trajectory tracing, and metrics collection for reasoning steps, tool calls, and execution paths.
• Build regression detection systems that identify quality degradation when prompts, models, tools, or system components change.
• Create synthetic test data generation pipelines and curated evaluation datasets that cover edge cases, adversarial scenarios, and real-world variability.
• Define comprehensive quality standards and evaluation methodologies specifically designed for agentic AI systems and LLM-based applications.
• Establish key quality metrics, SLIs, and SLOs for agent behavior including task completion rates, reasoning efficiency, cost per resolution, and safety compliance.
• Create quality gates and acceptance criteria that balance speed-to-production with reliability requirements.
• Develop responsible AI testing practices including bias detection, fairness evaluation, safety guardrails validation, and alignment verification.
• Build tooling and frameworks that enable both automated evaluation at scale and targeted diagnostic testing for failure investigation.
• Establish benchmark suites and golden datasets for continuous quality assessment across agent capabilities.
• Architect evaluation approaches for complex AI behaviors including chain-of-thought reasoning, tool orchestration, multi-turn conversations, and context management.
• Establish model evaluation practices including prompt testing, output validation, semantic correctness assessment, and hallucination detection.
• Partner closely with AI Engineering, Platform, and Product teams to embed quality practices into the development lifecycle from design through deployment.
• Serve as the technical authority on AI quality, influencing architectural decisions and advocating for quality as a foundational pillar.
• Collaborate with Data Science and ML Engineering teams to align evaluation methodologies with model development practices.
• Communicate quality insights, risk assessments, and recommendations clearly to technical and non-technical stakeholders.
• Build cross-functional alignment on quality standards, evaluation criteria, and production readiness requirements.
• Mentor and develop AI quality engineers, elevating team capabilities in evaluation frameworks, AI/ML concepts, and automation practices.
• Foster a culture of quality-first thinking, continuous improvement, and data-driven decision making.
• Build the organizational capability for AI quality as a core competency, not a reactive testing phase.
   

• 7+ years of experience in AI/ML engineering, data science, or ML quality/evaluation roles with deep technical expertise in model development and evaluation.
• Experience in LLM evaluation, Generative AI quality, or agentic system testing.
• Strong understanding of transformer architectures, prompt engineering, retrieval-augmented generation (RAG), and agentic frameworks (ReAct, chain-of-thought, tool use patterns).
• Deep knowledge of LLM failure modes including hallucinations, context limitations, prompt sensitivity, reasoning errors, and tool misuse.
• Hands-on experience with major LLM platforms (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI) and their evaluation capabilities.
• Proven track record building automated evaluation systems for AI/ML models or agentic workflows at scale.
• Strong experience with ML evaluation frameworks and tools (MLflow, Weights & Biases, LangSmith, custom evaluation pipelines).
• Expertise in designing evaluation metrics for non-deterministic systems beyond simple accuracy measures.
• Experience with A/B testing, experimentation frameworks, and statistical analysis for model comparison.
• Background in observability, instrumentation, and monitoring for production AI systems.
• Advanced programming skills in Python with experience building production-quality evaluation frameworks and automation tooling.
• Strong understanding of software architecture, APIs, distributed systems, and data pipelines.
• Experience with test automation frameworks (pytest, unittest) and CI/CD integration.
• Familiarity with infrastructure as code, containerization, and cloud platforms (AWS, Azure, GCP).
• Ability to write clean, maintainable, well-documented code and technical specifications.
• Strong analytical skills with ability to design statistically sound evaluation methodologies.
• Experience working with large-scale datasets, data quality assessment, and synthetic data generation.
• Understanding of experimental design, hypothesis testing, and confidence intervals for evaluation results.
• Ability to translate business requirements into measurable quality metrics and acceptance criteria.
   

Communication & Collaboration

• Excellent written and verbal communication skills with ability to explain complex technical concepts clearly.
• Experience presenting technical recommendations and quality assessments to senior leadership.
• Proven ability to build consensus and drive adoption of new practices across engineering organizations.
• Strong documentation skills for creating standards, runbooks, evaluation reports, and architectural specifications.
   

Learning & Innovation Mindset

• Self-directed learner who stays current with rapidly evolving AI quality practices and industry research.
• Comfortable operating in ambiguity and building new capabilities from the ground up.
   

• Master's in Computer Science, Machine Learning, Data Science, Statistics, or related field (or equivalent experience).
• 7+ years of professional experience in AI/ML & Software engineering, data science, ML operations, or ML quality roles.
• Hands-on experience with LLM evaluation, Generative AI testing, or agentic system quality.
• Demonstrated experience building automated evaluation frameworks and quality infrastructure for production AI systems.
   

• Published research or contributions in AI evaluation, LLM quality, agent benchmarking, or responsible AI.
• Experience building AI quality practices from the ground up in production environments.
• Deep expertise in agentic AI architectures including multi-agent systems, tool use, and autonomous decision-making.
• Background in both ML engineering/research and quality engineering/evaluation roles.
• Contributions to open-source AI evaluation frameworks or benchmarking tools.
• Hands-on experience fine-tuning or developing LLMs/SLMs with corresponding evaluation methodologies.
• Domain expertise in specific agentic AI applications (customer support, process automation, code generation, etc.).