Software Engineer - Quality

• Agentic AI - Customer & Commercial Pod
  In the Agentic AI – Customer & Commercial Pod, you’ll support quality assurance for AI experiences that help Customers get help, explore options, and complete transactions through natural, intuitive interactions. This role focuses on executing end‑to‑end testing strategies for agentic AI solutions, including multi‑agent workflows running in production environments. You’ll help build and maintain QA automation frameworks, create reusable test assets, and drive macro‑level validation across complex agentic flows. You’ll validate resiliency, reliability, accuracy, latency, orchestration correctness, and scale—helping ensure Customer‑facing AI experiences behave predictably and meet high quality standards from development through production.

• Agentic AI - ETO Support Pod
  In the Agentic AI – ETO Support Pod, you’ll help define and implement how the organization tests non‑deterministic, LLM‑driven systems that support enterprise and back‑office workflows. This role combines classic test automation with emerging AI evaluation techniques—using tools such as Playwright or pytest alongside LLM evaluation frameworks to validate behavior, safety, and regression over time. You’ll work shoulder‑to‑shoulder with AI Engineers to design quality gates, guardrails, and evaluation harnesses that directly influence what ships to production. Your contributions help establish practical, scalable quality practices for agentic systems that must operate reliably across internal technology platforms.

Responsibilities

• Apply knowledge and skills of software development and testing effectively to solve a range of problems
• Work alongside other engineers on the team to elevate technology and consistently apply best practices
• Collaborate closely with customers and cross-functional departments to communicate project statuses and proposals
• Document each aspect of a system or application as a reference for future upgrades and maintenance
• Determine and assess the needs of the user and then create software to meet the requirements
• Identify and resolve issues that arise during the design, testing and maintenance processes using problem-solving skills
• Work in an agile environment to deliver high-quality software
• Prepare and install solutions by determining and designing system specifications, standards, and programming
• Mentor junior members on the team
• Improve and expand technical capabilities by continuing their education thru reading, workshops, conferences, and/or communities of practice
• May perform other job duties as directed by Employee's Leaders

• Intermediate knowledge of software development methodologies, practices, concepts, and technologies obtained through formal training and / or work experience
• Ability to demonstrate consistent knowledge application, skills of software development, and testing to solve a range of problems
• Intermediate knowledge of at least one required programming language
• Ability to partner, communicate, and negotiate with various Technology or partner Teams
• Ability to analyze and manage large, complex, and vague Business or technical problems, articulating the problem or root cause, and translating the analysis into viable solution recommendations
• Ability to take on multiple assignments, whether administrative or project related, while maintaining a successful level of completion in all responsible work
• Ability to mentor others to do the same
• Ability to develop, present and effectively communicate ideas and strategies to a variety of audiences

• Required: Bachelor's degree in Computer Science, Engineering, Information Systems or related field and/or equivalent formal training

Experience

• Required: Intermediate-level experience, fully functioning broad knowledge in Software engineering
  • 2-5 years of relevant work-related experience
  • 3+ years in Software QA/Testing, with 1+ years in AI/ML or LLM-based systems; hands-on experience testing agentic/multi-agent architectures
  • Strong programming skills in Python or TypeScript/JavaScript; experience building test harnesses, simulators, and fixtures.
  • Experience with LLM evaluation (exact/soft match, BLEU/ROUGE, BERTScore, semantic similarity via embeddings), guardrails, and prompt testing.
  • Agentic AI performance/ Load testing experience and Agentic AI defect life cycle and defect triage experience
  • Expertise in distributed systems testing, latency profiling, resiliency patterns (circuit breakers, retries), chaos engineering, and message queues.

Other Qualifications

• Must meet confidentiality expectations as to confidential, proprietary and sensitive Company information
• Ability to work extended hours as needed