Senior Quality Engineer (AI Platform Testing)

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.

Lilly’s Purpose:

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our 42,000+ employees across the globe work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work and put people first.

Come build advanced software capabilities to accelerate our digital transformation and support Lilly’s evolution to be the leader in Pharma-tech!

The Role:

The Software Product Engineering (SPE) organization is actively looking for a Senior Quality Engineer with strong hands-on experience in AI platform testing, chatbot testing, AI model validation, agents testing, AI test automation, and API testing. This is a highly specialized role focused on validating complex AI and ML systems and ensuring scalable, safe, and effective deployment of AI-based solutions.

UI automation experience using tools like Selenium, Cypress, or Playwright is desirable as a secondary skill.

What You’ll Be Doing:

You will drive quality engineering initiatives specifically focused on AI-powered platforms and solutions, including LLMs, chatbots, AI agents, and intelligent workflows. You’ll build robust test strategies and frameworks to validate data pipelines, model inference accuracy, prompt engineering, hallucination control, API contracts, and performance under real-world conditions.

This role requires strong analytical and problem-solving skills, a deep understanding of AI systems testing, and the ability to collaborate across multidisciplinary teams such as SWE, SRE, ML Engineering, and Product.

Key Responsibilities:

• AI Platform & Model Testing (Primary Focus):

  • Validate the behaviour and performance of AI/ML models, including LLMs, RAG pipelines, chatbots, and autonomous agents.

  • Design and execute prompt evaluation, response accuracy, toxicity detection, and hallucination control test scenarios.

  • Implement and enhance automated AI testing frameworks tailored to model versioning, retraining, and feedback loops.

  • Ensure quality in human-in-the-loop (HITL) and continuous learning pipelines.

• API Testing:

  • Conduct thorough API validation using Postman, REST Assured, or GraphQL, with a focus on AI service endpoints, inference APIs, and orchestrators.

  • Build robust integration test suites to ensure seamless functionality between APIs and underlying AI systems.

• AI Test Automation:

  • Build test harnesses to validate AI features through synthetic data, mock services, and model stubs.

  • Integrate test suites into CI/CD pipelines to ensure continuous validation of AI behaviors.

• UI and Functional Test Automation (Secondary Focus):

  • Support end-to-end automation of AI-powered applications using tools such as Selenium, Cypress, Playwright, and WebdriverIO.

  • Automate critical user journeys involving AI-enabled decisions and interactions.

• Collaboration & Test Strategy:

  • Work closely with ML Engineers, SREs, and Product Managers to translate model design into testable components.

  • Monitor AI behavior in production using observability tools and adjust quality strategies based on live insights.

  • Drive discussions on fairness, bias, explainability, and model drift.

• Agile & DevOps Integration:

  • Participate in Agile ceremonies and actively contribute to sprint planning, test case reviews, and retrospectives.

  • Collaborate with DevOps teams to embed AI testing into CI/CD workflows using tools like GitHub, Jenkins, and Azure DevOps.

Required Technical Skills & Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, AI/ML, or a related field

• 6+ years of experience in Quality Engineering with at least 2 years in AI platform testing or model validation

• Hands-on experience in AI model testing, chatbot testing, prompt tuning, or agent workflows

• Proficiency in AI test automation and API testing tools (Postman, REST Assured, GraphQL)

• Working knowledge of Python, JavaScript, or TypeScript

• Experience integrating tests into CI/CD pipelines using GitHub, Jenkins, or Azure DevOps

• Knowledge of OpenAI, Bedrock, Anthropic, LangChain, RAG, and vector stores

• Understanding of LLM evaluation techniques, including metrics like BLEU, ROUGE, Toxicity Score, and RAGAs

Preferred Qualifications:

• Experience testing AI applications hosted in multi-geographical and cloud-native environments (e.g., AWS, GCP, Azure)

• Exposure to AI observability platforms such as Weights & Biases, Arize AI, or WhyLabs

• Understanding of prompt engineering, embedding quality, and tokenization behaviour

• Familiarity with security, performance, or accessibility testing

• Experience with AI governance frameworks and regulatory compliance (e.g., FDA, HIPAA in AI contexts)

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly