The Role
Perform manual and automated testing of web applications and APIs, validate LLM outputs, design and execute test plans in TestRail, report defects in Jira, contribute automation under an Automation Lead, and collaborate in Agile sprints.
Summary Generated by Built In
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
We are looking for a Senior QA Engineer to perform manual and automation testing of web applications, validate AI-generated test cases and LLM outputs, and contribute automation scripts to a stable existing framework under the guidance of a dedicated Automation Lead. You will design and execute test plans in TestRail, conduct API testing with Postman, document defects in Jira, and collaborate across Agile sprint cycles. The role suits an engineer with a strong manual background who is actively growing their automation skills using tools including Claude and GitHub Copilot.
WHAT YOU WILL DO
- Perform manual and automation testing of web applications to identify bugs, glitches, and inconsistencies;
- Take ownership of automation tasks within the team and proactively develop your automation skills;
- Develop, maintain, and execute detailed manual test cases based on product requirements and specifications;
- Perform thorough cross-browser UI testing to ensure consistent behavior across Chrome, Firefox, Safari, Edge, and other browsers;
- Validate AI-generated test cases, summaries, and recommendations produced by integrated LLMs to ensure accuracy, completeness, and alignment with project requirements;
- Design and execute test scenarios to verify AI behavior, including prompt-response consistency, contextual relevance, and deterministic versus non-deterministic outputs;
- Evaluate AI model quality using defined acceptance metrics such as precision, recall, hallucination rate, and duplication rate;
- Conduct manual API testing to validate request/response accuracy, error handling, and integration;
- Use TestRail to create, manage, and track test plans, test cases, and test execution results;
- Work with Jenkins CI/CD pipelines;
- Report and document defects clearly using Jira;
- Participate in daily stand-ups, sprint planning, and other Agile team meetings;
- Review user stories and acceptance criteria to understand the scope of testing;
- Estimate own tasks and report risks to the team lead;
- Collaborate with developers and other QA team members to reproduce issues and verify fixes;
- Maintain and update test documentation as needed.
MUST HAVES
- 4+ years of experience in software testing;
- Strong manual and automation testing knowledge;
- Basic understanding of SDLC and STLC (Software Development and Testing Life Cycle) models;
- Knowledge of Testing and Quality Assurance processes;
- Strong experience with Test Design activities to cover a set of product requirements;
- Knowledge of Test Design techniques;
- Knowledge of testing methodologies;
- Defect analysis skills and the ability to write detailed and useful defect reports with expected defect-fixing outcomes;
- Experience with Postman (API testing) and browser developer tools;
- Experience with UI testing;
- Experience with TestRail as a Test Management system;
- Basic understanding of CI/CD processes;
- Intermediate English or higher.
NICE TO HAVES
- Understanding of LLM evaluation principles, including prompt engineering, output validation, and model fine-tuning cycles;
- Familiarity with AI/ML concepts, APIs, and frameworks such as OpenAI API, Azure OpenAI, and LangChain;
- Knowledge of data protection and compliance standards for AI systems (GDPR, zero-data retention, PII anonymization);
- Experience with SSO testing;
- Experience with Performance or Load testing.
PERKS AND BENEFITS
- Professional growth: Mentorship, TechTalks, and personalized growth roadmaps.
- Competitive compensation: USD-based pay with education, fitness, and team activity budgets.
- Exciting projects: Modern solutions with Fortune 500 and top product companies.
- Flextime: Flexible schedule with remote and office options.
Meet Our Recruitment Process
Application → Coding Challenge → Video Interview → Technical Interview or Hiring Manager Interview
Each step helps us understand your skills and overall fit.
If it’s a match, you’ll receive an offer.
Skills Required
- 4+ years of experience in software testing
- Strong manual and automation testing knowledge
- Basic understanding of SDLC and STLC
- Knowledge of Testing and Quality Assurance processes
- Strong experience with Test Design activities
- Knowledge of Test Design techniques
- Knowledge of testing methodologies
- Defect analysis skills and ability to write detailed defect reports
- Experience with Postman (API testing) and browser developer tools
- Experience with UI testing and cross-browser validation (Chrome, Firefox, Safari, Edge)
- Experience with TestRail as a Test Management system
- Basic understanding of CI/CD processes and experience with Jenkins
- Intermediate English or higher
- Understanding of LLM evaluation principles, prompt engineering, and output validation
- Familiarity with AI/ML concepts and APIs such as OpenAI API, Azure OpenAI, LangChain
- Knowledge of data protection and compliance standards for AI systems (GDPR, PII anonymization)
- Experience with SSO testing
- Experience with Performance or Load testing
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
AgileEngine is a privately held company established in 2010 that builds dedicated teams of designers and developers. We turn good ideas into awesome software that people actually want to use. Some of the biggest names and the hottest startups around the world chose us to build their tech.
