Job Title: Senior AI QA Automation Engineer
Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring
Experience: 5+ YOE.
Location: Costa Rica, Peru, Colombia, Bolivia.
Mode: Remote
We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.
Key Responsibilities
Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.
Required Skills & Qualifications
5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
Strong Python skills with experience building reusable automated testing frameworks.
Hands-on experience with Playwright and/or Selenium.
Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
Experience implementing CI/CD quality gates and production monitoring.
Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.
Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.
Preferred Skills:
Experience testing Generative AI, RAG solutions, copilots, or AI agents.
Experience with LLM evaluation frameworks and advanced prompt testing.
Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.
Experience with adversarial testing and Responsible AI testing.
Experience monitoring AI quality, drift, reliability, and failure patterns in production environments.
Posted On: 12-10-2026
At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.
Skills Required
- 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering
- Strong Python skills and experience building reusable automated testing frameworks
- Hands-on experience with Playwright and/or Selenium
- Experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation
- Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing
- Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds
- Knowledge of hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation
- Experience implementing CI/CD quality gates and production monitoring
- Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria
- Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field
- Experience testing Generative AI, RAG solutions, copilots, or AI agents
- Experience with LLM evaluation frameworks and advanced prompt testing
- Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants
- Experience with adversarial testing and Responsible AI testing
- Experience monitoring AI quality, drift, reliability, and failure patterns in production environments
Encora Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Encora and has not been reviewed or approved by Encora.
-
Healthcare Strength — Health coverage is described as employer-provided in multiple locations, with private plans and family coverage highlighted in Spain and Mexico. Medical insurance quality is presented as a recurring bright spot alongside standard coverage.
-
Leave & Time Off Breadth — Time off includes paid holidays and PTO, with regional materials indicating additional leave provisions in certain countries. Leave is generally portrayed as conventional to generous depending on location.
-
Flexible Benefits — Work-from-home flexibility is frequently highlighted as a plus, though it varies by role and client needs. Remote and hybrid options are positioned as part of the overall package.
Encora Insights
What We Do
Headquartered in Santa Clara, California, and backed by renowned private equity firms Advent International and Warburg Pincus, Encora is the preferred technology modernization and innovation partner to some of the world’s leading enterprise companies. It provides award-winning digital engineering services including Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering. Encora's deep cluster vertical capabilities extend across diverse industries, including HiTech, Healthcare & Life Sciences, Retail & CPG, Energy & Utilities, Banking Financial Services & Insurance, Travel, Hospitality & Logistics, Telecom & Media, Automotive, and other specialized industries. With over 9,000 associates in 47+ offices and delivery centers across the U.S., Canada, Latin America, Europe, India, and Southeast Asia, Encora delivers nearshore agility to clients anywhere in the world, coupled with expertise at scale in India. Encora’s Cloud-first, Data-first, AI-first approach enables clients to create differentiated enterprise value through technology
