Senior AI QA Test Automation Engineer – CX Agent

Posted 2 Days Ago
Be an Early Applicant
4 Locations
Remote or Hybrid
Senior level
Fintech • Financial Services
The Role
Lead AI quality engineering for a multi-agent customer-support system. Build scalable evaluation pipelines and automation frameworks for accuracy, safety, hallucination, relevance, and task completion. Evaluate agent-orchestration technologies, integrate AI testing into GitLab CI/CD and cloud-native environments, and use observability data for quality analytics and test prioritization. Collaborate with QA, Data Science, and Engineering while establishing standards, researching AI testing methods, and mentoring engineers.
Summary Generated by Built In
The Role: 

We're seeking a Senior AI QA Test Automation Engineer to take a leading technical role in AI quality and test automation. You will work primarily with our AWS Bedrock AgentCore-based CX Agent Suite, a multi-agent customer-support system covering intent routing, FAQs, deposit-status queries, and human handoff. You will design and build the evaluation and QA tooling required to validate AI agents reliably, while also contributing to technical standards and best practices across QA. 

This is a Senior Individual Contributor (IC) role with a strong technical and strategic focus. You'll work closely with QA leadership, Data Science, and Engineering to architect scalable AI evaluation systems, establish technical standards, and drive the evolution of our AI evaluation framework. Our current evaluation harness is based on DeepEval and is evolving towards a broader agentic evaluation architecture. You will have the opportunity to evaluate and prototype emerging agent-orchestration approaches, including Strands Agents, and contribute to architectural decisions around their adoption. 

The main responsibilities of the position include:

  • Act as the primary technical enabler for QA, building scalable AI/ML frameworks, libraries, and tooling for the CX Agent Suite and beyond
  • Design and build AI evaluation pipelines that assess the CX Agent Suite's multi-agent responses (Intent Detector, FAQ Agent, Missing-Deposits Agent) for accuracy, relevance, tone, hallucination rate, safety/guardrail compliance, and task completion — extending or replacing the current DeepEval-based evaluation harness
  • Evaluate and prototype Strands Agents (or comparable agent-orchestration frameworks) for building self-evolving, autonomous QA agents, and drive the go/no-go decision on adoption
  • Collaborate with QA, Data Science, and Engineering to integrate AI-driven testing into the existing GitLab CI/CD pipeline (dev → test → staging → prod), alongside Terraform-provisioned, EKS/AgentCore-hosted services
  • Build resilient and adaptive automation that can detect and respond to changes in agent behavior, Bedrock Guardrails configuration, and routing logic
  • Develop data-driven quality analytics, including root-cause analysis, quality trends, and intelligent test prioritization, leveraging existing observability and evaluation data from OpenTelemetry, CloudWatch, X-Ray traces, and LangFuse offline evaluation runs
  • Lead research into emerging AI testing methodologies and mentor engineers through code reviews, workshops, and architectural guidance 

Main requirements:

  • BSc/MSc in Computer Science, AI, or a related discipline
  • 6+ years of hands-on experience in QA/Test Automation, with strong experience designing and maintaining automation frameworks
  • 1+ years of experience applying AI/ML in software testing or QA process improvement, with a proven track record of bringing AI/ML solutions into production workflows  
  • Strong Python skills, with experience working in Python/AWS-centric technology environments; Java and/or TypeScript is a plus
  • Hands-on experience with LLM evaluation techniques, including LLM-as-a-judge, human-in-the-loop evaluation, RAG, and multi-agent orchestration patterns  
  • Practical experience with DeepEval or comparable evaluation frameworks, along with familiarity with agent-orchestration SDKs such as Strands Agents
  • Knowledge of LLM tooling, vector databases, and MLOps pipelines
  • Experience integrating AI tooling into enterprise CI/CD environments, particularly GitLab, and working with containerized cloud-native environments such as Docker and Kubernetes/EKS
  • Strong communicator, able to influence technical decisions and drive engineering standards across teams 

The following will be considered an advantage:

  • Experience with autonomous QA agents or agentic orchestration frameworks for self-evolving test suites
  • Experience with LLM observability tools such as LangFuse, LangSmith, or Arize, particularly for measuring probabilistic and adversarial robustness  
  • Knowledge of AI ethics, fairness, and bias detection for guardrail and model validation
  • Experience with gRPC, WebSockets, and/or HTTP/2
  • Experience with AWS Bedrock, plus familiarity with GCP Vertex AI and/or Azure AI 

Benefit from:

  • Attractive remuneration package
  • Intellectually stimulating work environment
  • Continuous personal development and international training opportunities 

The Hiring Experience: What Awaits You

  • Let’s Connect – Intro Chat with Talent Acquisition
  • Deep Dive – First Interview with Your Future Team
  • Final Connection – Final Interview 

All applications will be treated with strict confidentiality!
 
By submitting this application, I confirm that I have read and agree to the terms of the Privacy and Data Protection Notice and the Confidentiality Undertaking.

Skills Required

  • BSc or MSc in Computer Science, AI, or a related discipline
  • 6+ years of hands-on QA or test automation experience
  • 1+ year applying AI/ML in software testing or QA process improvement, including production implementation
  • Strong Python skills in Python and AWS-centric technology environments
  • Experience with LLM evaluation techniques, including LLM-as-a-judge, human-in-the-loop evaluation, RAG, and multi-agent orchestration
  • Experience with DeepEval or comparable evaluation frameworks and agent-orchestration SDKs such as Strands Agents
  • Knowledge of LLM tooling, vector databases, and MLOps pipelines
  • Experience integrating AI tooling into enterprise CI/CD environments, particularly GitLab
  • Experience with containerized cloud-native environments such as Docker and Kubernetes/EKS
  • Strong communication skills and ability to influence technical decisions and engineering standards
  • Java and/or TypeScript experience
  • Experience with autonomous QA agents or agentic orchestration frameworks
  • Experience with LangFuse, LangSmith, or Arize
  • Knowledge of AI ethics, fairness, and bias detection
  • Experience with gRPC, WebSockets, and/or HTTP/2
  • Experience with AWS Bedrock, GCP Vertex AI, and/or Azure AI
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Limassol
3,353 Employees
Year Founded: 2009

What We Do

XM is a trading platform trusted by over 20 million traders, offering easy access to 1400+ global assets with low spreads, exceptional conditions, and super-fast execution for Forex and CFD trading.

Similar Jobs

CrowdStrike Logo CrowdStrike

Sales Engineer

Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Remote or Hybrid
Greece
11000 Employees

Argyle Logo Argyle

Senior Data Analyst

Big Data • Cloud • Fintech • Professional Services • Software
Remote
27 Locations
80 Employees
70K-85K Annually

Pfizer Logo Pfizer

Director, Oncology Scientific Communications

Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
In-Office or Remote
34 Locations
121990 Employees
177K-294K Annually

Pfizer Logo Pfizer

Director, AI Product Lead

Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Remote or Hybrid
28 Locations
121990 Employees

Similar Companies Hiring

Hanover Park Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
42 Employees
Kepler  Thumbnail
Artificial Intelligence • Fintech • Software
New York, New York
9 Employees
Onshore Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
85 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account