QA AI Automation Engineer

Posted An Hour Ago
Be an Early Applicant
St. Petersburg, FL, USA
In-Office
Senior level
Financial Services
The Role
Build and own AI evaluation and QA automation frameworks for chat assistants, APIs, and agentic workflows. Develop golden datasets, adversarial tests, LLM judges, CI/CD quality gates, production observability, and multi-agent defect-detection workflows. Establish standards for agent-authored code using coverage, mutation testing, static analysis, security scanning, and architectural checks. Partner with product and engineering teams on acceptance criteria, release readiness, and quality strategy.
Summary Generated by Built In

Description

About the Role 

We are building AI systems that advisors and their clients rely on inside a regulated wealth management platform — an internal AI assistant, an external-facing API layer on top of it, and a growing set of agentic workflows. The bar for correctness, grounding, and auditability is higher here than it is for most AI products. 

This role exists because traditional QA does not cover that surface. We need an engineer who can evaluate non-deterministic systems, build the tooling that catches AI regressions before they reach an advisor, and hold the line on code quality now that a meaningful share of our code is written by autonomous coding agents. This is a hands-on engineering role, not a test-execution role. 

What You'll Do 

1. Own and extend our AI evaluation framework for AI chat 

  • Build and maintain evaluation suite for our AI assistant and its API surface — correctness, grounding/citation fidelity, retrieval quality, refusal and escalation behavior, tone, latency, and cost per interaction. 
  • Curate and version golden datasets and adversarial test sets drawn from real advisor workflows; keep them representative as the product changes. 
  • Design LLM-as-judge and rubric-based scoring where deterministic assertions do not apply and validate the judges themselves against human-labeled sets. 
  • Wire evals into CI so prompt, model, retrieval, and tool changes are gated by measured regression — not by vibes. 
  • Instrument production traces and close the loop from live failures back into the eval suite. 

2. Build the multi-agent AI quality step that finds defects automatically 

  • Design and operate a multi-agent workflow in our CI/CD pipeline that runs on commit: executes the suite, triages failures, writes up defects with reproduction detail, proposes or applies fixes, and re-runs to verify. 
  • Define agent roles, hand-off logic, guardrails, and the quality checks that decide when an agent's output is trustworthy enough to auto-apply versus route to a human. 
  • Make cost-aware calls on where an LLM us needed and where deterministic automation is simply the better tool. 
  • Report on the loop's real impact — autonomous resolution rate, false-positive rate, engineer hours returned. 

3. Scale code quality in the age of autonomous agent coding 

  • Define what “good” means for agent-authored code and build the automated gates that enforce it: coverage and mutation testing, static analysis, security and dependency scanning, architectural conformance, and review checklists tuned to how agents fail. 
  • Identify the failure modes specific to agent-generated code — plausible-but-wrong logic, silent scope creep, duplicated abstractions, missing edge-case handling, tests written to pass rather than to verify — and build detection for them. 
  • Partner with architects and delivery teams to keep velocity from outrunning quality as agent-assisted development scales across the org. 

Across all three 

  • Partner with product, engineering, and AI Labs to define coverage and acceptance criteria for AI features before they are built. 
  • Contribute to release-readiness and go/no-go decisions with evidence. 
  • Maintain and modernize our existing automation estate (API, UI, integration) and mentor engineers on

Requirements

Required 

  • 5+ years in software quality, test automation, or SDET work, with real ownership of automation architecture — not just test authoring. 
  • Demonstrated production experience with LLM-powered systems: agent patterns, prompt engineering, tool/function calling, and orchestration. 
  • Hands-on with major model APIs (OpenAI, Anthropic, Azure OpenAI) and AI-assisted development tooling (Claude Code, Copilot, Codex or equivalent). 
  • Strong programming in at least one of Python, TypeScript/JavaScript, Java, Kotlin, or C#, and comfort reading across the others. 
  • Experience building and interpreting evaluation criteria for systems without a single correct answer. 
  • Deep CI/CD fluency — pipeline integration, gating, monitoring, and logging. 
  • API testing depth and experience validating third-party integrations. 
  • Clear written communication; you can explain a quality risk to a product owner and a root cause to an engineer in the same day. 

Preferred 

  • Eval and observability tooling: Langfuse, Promptfoo, OpenTelemetry, New Relic, or equivalents. 
  • RAG fundamentals — embeddings, chunking strategy, vector search, retrieval evaluation. 
  • Workflow orchestration tooling (n8n or similar). 
  • Azure cloud services; .NET ecosystem exposure. 
  • BDD/Selenium or comparable UI automation at scale. 
  • Financial services, wealth management, or another regulated domain — or a clear appetite for what data governance means in one. 
  • Experience mentoring or leading distributed QA engineers. 

First 90 Days 

  • Days 1–30: Learn the assistant, the API layer, and the current eval and automation estate. Ship a baseline eval suite for one high-traffic chat workflow with results visible in CI. 
  • Days 31–60: Stand up the first multi-agent defect-detection step on a single service; establish the trust threshold for auto-applied fixes and publish the metrics. 
  • Days 61–90: Propose and begin rolling out the agent-authored code quality standard across delivery teams, with automated gates. 

Skills Required

  • 5+ years in software quality, test automation, or SDET work with ownership of automation architecture
  • Production experience with LLM-powered systems, including agent patterns, prompt engineering, tool/function calling, and orchestration
  • Hands-on experience with OpenAI, Anthropic, Azure OpenAI, or equivalent model APIs
  • Experience with AI-assisted development tools such as Claude Code, Copilot, or Codex
  • Strong programming in at least one of Python, TypeScript/JavaScript, Java, Kotlin, or C#
  • Experience evaluating systems without a single correct answer
  • Deep CI/CD experience including pipeline integration, gating, monitoring, and logging
  • Deep API testing experience and third-party integration validation
  • Clear written communication skills
  • Experience with Langfuse, Promptfoo, OpenTelemetry, New Relic, or equivalent evaluation and observability tools
  • RAG fundamentals including embeddings, chunking strategy, vector search, and retrieval evaluation
  • Workflow orchestration experience with n8n or similar tools
  • Azure cloud services and .NET ecosystem exposure
  • BDD, Selenium, or comparable UI automation experience at scale
  • Financial services, wealth management, regulated-domain experience, or understanding of data governance
  • Experience mentoring or leading distributed QA engineers
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: St. Petersburg, FL
108 Employees
Year Founded: 2010

What We Do

Dynasty Financial Partners works exclusively with a select group of truly independent advisors of integrity to help them better serve their clients while ultimately growing their business and enterprise value. We are the industry advocate for advisors and accomplish this through aligned business interests with our Network Partners, a shared set of values that drive a connected Community, and a common vision to be forward looking leaders within the wealth management industry and advocates for the advancement of financial education and literacy across the country. With the power of our people, processes, partners and platform, their businesses have the potential to run more profitably and grow faster. We offer access to flexible capital solutions to help advisors develop, expand, and scale their business. We also provide M&A support to firms looking to grow inorganically or to plan for succession. Through a strategic and cultural partnership that we foster with each of our Network firms, we seek to enhance their enterprise value and accelerate their growth. We accomplish this through our proprietary core service model that provides middle and back office support in the areas of Finance, Technology, Marketing, Compliance, Practice Management, Investments, Consulting and Community. Headquartered in St. Petersburg, Florida, and with four offices across the country, the Dynasty team powers the industry’s most rewarding opportunity for entrepreneurial independent advisors of integrity. Visit our website for more information: www.dynastyfinancialpartners.com Follow us on Twitter: @DynastyFP Subscribe on Youtube: http://bit.ly/1MKXhC8

Similar Jobs

Photon Logo Photon

Quality Assurance Automation Engineer

Agency • Information Technology
In-Office or Remote
2 Locations
5017 Employees
48K-168K Annually

Hilton Logo Hilton

Supervisor - Kitchen - Hilton Tampa Downtown

Events • Food • Travel • Hospitality • Hotels & Resorts
In-Office
Tampa, FL, USA
500000 Employees

Liberty Mutual Insurance Logo Liberty Mutual Insurance

Inside Sales Representative

Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Remote or Hybrid
9 Locations
40000 Employees
45K-85K Annually

Zscaler Logo Zscaler

Product Specialist

Cloud • Information Technology • Security • Software • Cybersecurity
Easy Apply
Remote or Hybrid
USA
8697 Employees
172K-245K Annually

Similar Companies Hiring

Granted Thumbnail
Artificial Intelligence • Healthtech • Insurance • Mobile • Financial Services
New York, New York
23 Employees
Hanover Park Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
42 Employees
Onshore Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
85 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account