Automation Test Architect - Agentic AI

Reposted An Hour Ago
Be an Early Applicant
2 Locations
In-Office
90K-157K Annually
Senior level
Information Technology • Software
The Role
Define and lead an enterprise-level test automation strategy across various layers, focusing on scaling automation frameworks, improving quality, and mentoring engineers. Collaborate on testing AI systems, enhancing CI/CD processes, and reducing flakiness in automation.
Summary Generated by Built In
We are seeking a Test Automation Architect to define and lead an enterprise-grade test automation strategy across UI, API, data, and platform layers—while evolving our quality engineering capabilities to support Agentic AI / LLM-based systems (probabilistic outputs, confidence-based validation, hallucination detection, and model evaluation). This role will architect scalable automation frameworks, drive engineering excellence, and accelerate delivery through highly productive, maintainable automation programs.Key ResponsibilitiesAutomation Strategy & Architecture
  • Define and own the test automation architecture and multi-layer strategy (UI/API/contract/integration/e2e), aligned to product risk and release cadence.
  • Establish automation standards, patterns, and reference implementations (framework structure, page-object/screenplay usage, test data patterns, reporting).
  • Drive shift-left testing practices (contract testing, component testing, pre-merge validation) and reduce reliance on fragile end-to-end tests.
Agentic AI / LLM Test Model Alignment
  • Design and implement testing approaches for non-deterministic AI outputs, including:
    • confidence/threshold-based assertions
    • semantic similarity checks and rubric-based validation
    • goldens/benchmark datasets and evaluation harnesses
    • hallucination and groundedness testing strategies
    • regression detection across model/prompt/version changes
  • Define AI test coverage across accuracy, robustness, bias/safety (as applicable), performance, and reliability.
  • Partner with ML/AI, Product, and Engineering to establish acceptance criteria for AI capabilities and measurable quality gates.
High-Productivity Automation Programs
  • Build and scale high-throughput automation projects that are reliable, fast, and easy to extend.
  • Improve test execution time, reduce flaky tests, and optimize CI/CD pipelines (parallelization, test selection, retries with guardrails).
  • Create actionable, engineering-friendly quality dashboards (pass/fail trends, flake rate, coverage, defect escape metrics).
Tooling, CI/CD, and Quality Gates
  • Integrate automation into CI/CD with quality gates (PR checks, nightly suites, release readiness criteria).
  • Standardize observability for testing: logs, tracing hooks, screenshots/video, artifacts, and test run diagnostics.
  • Define practices for test environments, data management, service virtualization, and dependency control.
Technical Leadership & Mentorship
  • Serve as the QE technical authority: coach engineers on automation design, code quality, and best practices.
  • Lead design reviews for test architecture and automation PRs; enforce coding standards and maintainability.
  • Influence roadmap and delivery by translating risk into clear testing priorities.
Required Qualifications
  • 8–12+ years in Quality Engineering / Test Automation with architect-level ownership of frameworks and strategy.
  • Proven experience designing scalable automation across:
    • API testing (contract + integration)
    • UI automation with stable locator strategies (test IDs, accessibility roles)
    • Test data strategy and environment reliability
  • Strong programming skills in at least one modern language (e.g., Java, Python, TypeScript/JavaScript, C#).
  • Expertise with automation tools/frameworks (examples):
    • UI: Playwright / Cypress / Selenium
    • API: REST-assured / SuperTest / Postman/Newman / pytest
    • CI: GitHub Actions / Jenkins / Azure DevOps
  • Demonstrated ability to reduce flakiness and improve execution speed through engineering solutions.
Preferred Qualifications (Strongly Desired)
  • Experience testing LLM/Agentic AI features or other probabilistic systems, including:
    • evaluation harnesses, benchmark datasets, semantic scoring, and drift detection
    • prompt/version regression strategies and structured evaluation reports
  • Familiarity with modern quality approaches: test pyramid, contract testing, consumer-driven contracts, risk-based testing, shift-left.
  • Knowledge of security/performance testing integration (baseline understanding; hands-on is a plus).
  • Experience building reusable automation libraries for multiple teams/products.
Our Interview Practices

To maintain a fair and genuine hiring process, we kindly ask that all candidates participate in interviews without the assistance of AI tools or external prompts. Our interview process is designed to assess your individual skills, experiences, and communication style. We value authenticity and want to ensure we’re getting to know you—not a digital assistant. To help maintain this integrity, we ask to remove virtual backgrounds and include in-person interviews in our hiring process. Please note that use of AI-generated responses or third-party support during interviews will be grounds for disqualification from the recruitment process.

Applicants may be required to appear onsite at a Wolters Kluwer office as part of the recruitment process.


Compensation:

$89,600.00 - $157,000.00 USD
This role is eligible for Bonus.

Compensation range listed is based on primary location of the position.  Actual base salary offer is influenced by a wide array of factors including but not limited to skills, experience and actual hiring location. Your recruiter can share more information about the specific offer for the job location during the hiring process. 

Additional Information:

Wolters Kluwer offers a wide variety of competitive benefits and programs to help meet your needs and balance your work and personal life, including but not limited to: Medical, Dental, & Vision Plans, 401(k), FSA/HSA, Commuter Benefits, Tuition Assistance Plan, Vacation and Sick Time, and Paid Parental Leave. Full details of our benefits are available upon request.

Skills Required

  • 8 -12+ years in Quality Engineering / Test Automation
  • Proven experience designing scalable automation across API and UI testing
  • Strong programming skills in one modern programming language
  • Expertise with automation tools and frameworks
  • Ability to reduce flakiness and improve execution speed

Wolters Kluwer Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Wolters Kluwer and has not been reviewed or approved by Wolters Kluwer.

  • Leave & Time Off Breadth Time away benefits are positioned as broad, spanning vacation and sick time plus paid holidays and other covered leave types. Paid parental and caregiver leave, bereavement leave, and a volunteer day contribute to a more comprehensive time-off offering.
  • Retirement Support Retirement support is framed as meaningful through access to a 401(k)/retirement plan paired with company matching and additional contribution features in some descriptions. This is reinforced by mentions of profit sharing and other long-term savings-oriented programs.
  • Parental & Family Support Family-oriented support stands out through adoption assistance and paid parental leave provisions. These benefits are described alongside other caregiver supports that extend beyond basic leave categories.

Wolters Kluwer Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
Hagerstown, MD
18,996 Employees

What We Do

Wolters Kluwer (www.wolterskluwer.com) is a global leader in information services and solutions for professionals in the health, tax and accounting, risk and compliance, finance and legal sectors. We help our customers make critical decisions every day by providing expert solutions that combine deep domain knowledge with specialized technology and services. Founded in 1836 and headquartered in Alphen aan den Rijn, the Netherlands, the company serves customers in over 180 countries, maintains operations in over 40 countries and employs 18,600 people worldwide. Wolters Kluwer reported 2019 annual revenues of €4.6 billion. Listed on Euronext Amsterdam, Wolters Kluwer shares (WKL) are included in the AEX and Euronext 100 indices. Wolters Kluwer has a sponsored Level 1 American Depositary Receipt program. The ADRs are traded on the over-the-counter market in the U.S. (WTKWY).

Similar Jobs

Optum Logo Optum

Director, Quality Improvement & STARs - Medicare - Pearland, TX

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Pearland, TX, USA
160000 Employees
113K-193K Annually

Optum Logo Optum

Medical Assistant I/II/III - GI -Kelsey Seybold Clinic: Main Campus (Travel to Upper Kirby Clinic as needed)

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Houston, TX, USA
160000 Employees
16-32 Hourly

Optum Logo Optum

Senior Medical Assistant WellMed at Flower Mound, TX

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Flower Mound, TX, USA
160000 Employees
18-32 Hourly

SEON Logo SEON

Product Marketing Manager

Artificial Intelligence • Cybersecurity
Remote or Hybrid
US
415 Employees
5-7 Annually

Similar Companies Hiring

Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
178 Employees
Kepler  Thumbnail
Fintech • Software
New York, New York
6 Employees
Onshore Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
60 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account