QA Analyst

This position is a US based remote role.

About Inizio Evoke

In the journey of life, your career should make a difference.

We believe everyone has a purpose at Evoke, and it's everyone's role to make health more human™. When you join Evoke, you become part of a dynamic global healthcare platform-adaptive, agile and innovative. Organized by global practice areas and specialty agencies, we provide access to a uniquely comprehensive offering to solve the most complex of challenges in today's healthcare market.

About the role

The AI QA Analyst is responsible for validating the quality, reliability, and trustworthiness of outcomes produced by an agentic AI platform operating across both ad-hoc and structured work modes. Unlike traditional QA roles focused solely on deterministic software behavior, this role is accountable for testing AI-generated outputs, multi-agent workflows, execution traceability, and lifecycle integrity across complex, evolving work states.

This role ensures that our AI agents produce accurate, explainable, compliant, and reproducible results while adhering to platform invariants such as attribution, auditability, promotion paths, and lifecycle correctness.

The AI QA Analyst will partner closely with Product and Business Analysis teams to embed quality signals directly into our execution model.

Here's what you'll be doing:

• Agentic AI Output Validation

  • Design and execute QA strategies to validate AI-generated artifacts, recommendations, and decisions produced by autonomous and semi-autonomous agents.

  • Evaluate outputs for accuracy, relevance, completeness, hallucination risk, and alignment with provided context modules and knowledge sources.

  • Validate that agent outputs are attributable, traceable, and reproducible within our execution and memory model.

  Multi-Agent Workflow & Lifecycle Testing

  • Test end-to-end agent workflows across the full lifecycle: Intent → Objective → Plan → Execution → Artifacts → Memory → Evolution

  • Validate correct behavior across both ad-hoc requests and structured deliverables, including seamless promotion between the two without loss of context or artifacts.

  • Ensure lifecycle state transitions, dependencies, and blocking logic behave deterministically and transparently.

  Platform & Execution Quality Assurance

  • Develop test cases for plan creation, phase execution, step execution, retries, halts, and re-runs across human and AI agents.

  • Validate execution logs, artifacts, decisions, and audit events for completeness and consistency.

  • Test failure modes, edge cases, and recovery scenarios in agent execution paths.

  Compliance, Governance & Explainability

  • Partner with Legal, Compliance, and Data teams to ensure QA coverage for regulated workflows, including healthcare-grade auditability and review readiness.

  • Validate that AI outputs meet governance requirements for explainability, source grounding, and decision rationale.

  • Ensure platform behavior aligns with responsible AI standards, including bias awareness and data usage constraints.

  Continuous Quality Improvement

  • Contribute to the design of automated testing frameworks for agentic systems, including regression testing of AI outputs over time.

  • Define and track quality metrics related to AI performance, output stability, and user trust signals.

  • Provide structured feedback loops to Product and Engineering teams to improve agent behavior, prompts, and orchestration logic.

About you

• Professional Experience

  • 4+ years of experience in Quality Assurance, Test Engineering, or Product Quality roles within complex software platforms.

  • Hands-on experience testing AI-enabled, data-driven, or workflow-oriented systems.

  • Proven ability to validate non-deterministic outputs where “correctness” is probabilistic rather than binary.

  Skills & Competencies

  • Strong understanding of QA methodologies across functional, integration, regression, and UAT testing.

  • Experience designing test cases for LLMs, generative AI systems, or intelligent automation.

  • Ability to reason about agent behavior, execution traces, and output quality across multi-step workflows.

  • Familiarity with Agile delivery environments and tools such as JIRA, Confluence, or similar.

  • Exceptional attention to detail and comfort working with complex system states and abstractions.

  Preferred Qualifications

  • Experience with agentic AI frameworks, orchestration systems, or multi-agent workflows.

  • Background in regulated industries such as healthcare, life sciences, or financial services.

  • Exposure to LLM evaluation techniques, prompt testing, or AI output benchmarking.

  • Familiarity with audit logging, compliance validation, or responsible AI practices.

  Success Metrics

  • Reduction in AI-related defects, regressions, and post-release quality issues.

  • Demonstrated improvement in consistency, trustworthiness, and explainability of agent outputs.

  • High confidence from Product, Compliance, and Engineering stakeholders in platform readiness.

  • Successful validation of complex agent workflows across structured and ad-hoc use cases.

  What This Role Offers

  • A front-row seat in shaping quality standards for agentic AI systems at enterprise scale.

  • Direct influence on how AI behavior is validated, trusted, and operationalized in real client work.

  • Collaboration with cross-functional leaders building a next-generation AI operating system.

  • The opportunity to help define what “quality” means when software reasons, plans, and executes autonomously.

Benefits: