We're looking for a QA Engineer (Mid or Senior level) who can own quality across AI-driven systems and the integrations that hang off them. This is not traditional app QA. You'll test LLM-powered features, prompt pipelines, agent workflows, MCP integrations, and the GitHub-based delivery pipelines that power them. You'll work directly inside our repos (including degen-engine and skeleton), partner with engineers using Claude Code and Gemini Code Assist, and shape how we verify non-deterministic systems.
If "how do you QA an LLM?" is a question you've already started answering — keep reading.
What You'll Do- Own end-to-end QA for Skeleton: AI agents, prompt pipelines, MCP server integrations, scheduled jobs (Vercel Cron), data ingestion (Apify), and database flows (Drizzle ORM).
- Design test strategies for non-deterministic systems: evaluation harnesses, golden datasets, regression suites for prompts, output quality scoring, hallucination and drift detection.
- Write and maintain integration tests across our stack (Next.js, TypeScript, pnpm, Vercel, Sentry, Jira) including API contract tests for third-party integrations.
- Test inside GitHub directly: review PRs, run test suites in CI/CD, validate auto-deploys to main, and verify fixes before they ship.
- Partner with engineers using Claude Code, Gemini Code Assist, and our broader AI dev workflow — including writing test prompts, validating tool-use outputs, and stress-testing prompt caching strategies.
- Build and maintain monitoring and observability for AI features in production (Sentry, custom eval dashboards, cost and latency tracking).
- Define quality gates and release criteria for AI-powered features, and partner with engineering on incident response when production outputs drift.
- Triage and reproduce issues across integrated systems — when something breaks, you trace it from Slack notification through Vercel logs, Sentry traces, the database, and back to the prompt.
- 3+ years of QA / SDET / Test Engineering experience on production software.
- Hands-on experience testing AI / LLM-powered features in production (OpenAI, Anthropic, Gemini, or similar) — prompt evals, output validation, regression testing.
- Strong TypeScript / JavaScript fundamentals; comfortable reading and writing code, not just black-box testing.
- Experience with modern web stacks: Next.js, REST/GraphQL APIs, serverless (Vercel / AWS Lambda), and at least one ORM (Drizzle, Prisma, etc.).
- Fluency in Git and GitHub workflows: PR review, branch protection, CI/CD pipelines, status checks.
- Experience writing automated tests with modern frameworks (Vitest, Jest, Playwright, Cypress).
- Comfort working in repos alongside engineers and contributing test code directly — not just filing tickets.
- Everything in Mid-Level, plus: deep experience defining QA strategy for AI / ML systems in production.
- Track record of building eval frameworks for LLM outputs (LLM-as-judge, golden datasets, A/B prompt testing, regression suites for non-deterministic systems).
- Experience with MCP (Model Context Protocol), tool use / function calling, agent frameworks, or multi-step LLM workflows.
- Comfort with observability stacks (Sentry, Datadog, custom dashboards) and ability to build them where they don't exist.
- Experience mentoring engineers on quality practices and shaping team-wide testing culture.
- Familiarity with prompt caching, model selection, context management, and other techniques for keeping AI systems fast and cheap in production.
- Direct experience with Claude (API, Claude Code, Anthropic SDK), Gemini Code Assist, or similar AI dev tools.
- Experience with Apify, Playwright, or other scraping / browser automation frameworks.
- Background testing data pipelines, ETL flows, or analytics systems.
- Experience with Jira automation, Slack apps, or Notion API.
- Open-source contributions to AI tooling or testing frameworks.
- Curiosity about prompt engineering, agent design, or the science of evaluating language models.
- Work on genuinely novel problems: QA for AI systems is being invented right now, and you'll help invent it here.
- Direct access to a small senior team building production AI pipelines from scratch — not a maintenance role, a frontier one.
- Modern stack, modern tools, no legacy debt to drag through.
Similar Jobs
What We Do
Builders leverages highly skilled distributed engineering teams with Latin America’s top software talent. Connecting top startups, mid-size companies and large enterprises across North America and beyond with superior tech talent needed to innovate, compete and excel in their field of business. Our deep expertise spreads across many areas, such as Data Science, Machine Learning, Software Engineering, DevOps, UX/UI Designers, Product Managers, and many more. Builders handles talent assessment, on-boarding, management, retention and continuous improvement. We provide unparalleled support, training, workshops, company events, benefits and other perks that make it fun and incredibly rewarding for everyone joining our team. Our team has extensive experience, deep relationships, an extensive network and a growing database, which make us a valuable partner for our clients. Builders prides itself on making the perfect match as easy as possible. All while letting you get back to building.
