Senior QA Automation Engineer (Python | Selenium | Pytest)

Posted 5 Hours Ago
Be an Early Applicant
Hiring Remotely in Guatemala, GTM
Remote
Senior level
Software
The Role
Design and implement test architecture and semantic validation for AI/LLM pipelines. Build golden-set validation harnesses, automated unit/integration/E2E/access-control tests, and CI/CD quality gates. Verify multi-tenant isolation and DB-level RLS, mentor junior engineers, own test documentation, and collaborate with engineering and AI leads to enforce reproducible, blocking semantic regressions before production.
Summary Generated by Built In
3Pillar is an AI transformation partner on a mission to help enterprises build the AI-native products and intelligent agents that will define the next era of business. With teams across North America, Europe, Latin America, and Asia, we work with the most ambitious companies in financial services, healthcare, media, and technology — helping them move faster, modernize boldly, and compete on their own terms. Our HelixAI platform and Helix Pods delivery model put our engineers at the center of real agentic transformation — doing work that is open, portable, and built to last. We are building the future of enterprise AI. 
 

Senior QA Automation Engineer 

You'll own test architecture and semantic validation for a mission-critical AI-powered system where consistency guarantees are non-negotiable. This isn't traditional QA; you're designing test harnesses that prove AI outputs are correct, not just crash-free.

Your core responsibility: build a golden-set validation harness that measures whether the system's LLM stage produces the right decisions at the right confidence levels. You'll design automated tests across unit, integration, E2E, and access-control layers. You'll also establish CI/CD quality gates that block bad builds—schema validation failures, cross-tenant isolation defects, missing floor controls, semantic inconsistencies—before they reach production.

This is front-loaded work (weeks 1–5), then you support the team through delivery. You'll mentor a junior test engineer and set the standard for test coverage in a compressed timeline.

Desired Capabilities

  • Leadership skills

  • Mentoring and coaching abilities

  • Strategic thinking and decision-making skills

  • Project management skills

  • Stakeholder management and collaboration skills

  • Ability to motivate and inspire team members

  • Continuous improvement mindset

  • Familiarity with DevOps processes and tools.

  • Proficiency in programming languages such as Java, Python, or Ruby.

Key Responsibilities

  • Design comprehensive test architecture – plan unit, integration, E2E, and semantic validation test suites that cover the full pipeline from profile intake through roadmap generation; define coverage targets and acceptance criteria

  • Build semantic validation harnesses for AI outputs – design golden-set test cases (5–8 CISO-approved reference profiles) that measure AI stage precision/recall, rigor-tier correctness, floor-control invariants, traceability completeness, and run-to-run stability; a semantic regression blocks merge

  • Implement access-control testing – design automated probes at API and database layers that verify cross-tenant isolation; simulate compromised sessions and verify RLS policies enforce tenant boundaries

  • Design and maintain CI/CD quality gates – build automated test runners that execute on every commit: unit tests, integration tests, schema validation, migration guards, image scanning, access-control probes; red builds block merge

  • Lead test automation practices – expand automation frameworks and practices; identify opportunities for improving test processes; recommend tools and technologies for testing efficiency

  • Collaborate across the team – serve as the QA voice in technical design reviews; flag testability issues early; work with the AI/LLM Lead to define what "correct" means for each pipeline stage

  • Mentor junior test engineers – guide the team on test design, semantic validation, and CI/CD best practices; share knowledge on testing AI systems where reproducibility is critical

  • Own test documentation and reporting – maintain clear test plans, results, and traceability matrices; report test coverage and quality metrics to leadership; communicate blockers and risks transparently

Minimum Qualifications

  • 5+ years of hands-on software QA with a track record of:

  • Test strategy, planning, execution, and maintenance

  • Comprehensive test documentation and traceability

  • Advanced test automation (custom frameworks, advanced scripting)

  • 3+ years with automated testing tools – Selenium, Pytest, or equivalent; ability to write custom testing frameworks

  • Advanced proficiency in a programming language relevant to test automation:

  • Python (primary for this engagement; Pytest, async testing)

  • Or Java, C#, Go

  • Deep knowledge of databases and SQL – execute complex data validation, verify multi-tenant isolation at the DB layer, test RLS policies

  • CI/CD pipeline expertise – understand how tests integrate into automated deployment workflows; experience optimizing test execution for speed and reliability

  • Understanding of Agile/Scrum methodologies – ability to drive QA strategies in rapid, iterative development cycles

  • Proficiency in defect tracking (Jira or equivalent)

  • Proven leadership – mentor junior engineers, advocate for quality standards, drive process improvements

  • Exceptional communication skills – articulate complex technical issues to engineers, product, and stakeholders

  • Self-motivated and detail-oriented – work independently under time pressure, manage multiple priorities

  • Exceptional English proficiency – work effectively with distributed, global teams

Additional Experience Desired

  • ISTQB Advanced Level, CSTE, or equivalent QA certification

  • Performance and load testing (JMeter, LoadRunner) – design and execute tests under various conditions

  • Security testing practices – knowledge of OWASP vulnerabilities, experience with security testing tools (OWASP ZAP, Fortify)

  • Test-driven development (TDD) and behavior-driven development (BDD) – familiarity with Gherkin, Cucumber, or equivalent

  • Experience testing multi-tenant systems – understanding of isolation, RLS, and cross-tenant security validation

  • Experience with AI/ML systems – familiarity with testing LLM outputs, semantic validation, and consistency guarantees

  • API testing – REST/GraphQL endpoint validation, contract testing

  • Experience in regulated environments – audit logging, reproducibility, compliance testing

  • Demonstrated track record of process improvement – initiated and led projects that enhanced quality, reduced costs, or improved efficiency

Benefits

  • Medical Insurance benefits as per company policy. 

  • Life Insurance as per company policy

  • 15 days of paid vacation, sick leave, and paid holidays as per local law

  • Paternity and maternity leave as per as per local law

  • Marriage, bereavement and graduation leaves as per company policy

  • Sick leave and paid holidays as per local law

  • Christmas and mid-year bonuses as per local law

  • Monthly productivity bonus as per local law

  • Discounts in local shops

  • Direct deposit of payroll.

  • Paid professional certifications

What is it like working for 3Pillar Global?

    At 3Pillar, we offer a world of opportunity:

  • Imagine a flexible work environment – whether it's the office, your home, or a blend of both. From interviews to onboarding, we embody a remote-first approach. 

  • You will be part of a global team, learning from top talent around the world and across cultures, speaking English everyday. Our global workforce enables our team to leverage global resources to accomplish our work in efficient and effective teams. 

  • We’re big on your well-being – as a company, we spend a whole trimester in our annual cycle focused on wellbeing. Whether it is taking advantage of fitness offerings, mental health plans (country-dependent), or simply leveraging generous time off, we want all of our team members operating at their best.

  • Our professional services model enables us to accelerate career growth and development opportunities - across projects, offerings, and industries.

  • We are an equal opportunity employer. It goes without saying that we live by values like Intrinsic Dignity and Open Collaboration to create cutting-edge technology AND reinforce our commitment to diversity - globally and locally. 

  • Join us and be a part of a global tech community! 🌍💼  Check out our Linkedin site and Careers page to learn more about what it’s like to be part of our #oneteam! 

Skills Required

  • 5+ years of hands-on software QA experience
  • Test strategy, planning, execution, and maintenance
  • Comprehensive test documentation and traceability
  • Advanced test automation (custom frameworks, advanced scripting)
  • 3+ years with automated testing tools (Selenium, Pytest, or equivalent)
  • Advanced proficiency in a test automation language (Python primary; Java, C#, or Go accepted)
  • Experience with async testing in Python / Pytest
  • Ability to write custom testing frameworks
  • Deep knowledge of databases and SQL, including multi-tenant validation and RLS policies
  • CI/CD pipeline expertise and integrating tests into automated deployments
  • Experience with Agile/Scrum methodologies
  • Proficiency with defect tracking (Jira or equivalent)
  • Proven leadership, mentoring, and coaching experience
  • Exceptional communication skills and strong English proficiency
  • Self-motivated and detail-oriented with ability to manage priorities under pressure
  • ISTQB Advanced Level, CSTE, or equivalent QA certification
  • Performance and load testing tools and experience (JMeter, LoadRunner)
  • Security testing practices and tools (OWASP ZAP, Fortify)
  • TDD/BDD familiarity (Gherkin, Cucumber)
  • Experience testing multi-tenant systems and cross-tenant isolation
  • Experience with AI/ML systems testing LLM outputs and semantic validation
  • API testing experience (REST/GraphQL, contract testing)
  • Experience in regulated environments (audit logging, reproducibility, compliance)
  • Demonstrated track record of process improvement
  • Familiarity with DevOps processes and tools
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Fairfax, VA
1,784 Employees
Year Founded: 2006

What We Do

3Pillar Global builds breakthrough software products that power digital businesses. 3Pillar is an innovative product development partner whose solutions drive rapid revenue, market share, and customer growth for industry leaders in Software and SaaS, Media and Publishing, Information Services, and Retail. Leveraging a lean and agile approach, 3Pillar delivers value-generating, digital solutions with specialized product strategy and management, user experience design, as well as software and data engineering expertise across mobile, cloud, and disruptive technologies.

Similar Jobs

Smartly Logo Smartly

Paid Media Specialist

AdTech • Artificial Intelligence • Digital Media • Marketing Tech • Social Media • Software • Generative AI
Easy Apply
Remote or Hybrid
3 Locations
805 Employees
150K-150K Annually

Devsu Logo Devsu

Artificial Intelligence Engineer

Information Technology • Software
Remote
Guatemala, GTM
223 Employees
Remote
Guatemala, GTM
29452 Employees
Remote
Guatemala, GTM
1784 Employees

Similar Companies Hiring

Hanover Park Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
42 Employees
Kepler  Thumbnail
Fintech • Software
New York, New York
6 Employees
Onshore Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
60 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account