QA Engineer

We are looking for a QA Engineer based in Latin America to work on a long-term project for one of our clients, a Software Development company based in Atlanta, Georgia.

Our client is building a secure, bank-grade AI infrastructure and developing cutting-edge applications designed to power the next generation of modern banking and financial services.

Responsibilities

• Contribute to the development and maintenance of the AI evaluation framework for LLM and agentic AI outputs across Foundry, Agent Builder, and client-deployed environments, in collaboration with the QA Lead.
• Define and implement test assertions for new AI workflows, establish behavioral contracts under the guidance of the QA Lead, and help maintain regression baselines for model behavior across releases.
• Apply advanced QA approaches for non-deterministic AI outputs, using statistical reasoning (e.g., distributions and confidence intervals) instead of purely deterministic assertions, particularly when evaluating complex tasks such as summarization of long-form documents (e.g., loan agreements).
• Develop and maintain significant portions of the automated test suite, including end-to-end, integration, and regression tests covering backend APIs, document ingestion pipelines, AI inference workflows, and frontend surfaces.
• Contribute to performance and load testing for latency-sensitive inference paths.
• Ensure CI/CD quality gates remain stable and effective, and actively support the team in resolving pipeline failures.
• Participate in production incident triage, create reliable reproduction cases for identified issues, and implement regression tests to prevent recurrence.
• Produce test artifacts, audit logs, and process documentation supporting SOC 2 Type II compliance under the direction of the QA Lead.
• Partner with Forward Deployed Engineering to support client-side validation, investigate production issues, and assist with reproduction of real-world customer scenarios.

Requirements

• Advanced Level of English.
• 5+ years of hands-on Software Quality Assurance/Quality Engineering experience, with strong software engineering and test automation skills, including experience testing complex distributed systems and at least 1 year of hands-on testing and validation of AI/ML systems.
• Experience designing and executing test cases for Large Language Model (LLM) outputs, working within evaluation pipelines, and distinguishing between flaky tests and expected non-deterministic behavior in AI systems.
• Proficient in Python and TypeScript, with hands-on experience developing and maintaining automated test suites using pytest and Playwright.
• Hands-on experience with AI evaluation frameworks and tools such as RAGAS, DeepEval, LangSmith, or comparable LLM evaluation and testing solutions.
• Ability to independently troubleshoot and trace failures across the full technology stack—from application layer to infrastructure—with practical knowledge of Microsoft Azure, asynchronous/distributed systems, and REST APIs.
• Experience developing and maintaining automated tests within CI/CD pipelines, with a strong understanding of release quality gates and the ability to distinguish between controls that effectively mitigate risk and those that unnecessarily impede delivery.
• Strong collaboration and communication skills, with the ability to work effectively with QA Leads, fellow QA engineers, and software engineers to ensure product quality and successful delivery.

Bonus Points

• Bachelor’s Degree in Computer Science, Systems Engineering or related fields.
• Experience in fintech, banking, or other highly regulated industries.
• Familiarity with document processing pipelines, multi-agent AI architectures, Retrieval-Augmented Generation (RAG) validation, and observability platforms such as Arize, Langfuse, or similar AI monitoring and evaluation tools.

What we offer

• Long term positions
• Compensation in USD
• Paid time off
• Cool clients and products
• Work with great engineers

4tech