We are Software Mind, an awesome team of engineers who are ready to ramp up any top-notch company’s projects! Our aim? To always be one step ahead. Become part of a multicultural company in constant growth with an excellent work environment certified by Great Place To Work!
Job DescriptionAbout the Project
Software Mind is building a private, tenant-isolated AI assistant for the real estate title and settlement industry. The platform is a retrieval-first (RAG) system that ingests historical email, documents, and structured metadata into a per-tenant vector index, and serves grounded, cited, expert-weighted answers through a chat-style Q&A interface with single sign-on and full audit logging.
The platform is AWS-native with a Python/FastAPI backend, Vue.js frontend, OpenSearch/Pinecone vector store, and OpenAI/Anthropic/Bedrock as LLM provider. You will join a senior, cross-functional LATAM-based team where hands-on AI delivery experience not just familiarity is the baseline expectation.
You build the validation harness that determines whether the AI system meets accuracy standards this is your primary deliverable. This is not generic test automation. You need to understand what 'correct' means for a RAG-based retrieval system and design test frameworks that can surface retrieval failures, hallucinated citations, confidence score drift, and RBAC access violations.
Your Responsibilities
Design and implement the validation harness for RAG output quality: retrieval accuracy, citation correctness, and grounding
Build automated test suites for the AI Extraction Gateway across Simple RAG and Complex RAG implementations
Develop and execute accuracy rubric test cases in collaboration with the BA and Designated Subject Matter Expert
Automate regression testing for confidence score calibration and source weighting behaviour
Test RBAC enforcement and role-specific filtered view access controls
Validate audit logging completeness and document lifecycle traceability
Build and maintain the incremental ingestion pipeline test suite
Contribute to go/no-go decision packs: produce accuracy reports and test evidence documentation
Tech Stack: Python, pytest, AWS, REST API Testing, Jira, Confluence
Qualifications
Must-Have Skills & Experience
6+ years in QA automation engineering; senior seniority required
Strong test automation engineering skills Python preferred (pytest or equivalent framework)
Experience with API testing and contract testing
Comfortable designing test frameworks for non-deterministic or probabilistic systems
Experience in agile/scrum environments with Jira-based test management
Nice-to-Have
Prior experience testing RAG systems, LLM outputs, or semantic search relevance this is a strong differentiator
Familiarity with AI evaluation frameworks: RAGAS, TruLens, or custom rubric-based evaluation approaches
Background in compliance-sensitive testing: audit trail validation, access control verification, or regulated-data environments
We are accepting applications from LATAM countries
Skills Required
- 6+ years in QA automation engineering
- Senior-level QA automation experience
- Strong automation skills with Python (pytest or equivalent)
- Experience with API testing and contract testing (REST APIs)
- Ability to design test frameworks for non-deterministic or probabilistic systems
- Experience in agile/scrum environments with Jira-based test management
- Familiarity with tech stack: FastAPI, Vue.js, OpenSearch, Pinecone, OpenAI/Anthropic/Bedrock, AWS, Confluence
- Prior experience testing RAG systems, LLM outputs, or semantic search relevance
- Familiarity with AI evaluation frameworks (RAGAS, TruLens, rubric-based evaluation)
- Background in compliance-sensitive testing: audit trail validation and access control verification
Software Mind Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Software Mind and has not been reviewed or approved by Software Mind.
-
Fair & Transparent Compensation — Pay is considered competitive for core hiring markets, with “good salary” cited in multiple locales. Public salary snapshots provide a baseline that helps candidates assess offers and negotiations.
-
Flexible Benefits — Remote or hybrid options are prominently highlighted, and a remote‑work program is publicly noted alongside positively cited work‑from‑home experiences. Flexibility around schedules and location is presented as part of the package.
-
Wellbeing & Lifestyle Benefits — Private medical care, language classes, sports/fitness support, and learning initiatives are listed for several Central/Eastern European locations, with occasional workation perks promoted. These lifestyle‑oriented offerings complement base pay and can enhance perceived total rewards.
Software Mind Insights
What We Do
Software Mind is a global digital transformation partner with operations throughout Europe, the US and LATAM. Driven by tech and empowered by people, we provide companies with software engineers and autonomous, cross-functional development teams who manage software life cycles from ideation to release and beyond. For over 20 years we’ve been enriching organizations with the talent they need to boost scalability, drive dynamic growth and bring disruptive ideas to life. Our top-notch engineering teams combine ownership with leading technologies, including cloud, AI, data science and embedded software to accelerate digital transformations and boost software delivery. A culture, driven by trust, that embraces openness, craves more and acts with respect enables our experts to create evolutive solutions that support scale-ups, unicorns and enterprise-level companies around the world.
