The Role
The AI QA Analyst ensures generative models' accuracy and safety, evaluates model performance, and maintains high-quality datasets through rigorous testing and feedback processes.
Summary Generated by Built In
At Ryz Labs, we are looking for an AI QA Analyst to join one of our clients, an AI-first travel and hospitality company. In this role, you will act as the bridge between model development and user experience. Your primary goal is to ensure our generative models are accurate, safe, and contextually aware by implementing rigorous evaluation frameworks and data-driven insights.
Qualifications:
-Ability to spot subtle logical inconsistencies or factual inaccuracies in AI-generated content.
-Experience managing, cleaning, or labeling datasets for machine learning or NLP projects.
-Familiarity with the AI lifecycle, Prompt Engineering, and tools like Google Sheets/Excel (SQL or Python is a significant plus).
Key Responsibilities:
-Review and annotate complex conversation traces to rate response quality based on metrics such as helpfulness, honesty, and harmlessness (HHH).
-Build and maintain high-quality "Golden Datasets" and benchmarks to stress-test the model across various domains and edge cases.
-Conduct pre-deployment testing and A/B model comparisons to identify performance regressions or improvements.
-Categorize model failures (hallucinations, logic errors, tone drift) to provide actionable feedback to the Engineering and Research teams.
-Help define and refine the rubric for "what a good response looks like" as the product evolves.
About RYZ Labs:
RYZ Labs is a startup studio founded in 2021 by two lifelong entrepreneurs. The founders of RYZ have worked at some of the world's largest tech companies and some of the most iconic consumer brands. They have lived and worked in Argentina for many years and have decades of experience in Latam. What brought them together was their passion for the early phases of company creation and the idea of attracting the brightest talents in order to build industry-defining companies in a post-pandemic world.
Our teams are remote and distributed throughout the US and Latam. They use the latest cutting-edge cloud computing technologies to create scalable and resilient applications. We aim to provide diverse product solutions for different industries and plan to build a large number of startups in the upcoming years.
At RYZ, you will find yourself working with autonomy and efficiency, owning every step of your development. We provide an environment of opportunities, learning, growth, expansion, and challenging projects. You will deepen your experience while sharing and learning from a team of great professionals and specialists.
Our values and what to expect:
- Customer First Mentality - Every decision we make should be made through the lens of the customer.
- Bias for Action - urgency is critical, expect that the timeline to get something done is accelerated.
- Ownership - Step up if you see an opportunity to help, even if it's not your core responsibility.
- Humility and Respect - Be willing to learn, be vulnerable, and treat everyone who interacts with RYZ with respect.
- Frugality - being frugal and cost-conscious helps us do more with less
- Deliver Impact - get things done most efficiently.
- Raise our Standards - always be looking to improve our processes, our team, and our expectations. The status quo is not good enough and never should be.
Top Skills
AI
Excel
Google Sheets
Python
SQL
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Where startups soar! We're not just building startups - we're crafting the future. From ideation to execution, we provide the blueprint for startup success. We specialize in nurturing startups from the ground up, and supercharging existing ones with our top-tier technical talent solutions. Join us at Ryz Labs, where we turn promising ideas into thriving businesses. Let's shape the future of innovation together!
