At Roche you can show up as yourself, embraced for the unique qualities you bring. Our culture encourages personal expression, open dialogue, and genuine connections, where you are valued, accepted and respected for who you are, allowing you to thrive both personally and professionally. This is how we aim to prevent, stop and cure diseases and ensure everyone has access to healthcare today and for generations to come. Join Roche, where every voice matters.
The PositionJob description
We are seeking a highly skilled AI Quality Engineer specializing in testing Generative (and non-Generative) AI applications and agentic systems to join our team. The ideal candidate will have expertise in both traditional software testing and specialized testing methodologies for Large Language Models (LLMs)-based and AI-based systems.
As a fullstack Quality Engineer within a lean, agentic software team, you will cover the entire spectrum of quality assurance. You will span manual exploratory testing, traditional automated UI/API/MCP testing (across our Python and TypeScript stack), classical machine learning validation, and cutting-edge generative and agentic AI evaluation. This role requires a deep understanding of LLM, agentic and ML systems behaviors, and testing frameworks to ensure the reliability, fairness, and effectiveness of our AI-powered solutions.
Description of the area
At Roche Digital Technology, we are advancing the boundaries of Applied AI. The Applied AI Engineering Team focuses on architecting, building, and operating high-value AI solutions and services to solve complex business challenges in healthcare.
In the 2026 tech landscape, we operate in small, highly autonomous agile teams (e.g., ~9 members) powered by advanced coding agents (like Claude Code) to develop and ship solutions faster than ever before. In this highly regulated environment, quality cannot be an afterthought. You will be the foundational pillar ensuring our rapidly developed agentic workflows and AI, GenAI, and agentic applications are safe, compliant, and robust before they reach the clinical or enterprise user.
Job Responsibilities
Generative & Agentic AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications and agentic systems, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability. You will expand this to evaluate autonomous multi-agent behaviors and tool-use accuracy.
Full-Stack Test Automation: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests. This spans both our Python backend services and modern TypeScript/React/Angular frontends.
Quality Assurance Framework & Compliance: Utilize Roche's testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required. Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with Roche practices.
Edge Case, Bias, and Fairness Testing: Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns. Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases.
Security & Performance Testing: Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns. Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring.
Qualifications
Hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies.
Be passionate about quality assurance and testing, particularly in the context of AI and GenAI applications and agents.
Have strong analytical and problem-solving skills with attention to detail.
Utilize critical thinking and problem-solving when using AI prompts (in e.g. NLP, data distributions and edge cases).
Be proactive in identifying potential issues and suggesting improvements.
Education / Experience
Experience: 7+ years of experience in software testing, with at least 1 year focused on genAI and/or agentic applications.
Regulated Environments: Be ready to work in a regulated environment with qualified infrastructure and validated applications, understand the difference between GxP and non-GxP products and the importance of CSV and ITSM processes in the delivery.
Agile Integration: Be experienced in working with Agile development teams.
Skills
Must-have:
Comprehensive Full-Stack Testing Mastery
Core Languages & Testing Frameworks: Advanced command of Python and standard testing frameworks (e.g., pytest, allure), alongside practical familiarity with specialized AI evaluation tools. High competence in TypeScript-driven test frameworks (such as Playwright, Jest, and Cypress) for thoroughly verifying dynamic generative user interfaces.
End-to-End Automation & CI/CD: Proven expertise in architecting and maintaining end-to-end automated testing suites that bridge traditional UI/API validation with complex, asynchronous AI/LLM evaluation pipelines.
Pipeline Integration: Hands-on experience integrating these comprehensive suites within GitHub Actions or modern CI/CD pipelines to ensure continuous, reliable quality checks throughout the entire development lifecycle.
Advanced AI Quality & Evaluation Mastery
AI Evaluation Frameworks: Expertise in implementing evaluation harnesses (e.g., RAGAS, DeepEval, Promptfoo) to quantify LLM performance, hallucination rates, and answer relevancy.
Agentic Workflow Validation: Ability to test autonomous agentic loops, including tool-use accuracy, step-by-step reasoning verification, and goal-directed performance metrics.
Evaluation Datasets: Experience in curating, managing, and versioning 'Golden Datasets' for benchmarking model responses against ground truth in both development and production environments.
Observability & Feedback Loops: Leveraging tracing (e.g., LangSmith, Arize Phoenix) to monitor production inputs/outputs, identify drift, and capture user feedback for continuous improvement.
Basic broad AI knowledge: Understanding of LLM architectures, RAG systems, agents, and common failure modes in AI applications.
Python Expertise: Advanced proficiency in writing robust, maintainable Python code for test automation (e.g., pytest). Experience using Python to orchestrate complex AI evaluation workflows, process large datasets for benchmarking, and automate interactions with LLM/AI services. Ability to integrate Python scripts into CI/CD pipelines for automated quality checks and performance observability in agentic systems.
Should-have:
Cloud: Practical experience with testing applications on cloud platforms (AWS) and working with cloud-based AI services.
Observability & ALM: Experience with monitoring and observability tools, log analysis, and performance metrics tracking (Grafana). Experienced in Roche mandatory ALM tools (Jira, GitHub).
Agentic SDLC & Engineering Excellence: Leverage AI coding assistants and autonomous agents (e.g., Claude Code, Ona) daily to accelerate full-stack development and testing cycles. Conduct rigorous code reviews for both human-written and AI-generated code.
Security: Understanding of security testing methodologies, particularly in the context of AI applications.
Could-have:
Regulatory Compliance: Proven understanding of AI quality assurance and testing standards within highly regulated industries.
Additional Qualifications
Stay current with the latest developments in AI and GenAI testing methodologies and best practices.
Have excellent communication skills and the ability to clearly document findings and recommendations.
Be able to communicate in English at the level of C1+.
Located in Hyderabad, India, with working hours structured to capture the 'golden hours' of overlap with Central European Time (typically running through the IST evening).
#Hyd2026
Who we are
A healthier future drives us to innovate. Together, more than 100’000 employees across the globe are dedicated to advance science, ensuring everyone has access to healthcare today and for generations to come. Our efforts result in more than 26 million people treated with our medicines and over 30 billion tests conducted using our Diagnostics products. We empower each other to explore new possibilities, foster creativity, and keep our ambitions high, so we can deliver life-changing healthcare solutions that make a global impact.
Let’s build a healthier future, together.
Roche is an Equal Opportunity Employer.
Skills Required
- Bachelor’s or master’s degree in Computer Science, Software Engineering, or an equivalent field
- 7+ years of experience in software testing
- At least 1 year of experience with generative AI and/or agentic applications
- Advanced Python proficiency for test automation and AI evaluation workflows
- Experience with pytest and standard testing frameworks
- Strong TypeScript test automation experience with Playwright, Jest, or Cypress
- Experience architecting UI, API, integration, performance, and end-to-end automated test suites
- Experience integrating automated testing into GitHub Actions or modern CI/CD pipelines
- Experience with AI evaluation frameworks such as RAGAS, DeepEval, or Promptfoo
- Experience testing agentic workflows, tool-use accuracy, and goal-directed performance
- Experience curating, managing, and versioning golden evaluation datasets
- Understanding of LLM architectures, RAG systems, agents, and AI failure modes
- Experience with observability and tracing tools such as LangSmith or Arize Phoenix
- Experience working with Agile development teams
- Ability to work in regulated environments and understand GxP, CSV, and ITSM processes
- Practical experience testing applications on AWS and cloud-based AI services
- Experience with Grafana, Jira, and GitHub
- Experience using AI coding assistants or autonomous agents such as Claude Code or Ona
- Understanding of AI security testing methodologies
- Understanding of AI quality assurance standards in regulated industries
- English communication proficiency at C1 or higher
- Located in Hyderabad, India and able to work during IST evening hours
Roche Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Roche and has not been reviewed or approved by Roche.
-
Retirement Support — U.S. materials describe a 401(k) with both matching and an additional company contribution, supported by formal plan documents and true‑up features. This structure is positioned as a standout element of the total package, particularly at Genentech.
-
Leave & Time Off Breadth — Time‑off provisions include substantial vacation, a year‑end shutdown, and a paid six‑week sabbatical after six years. These elements indicate a recharge‑oriented approach within the U.S. offering.
-
Healthcare Strength — Company materials emphasize comprehensive medical, dental, vision, and mental‑health resources alongside well‑being programs. Benefits pages consistently highlight breadth across core health coverage elements.
Roche Insights
What We Do
Roche is a global pioneer in pharmaceuticals and diagnostics focused on advancing science to improve people’s lives. The combined strengths of pharmaceuticals and diagnostics under one roof have made Roche the leader in personalised healthcare – a strategy that aims to fit the right treatment to each patient in the best way possible. Roche is the world’s largest biotech company, with truly differentiated medicines in oncology, immunology, infectious diseases, ophthalmology and diseases of the central nervous system. Roche is also the world leader in in vitro diagnostics and tissue-based cancer diagnostics, and a frontrunner in diabetes management. Founded in 1896, Roche continues to search for better ways to prevent, diagnose and treat diseases and make a sustainable contribution to society. The company also aims to improve patient access to medical innovations by working with all relevant stakeholders. Thirty medicines developed by Roche are included in the World Health Organization Model Lists of Essential Medicines, among them life-saving antibiotics, antimalarials and cancer medicines. Roche has been recognised as the Group Leader in sustainability within the Pharmaceuticals, Biotechnology & Life Sciences Industry ten years in a row by the Dow Jones Sustainability Indices (DJSI).
.jpeg)
