The Role
Design and scale automated testing and validation frameworks for AI/ML models, LLM applications, and agentic systems. Develop unit, integration, regression, performance, safety, and end-to-end tests; create evaluation datasets and benchmarking pipelines; integrate validation into CI/CD; and implement observability, telemetry, quality gates, and audit readiness. Collaborate with ML, MLOps, Product, and Security teams to improve reliability, safety, compliance, and production quality.
Summary Generated by Built In
At Cadence, we hire and develop leaders and innovators who want to make an impact on the world of technology.
Senior AI Test / Automation Engineer
Overview
Role: AI Test / Automation Engineer
Location: Bangalore India
Department: AI Engineering / Quality Assurance
Experience Level: Mid to Senior
We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.
Key Responsibilities
- Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
- Develop comprehensive test suites, including:
- Unit, integration, and end-to-end (E2E)
- Functional, regression, performance, and safety testing
- Validate AI system behavior, including:
- Non-deterministic LLM outputs
- Hallucinations and edge cases
- Multi-step agent decision-making
- Design and manage evaluation systems:
- Golden datasets
- Benchmarking pipelines (accuracy, latency, reliability)
- Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
- Implement observability and telemetry to enable traceability, monitoring, and audit readiness
- Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
- Track and report quality KPIs, including test coverage, defect leakage, and system reliability
- Drive root-cause analysis and continuous improvement across the AI testing lifecycle
Required Skills
Core Engineering
- Strong programming skills in Python; familiarity with Bash, TypeScript, or Go
- Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
- Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
- Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)
AI / ML & Agentic Systems
- Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
- Familiarity with:
- RAG architectures and vector databases (Pinecone, Weaviate)
- Agent frameworks (LangChain, LlamaIndex, AutoGen)
AI Testing Techniques
- Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
- Knowledge of evaluation methods:
- LLM-as-a-judge
- BLEU, ROUGE, semantic similarity scoring
- Experience with prompt and agent regression testing
- Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection
Tooling (Preferred)
- AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases
- Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
- Monitoring: Prometheus, Grafana, OpenTelemetry
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and cross-functional collaboration
- Data-driven mindset with focus on quality KPIs
- Detail-oriented with a strong bias toward automation and scalability
Experience Requirements
- 7+ years in QA, SDET, or test automation engineering
- Proven experience building and scaling automation frameworks
- Hands-on experience with AI/ML systems or LLM-based applications
- Experience testing RAG pipelines or agentic workflows
- Owned end-to-end AI test strategy and architecture
- Defined quality metrics and release gates
- Delivered scalable validation pipelines for production AI systems
- Supported audit and compliance readiness
Preferred
- Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
- Exposure to:
- Shift-left testing practices
- Production observability and monitoring
- Chaos or resilience testing
Senior-Level Differentiators
Education
- Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field
Nice-to-have:
- ISTQB certification
- Cloud/ML certifications (AWS, Azure, GCP)
- AI testing certifications
What Success Looks Like
- AI systems that are accurate, reliable, and safe
- Fully automated test pipelines integrated into CI/CD
- Measurable improvements in defect leakage and model quality
- Strong observability and auditability across AI systems
- Scalable validation frameworks supporting rapid AI innovation
Skills Required
- 7+ years of experience in QA, SDET, or test automation engineering
- Strong Python programming skills
- Familiarity with Bash, TypeScript, or Go
- Experience building and scaling test automation frameworks
- Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
- Proficiency with CI/CD tools such as GitHub Actions, Jenkins, or GitLab CI
- Experience with cloud platforms such as AWS, Azure, or GCP
- Experience with Docker and Kubernetes
- Hands-on experience with AI/ML systems or LLM-based applications
- Experience testing RAG pipelines or agentic workflows
- Experience with non-deterministic testing approaches, statistical assertions, and tolerance thresholds
- Knowledge of LLM evaluation methods including LLM-as-a-judge, BLEU, ROUGE, and semantic similarity scoring
- Experience with prompt and agent regression testing
- Understanding of AI safety testing, adversarial testing, bias/fairness validation, and jailbreak detection
- Experience owning end-to-end AI test strategy and architecture
- Experience defining quality metrics and release gates
- Experience delivering scalable validation pipelines for production AI systems
- Experience supporting audit and compliance readiness
- Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field
- Experience in enterprise or regulated environments, including SOC 2 or ISO 27001
- Exposure to shift-left testing, production observability, chaos testing, or resilience testing
- ISTQB certification
- Cloud or ML certification from AWS, Azure, or GCP
- AI testing certification
Cadence Design Systems Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Cadence Design Systems and has not been reviewed or approved by Cadence Design Systems.
-
Equity Value & Accessibility — A discounted ESPP with a lookback feature and equity included in total compensation make ownership broadly accessible and potentially meaningful. Structured compensation at an industry leader adds predictability to equity participation.
-
Healthcare Strength — Medical, dental, and vision coverage are described as solid, with mental‑health/EAP and fertility support enhancing the offering. The breadth across core care and family‑building needs strengthens the healthcare package.
-
Leave & Time Off Breadth — Global Recharge Days, volunteer time off, and companywide breaks indicate a comprehensive time‑off framework. In addition, many salaried roles are described as having flexible or generous PTO policies.
Cadence Design Systems Insights
What's It Like to Work at Cadence Design Systems?
Cadence Design Systems Culture & Values
Cadence Design Systems Career Growth & Development
What's the Work-Life Balance Like at Cadence Design Systems?
Cadence Design Systems Leadership & Management
Cadence Design Systems Company Growth, Stability & Outlook
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Cadence enables electronic systems and semiconductor companies to create the innovative end products that are transforming the way people live, work and play. Cadence® software, hardware and IP are used by customers to deliver products to market faster. The company's Intelligent System Design strategy helps customers develop differentiated products—from chips to boards to intelligent systems—in mobile, consumer, cloud, data center, automotive, aerospace, IoT, industrial and other market segments. Cadence is listed as one of Fortune Magazine's 100 Best Companies to Work For.
