Job Title: Quality Assurance Engineer – AI Agentic Applications
Location: Pune, India
Corporate Title: AVP
Role Description
Quality Assurance Engineer to validate AI agentic applications across development and production. The position combines specialized AI testing, automation, safety evaluation, and continuous quality measurement. The engineer is responsible to design, build and execute the test strategy for LLM accuracy, RAG ingestion and attribution, agent planning and tool use, recovery behavior, and production failure patterns
What we’ll offer you
As part of our flexible scheme, here are just some of the benefits that you’ll enjoy
- Best in class leave policy
- Gender neutral parental leaves
- 100% reimbursement under childcare assistance benefit (gender neutral)
- Sponsorship for Industry relevant certifications and education
- Employee Assistance Program for you and your family members
- Comprehensive Hospitalization Insurance for you and your dependents
- Accident and Term life Insurance
- Complementary Health screening for 35 yrs. and above
Your key responsibilities
- Design, execute, and maintain test strategies and automation frameworks for AI-powered agentic workflows
- Develop and implement specialized testing approaches for LLM-powered applications, validating response accuracy, relevance, and grounding, while identifying hallucinations and edge-case behaviours
- Test and evaluate the behaviour of autonomous and semi-autonomous AI agents, ensuring proper planning, tool selection/sequencing, agent-to-agent interactions, and graceful error/timeout recovery
- Validate Retrieval-Augmented Generation (RAG) performance, including document ingestion, chunking, metadata quality, embedding accuracy, and citation/source attribution
- Perform AI safety and security testing to identify vulnerabilities such as prompt injection (direct/indirect), data leakage, unauthorized tool execution, and privilege escalation
- Establish quantitative AI evaluation metrics and quality gates (e.g., classification accuracy, groundedness, hallucination rate, tool-selection accuracy, and cost/latency) to measure performance across releases
- Build and maintain automated test suites (covering APIs, UI, integration, RAG, and agent trajectories) to progressively transition the testing suite toward automated continuous evaluation
- Analyze production AI interactions and failed agent journeys to identify recurring failure patterns, create regression tests from production incidents, and ensure decision auditability
Your skills and experience
- Bachelor's degree or above in Computer Science, Engineering, Information Technology, or a related discipline
- 5+ years of experience in software quality assurance and testing, with strong knowledge of test planning, functional, integration, regression, and end-to-end testing
- 1–2 years of hands-on experience specifically testing GenAI, LLM, or Agentic AI applications
- Strong experience with test automation and building automation frameworks and test utilities (rather than relying solely on manual tools)
- Strong programming ability in at least one language: Java / Kotlin or Python
- Strong understanding of generative AI technologies, including LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents/agentic workflows, and tool/function calling
- Hands-on experience with LLM and agent evaluation tools such as Langsmith/Langfuse, Ragas, or equivalent frameworks for dataset-based testing, custom evaluation metrics, experiment tracking, tracing, and continuous regression evaluation
Add On Experience (Nice to Have):
- Experience testing API and data integrations, including REST APIs, JSON, SQL, databases, and messaging systems like Kafka
- Good understanding of CI/CD integration and establishing automated quality gates
- Experience working in the financial services industry, with knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controls
- Experience with AI/LLM orchestration frameworks such as LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar technologies
- Hands-on experience with advanced AI testing capabilities such as prompt injection testing, AI evaluation/evals, and AI observability
- Experience testing distributed systems
Proven ability to leverage AI tools to enhance productivity, optimise workflows to solve business problems, while applying critical judgment to ensure responsible and ethical use of data and AI outputs.
How we’ll support you
- Training and development to help you excel in your career
- Coaching and support from experts in your team
- A culture of continuous learning to aid progression
- A range of flexible benefits that you can tailor to suit your needs
About us and our teams
Please visit our company website for further information:
https://www.db.com/company/company.html
We strive for a culture in which we are empowered to excel together every day. This includes acting responsibly, thinking commercially, taking initiative and working collaboratively.
Together we share and celebrate the successes of our people. Together we are Deutsche Bank Group.
We welcome applications from all people and promote a positive, fair and inclusive work environment.
Skills Required
- Bachelor's degree or above in Computer Science, Engineering, Information Technology, or a related discipline
- 5+ years of experience in software quality assurance and testing
- Strong knowledge of test planning, functional, integration, regression, and end-to-end testing
- 1-2 years of hands-on experience testing GenAI, LLM, or Agentic AI applications
- Strong experience building test automation frameworks and test utilities
- Strong programming ability in Java, Kotlin, or Python
- Strong understanding of LLMs, prompt engineering, RAG, embeddings, vector databases, AI agents, agentic workflows, and tool/function calling
- Hands-on experience with LLM and agent evaluation tools such as LangSmith, Langfuse, Ragas, or equivalent
- Experience testing APIs and data integrations, including REST APIs, JSON, SQL, databases, and Kafka
- Understanding of CI/CD integration and automated quality gates
- Financial services industry experience and knowledge of regulatory reporting, trade lifecycles, transaction reporting, reconciliations, and regulatory controls
- Experience with LangChain, LangGraph, LlamaIndex, Vertex AI, OpenAI, or similar orchestration technologies
- Experience with prompt injection testing, AI evaluations, and AI observability
- Experience testing distributed systems
Deutsche Bank Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Deutsche Bank and has not been reviewed or approved by Deutsche Bank.
-
Healthcare Strength — Medical coverage, mental-health support, counseling, preventive care, employee assistance programs, and wellness initiatives are broadly offered across regions. Availability and specifics vary by country and role.
-
Parental & Family Support — Paid parental leave, adoption and fertility assistance, surrogacy support, backup childcare, and family-care programs are available in many locations. In the United States, the package includes up to 16 weeks of paid parental leave.
-
Retirement Support — Pension or retirement plans, life insurance, disability coverage, employee banking benefits, and U.S. 401(k) matching with fixed employer contributions are highlighted. Employer retirement contributions in the U.S. vest over five years, requiring tenure to realize full value.
Deutsche Bank Insights
What We Do
At Deutsche Bank, we give original thinkers the space and support they need to shine. Merging local knowledge with global vision, in-depth insight with industry-leading digital expertise, if you’re an innovator by nature, we can help you to unleash your potential. We see things differently at Deutsche Bank – and we’re proud of our fresh perspective. Today, we’re driving growth through our strong client franchise, investing heavily in digital technologies, prioritising long-term success over short term gains, and serving society with ambition and integrity. Wherever your interests lie – in investment banking, trading, private wealth, asset management, retail banking - or many of the infrastructure functions that support them – you’ll discover resources, training and opportunities designed to keep you ahead of the curve. Intelligence has no boundaries: we welcome high-achieving, talented individuals from any background. If you’re full of imagination, enjoy solving problems and respond positively to complex challenges, discover a career to look forward to and join us!
.png){kind=logo}
