Role
Overview
As a Test
Automation Lead at Dailoqa, you’ll architect and implement robust testing
frameworks for both software and AI/ML systems. You’ll bridge the gap between traditional
QA and AI-specific validation, ensuring seamless integration of automated
testing into CI/CD pipelines while addressing unique challenges like model
accuracy, GenAI output validation, and ethical AI compliance.
Key
Responsibilities
Test
Automation Strategy & Framework Design
- Design and implement scalable
test automation frameworks for frontend (UI/UX), backend APIs, and AI/ML model-serving endpoints using tools like Selenium,
Playwright, Postman, or custom Python/Java solutions.
- Build GenAI-specific test suites for validating prompt outputs,
LLM-based chat interfaces, RAG systems, and vector search accuracy.
- Develop performance testing strategies for AI pipelines (e.g., model
inference latency, resource utilization).
Continuous
Testing & CI/CD Integration
- Establish and maintain continuous testing pipelines integrated with GitHub Actions,
Jenkins, or GitLab CI/CD.
- Implement shift-left testing by embedding automated checks
into development workflows (e.g., unit tests, contract testing).
AI/ML
Model Validation
- Collaborate with data
scientists to test AI/ML models for accuracy, fairness, stability, and bias mitigation using tools like TensorFlow
Model Analysis or MLflow.
- Validate model drift and retraining pipelines to
ensure consistent performance in production.
Quality
Metrics & Reporting
- Define and track KPIs.
- Test coverage (code, data, scenarios)
- Defect leakage rate
- Automation ROI (time saved vs. maintenance
effort)
- Model accuracy thresholds
- Report risks and quality trends
to stakeholders in sprint reviews.
- Drive adoption of AI-specific
testing tools (e.g., LangChain for LLM testing, Great Expectations for
data validation).
Soft
Skills
- Strong problem-solving skills
for balancing speed and quality in fast-paced AI development.
- Ability to communicate
technical risks to non-technical stakeholders.
- Collaborative mindset to work
with cross-functional teams (data scientists, ML engineers, DevOps).
Requirements
Technical
Requirements
Must-Have
- 10 years in test automation,
with 2+ years validating AI/ML systems.
- Expertise in: Automation tools: Selenium, Playwright,
Cypress, REST Assured, Locust/JMeter
- CI/CD: Jenkins, GitHub Actions,
GitLab
- AI/ML testing: Model validation, drift
detection, GenAI output evaluation
- Languages: Python, Java, or JavaScript
- Certifications: ISTQB Advanced,
CAST, or equivalent.
- Experience with MLOps tools: MLflow, Kubeflow, TFX
- Familiarity with vector databases (Pinecone, Milvus) and RAG workflows.
- Strong programming/scripting
experience in JavaScript, Python, Java, or similar
- Experience with API testing, UI
testing, and automated pipelines
- Understanding of AI/ML model testing, output evaluation, and non-deterministic
behavior validation
- Experience with testing AI chatbots, LLM responses, prompt engineering
outcomes, or AI fairness/bias
- Familiarity with MLOps pipelines and automated validation of model performance
in production
- Exposure
to Agile/Scrum methodology and tools like Azure Boards
Skills Required
- 10 years of experience in test automation
- 2+ years validating AI/ML systems
- Expertise with Selenium, Playwright, Cypress, REST Assured, Locust, or JMeter
- Experience with Jenkins, GitHub Actions, or GitLab CI/CD
- Experience with AI/ML model validation, drift detection, and GenAI output evaluation
- Programming or scripting experience in Python, Java, JavaScript, or similar
- ISTQB Advanced, CAST, or equivalent certification
- Experience with MLflow, Kubeflow, or TFX
- Familiarity with Pinecone, Milvus, vector databases, and RAG workflows
- Experience with API testing, UI testing, and automated pipelines
- Understanding of AI/ML model testing, output evaluation, and non-deterministic behavior validation
- Experience testing AI chatbots, LLM responses, prompt engineering outcomes, or AI fairness and bias
- Familiarity with MLOps pipelines and automated production model-performance validation
- Exposure to Agile/Scrum methodology and Azure Boards
What We Do
Dailoqa is a compliance-first, AI-native, engineering-powered consultancy that builds agentic, enterprise-scale AI solutions for regulated financial institutions. Focused exclusively on financial services — retail, investment banking, corporate, wealth and insurance — it delivers Broccoli™, a hyperscaler-agnostic platform for production-ready, regulation-aware workflows and deploys live solutions across multiple regions.
