This role is for one of the Weekday's clients
Min Experience: 2+ years
Location: Bengaluru
JobType: full-time
We’re hiring an AI Product & Evals Analyst to own evaluation, quality measurement, dashboarding, and enterprise analytics for AI voice agents.
You will work at the intersection of AI Engineering, Product, and Customer Delivery to define how voice agent quality is measured, monitored, and improved at scale. This includes building evaluation frameworks, detecting regressions, analyzing business impact, and enabling enterprise teams with clear, actionable insights.
Beyond evaluation, we’re looking for someone with an AI-first mindset who can leverage modern AI tools and automation workflows to improve analytics, reporting, and operational efficiency. The role will also contribute toward building reusable analytics frameworks, internal intelligence tooling, and self-serve analytics capabilities for enterprise clients.
Your work will directly influence launch quality, customer trust, operational transparency, and how enterprise AI systems are measured and scaled.
Requirements
What will you do?
- Business & Enterprise Analytics
- Track and analyze business KPIs tied to voice agent deployments (automation, containment, conversion, CSAT, cost savings, etc.)
- Support enterprise billing, audit, and reconciliation workflows through reliable usage and conversation analytics
- Translate AI behavior into clear operational and business impact insights for enterprise clients
- AI-First Analytics & Automation
- Leverage AI tools and agentic workflows to improve speed, quality, and scalability of analytics, reporting, and RCA workflows
- Automate repetitive analytics and reporting tasks using LLMs, copilots, and workflow tooling
- Drive AI-native ways of working across SQL, Python, dashboards, and insight generation
- Analytics Platform & Internal Tooling
- Build reusable analytics frameworks, standardized dashboards, and internal tooling for enterprise deployments
- Contribute to self-serve analytics and intelligence capabilities such as text-to-data, AI-powered insights, and conversational analytics interfaces
- Help evolve Bluemachines’ analytics and observability layer for enterprise AI voice agents
- Voice Agent Evaluation
- Design and own AI eval frameworks specifically for voice agents
- Define conversation- and turn-level metrics
- Build and maintain golden voice datasets across accents, languages, and noise conditions
What We’re Looking For
Must-Have
- 2–5 years of experience in Product Analytics, Data Analytics, AI Quality, or related analytical roles
- Strong analytical thinking, metric design, and problem-solving skills
- Hands-on proficiency in SQL and Python for analysis, evaluation, and data investigation
- Ability to reason deeply about conversational, voice, and workflow-driven systems
- Experience building dashboards and enterprise-grade visualizations using tools like Tableau, Power BI, Looker, or equivalent
- Strong communication and storytelling skills with the ability to translate AI/system behavior into business insights
- Comfort working cross-functionally with AI Engineering, Product, Operations, and customer-facing teams
- AI-first mindset with strong adoption of modern AI tools, copilots, and automation workflows to improve productivity and scale analytics
Good-to-Have
- Experience with voice systems such as STT, TTS, call analytics, or conversational AI platforms
- Exposure to LLM-based products, prompt engineering, AI evals, or agentic workflows
- Experience with enterprise analytics, client-facing reporting, billing/reconciliation, or operational analytics
- Interest in building internal analytics tooling, reusable frameworks, or self-serve intelligence systems
- Experience working in fast-moving AI or startup environments
SQL, Python
Good-to-have skillsOpenAI Products, voice AI
Skills Required
- 2-5 years experience in Product Analytics, Data Analytics, AI Quality, or related analytical roles
- Proficiency in SQL for analysis and investigation
- Proficiency in Python for analysis, evaluation, and data investigation
- Experience building dashboards and enterprise-grade visualizations (Tableau, Power BI, Looker or equivalent)
- Strong analytical thinking, metric design, and problem-solving skills
- Ability to reason about conversational, voice, and workflow-driven systems
- Strong communication and storytelling skills to translate AI/system behavior into business insights
- Comfort working cross-functionally with AI Engineering, Product, Operations, and customer-facing teams
- AI-first mindset with adoption of modern AI tools, copilots, and automation workflows
- Experience with voice systems such as STT, TTS, call analytics, or conversational AI platforms
- Exposure to LLM-based products, prompt engineering, AI evals, or agentic workflows
- Experience with enterprise analytics, client-facing reporting, billing/reconciliation, or operational analytics
- Interest or experience building internal analytics tooling, reusable frameworks, or self-serve intelligence systems
- Experience working in fast-moving AI or startup environments
- Familiarity with OpenAI products or voice AI
What We Do
Weekday is an AI-powered recruitment platform that helps startups hire top-tier engineering and product talent. By leveraging a massive database of white-collar professionals and advanced outreach tools, the company streamlines the hiring process through automated sourcing, AI-driven resume screening, and white-glove contingency services. Their mission is to modernize recruitment by enabling companies to discover and engage passive candidates efficiently, ensuring high-quality hires for critical roles.
.png)
