The Role
Build evaluation systems, model routing experiments, and scoring pipelines for LLM features. Work across MLOps, model versioning, monitoring, and product-facing engineering to ensure reliable, calibrated AI feature quality.
Summary Generated by Built In
A builder mindset is the core of this role and where you'll spend most of your time. But we're a small team building a whole product, not a research lab. The best person here treats ML systems as their primary craft while staying willing to do whatever the product needs — thinking through the product itself, shipping backend or frontend code, untangling data pipelines. We're looking for someone energized by the breadth, not someone who wants to stay in their lane.
What you'll work on
Evaluation systems for AI features
What you'll work on
Evaluation systems for AI features
- Help build the eval backbone our AI features ship against — failure taxonomies, LLM-as-judge rubrics, golden datasets, calibration against human judgment.
- Learn what it takes to keep automated scores honest as models and prompts change. A feature with no eval has no quality floor.
Model routing & inference economics
- Get hands-on with how we route work across models — balancing cost, quality, and latency per task.
- Help run the experiments that justify those choices and catch regressions.
Scoring, measurement & signal quality
- Work on turning noisy, real-world signals into scores you can actually trust — grounded in real statistical rigor, not vibes.
- Help move heuristic-driven approaches toward calibrated, monitored systems.
MLOps & production
- Get exposure to the full lifecycle — feature pipelines, model versioning, rollout, monitoring for drift and silent quality decay.
- Work alongside engineering to see how models get served reliably at low latency.
What we're looking for
Must have
Must have
- Currently pursuing or recently completed a degree in CS, DS, ML, or a related field.
- Some hands-on DS/ML experience — coursework, personal projects, research, or a prior internship — where you've built and run something end to end, not just notebooks.
- Comfort with Python and working SQL knowledge.
- Basic grounding in applied statistics — you can explain what a metric means and when it might be misleading.
- A builder's instinct — genuinely curious about product decisions, backend, or frontend, not just the modeling layer.
- Some exposure to LLMs — prompting, using APIs, or experimenting with model behavior.
Nice to have
- Any exposure to evaluation or observability tooling for LLM features.
- Coursework or projects in information retrieval, entity-matching, or record-linkage.
- Interest in developer-productivity, code analytics, or DevEx data.
We aspire to create an inclusive culture of diverse people not just because it's the right thing to do but because heterogeneity inspires us and is more fun! We employ people solely on merit and do not discriminate against any employee or applicant because of race, creed, color, religion, gender, sexual orientation, gender identity/expression, national origin, disability, age, genetic information, marital status, pregnancy or related condition (including breastfeeding), or any other basis protected by law.
Skills Required
- Currently pursuing or recently completed a degree in CS, Data Science, Machine Learning, or related field
- Hands-on DS/ML experience building and running end-to-end projects (coursework, projects, research, or prior internship)
- Comfort with Python
- Working knowledge of SQL
- Basic grounding in applied statistics (understand metrics and pitfalls)
- Builder's instinct: interest in product, backend, or frontend work beyond modeling
- Some exposure to LLMs (prompting, using APIs, experimenting with model behavior)
- Exposure to evaluation or observability tooling for LLM features
- Coursework or projects in information retrieval, entity-matching, or record-linkage
- Interest in developer-productivity, code analytics, or DevEx data
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Fi Money is on a path to revolutionise the way the next generation of Indians handle their money. And here’s why: 👉 We believe that information is power. The Fi Money app presents you with easy to understand information on your spending, investing, and saving habits. The more you know, the better your decision making. 👉 Simplicity wins. We shave off layers of complexity using tech, design, and communication to help you get closer to your money. Your money should always be in your control. 👉 Safe, Secure, World Class. Our team comes from the who’s-who of tech companies from around the world, distilling decades worth of knowledge into a product built for Indians. Our experience exists to improve yours. 👉Join us and build for the next billion. From data-driven decision making, to high quality tech and mentorship, we believe in giving Fi-ans a fulfilling career, because it takes the best to build the best.
