Research Engineering Intern, AI Evaluations

Posted 19 Days Ago
Bengaluru, Bengaluru Urban, Karnataka, IND
In-Office
Internship
Artificial Intelligence • Information Technology
The Role
Build and operate AI evaluation pipelines, including dataset preparation, model inference, annotation task setup, monitoring, quality checks, statistical analysis, experiments, and technical reporting. The intern will develop reliable, reproducible tooling for data processing and evaluation while debugging platform issues and collaborating with engineering, research, operations, and domain experts.
Summary Generated by Built In

Karya’s mission is to leverage the AI revolution to create dignified earning and learning opportunities for low-income communities across the world. In India, our smartphone-based digital work platform has enabled over 120,000 people to build language datasets and evaluate AI models while earning fair wages. With pilots in Kenya and Ethiopia and partnerships with leading AI labs, we are advancing language technologies for underrepresented communities. By combining dignified digital work with frontier AI innovation, Karya hopes to build a new AI economy that is equitable, inclusive, and community driven.

About the Role:

Karya's AI Evaluations team measures how well AI models serve people in the languages and contexts global benchmarks miss. We work with partners including Anthropic and Microsoft Research, alongside government and civil society, running evaluations on our own platform with native speakers and domain experts across India and beyond. Our benchmarks, leaderboards and reports aim to help model developers and decision-makers understand where models work well and where they fall short.

As a Research Engineering Intern, you will work alongside research engineers to build and run these evaluations, from setting up annotation tasks and running model inference to analysing results. You will own real parts of the pipeline end to end, including keeping them running when things break.

What You'll Do 
  • Evaluation setup: prepare datasets, run model inference through APIs and self-hosted setups, and configure, batch and launch annotation tasks on the Karya platform.
  • Pipeline operations: monitor live evaluations, triage and reproduce annotation app issues, coordinate fixes with platform and operations teams, and document recurring problems.
  • Quality control: run automated checks on annotation data and flag anomalies before they affect results.
  • Analysis and experiments: compute agreement, scores and breakdowns; support focused studies such as validating automated judges against human ratings; and contribute to technical reports and public releases.
  • Infrastructure: build and maintain tested, documented tooling for data processing, QC and analysis, with logged configs for reproducibility.
What You BringMust-Have:
  • Currently pursuing or recently completed a Bachelor's or Master's in Computer Science, Machine Learning, NLP or a related field.
  • Strong Python and comfort working with data (pandas or similar).
  • Working knowledge of LLMs and modern ML, and familiarity with calling models via APIs.
  • Solid foundation in probability and statistics.
  • Methodical debugging skills, and willingness to dig into unfamiliar systems when something breaks.

Strong differentiators

  • Experience with ML projects involving evaluation, annotation or human feedback data.
  • Exposure to speech or multilingual NLP, especially Indian languages.
  • Experience with GPU inference, cloud tooling or Android/app debugging.

Mindset

  • Takes ownership of work end to end, knowing that reliable operations are what make evaluations trustworthy.
  • Thrives in interdisciplinary teams spanning engineering, research, social science, operations and domain experts.
  • Comfortable with ambiguity, messy data and shifting priorities.
  • Cares about building AI that works for the people it is meant to serve.

Karya celebrates diversity and is an equal opportunity employer. All applicants will be considered without regard to race, religion, gender identity, sexual orientation, disability, or any other protected status.

 

Skills Required

  • Currently pursuing or recently completed a Bachelor's or Master's degree in Computer Science, Machine Learning, NLP, or a related field
  • Strong Python skills and comfort working with data using pandas or similar tools
  • Working knowledge of large language models and modern machine learning, including familiarity with calling models through APIs
  • Solid foundation in probability and statistics
  • Methodical debugging skills and willingness to investigate unfamiliar systems
  • Experience with machine learning projects involving evaluation, annotation, or human feedback data
  • Exposure to speech or multilingual NLP, especially Indian languages
  • Experience with GPU inference, cloud tooling, or Android/app debugging
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
Bengaluru
92 Employees
Year Founded: 2022

What We Do

At Karya, we are on a mission to bring AI-based earning and learning opportunities to low-income communities. The AI revolution has created a massive amount of digital work surrounding data generation, labelling, and enrichment. From building multimodal Indic language datasets and human-in-the-loop tasks to mitigating bias and culturally-sensitive LLM evaluations, Karya’s workers in rural India enable cutting-edge, inclusive AI innovations. We have also built a one-of-a-kind digital work platform that divides AI-data tasks into microtasks and distributes them to individuals in need of supplementary income. So far, our platform has been used to distribute 40 million+ tasks to 40,000+ people in India. Our platform and model allows us to provide our workers some of the highest wages in the global data market - nearly 20 times the Indian minimum wage. We were on the cover of the Time Magazine and Bloomberg last year, and Microsoft CEO, Satya Nadella recently highlighted our work in his keynote. Karya has been supported by organisations like Microsoft, Google, Gates Foundation, ACT Grants, Fast Forward, 100X Accelerator, DRK Foundation, Mulago, Echoing Green, The/Nudge Institute among others

Similar Jobs

Atlassian Logo Atlassian

Senior Forward Deployed Engineer, AI

Cloud • Information Technology • Productivity • Security • Software • App development • Automation
In-Office or Remote
560071, Domlur, Karnataka, IND
11000 Employees

Cloudflare Logo Cloudflare

Engineering Manager

Cloud • Information Technology • Security • Software • Cybersecurity
Remote or Hybrid
India
4400 Employees

Cloudflare Logo Cloudflare

Account Executive

Cloud • Information Technology • Security • Software • Cybersecurity
Hybrid
Bengaluru, Bengaluru Urban, Karnataka, IND
4400 Employees

ServiceNow Logo ServiceNow

Technical Support

Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Hybrid
Bangalore, Bengaluru, Karnataka, IND
29000 Employees

Similar Companies Hiring

Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
Vega Thumbnail
Artificial Intelligence • Automotive • Insurance • Transportation
US
65 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account