About Us
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
This opening is on behalf of one of our clients, and we’ll work closely with you to make the process clear and straightforward.
Role Overview
We are looking for Chinese-speaking AI Quality Analysts to evaluate personalized AI conversations and assess response quality across dimensions such as relevance, grounding, helpfulness, and natural integration of personal context. This role combines analytical thinking, creativity, and attention to detail. You will create realistic conversational scenarios, compare AI-generated responses, and provide structured evaluations to help improve the quality of next-generation AI systems.
CONTRACT: Freelance (2 months)
COMMITMENT: 30–40h/week with minimum 4h PST overlap
LOCATION: 100% Remote — India, Pakistan, Bangladesh, Egypt, Ghana, Nigeria
PROCESS: Online Assessment (~60min)
NOTES: Chinese fluency and willingness to use a primary personal Google account for evaluation purposes are mandatory.
Responsibilities
- Create realistic multi-turn conversational scenarios in Chinese
- Evaluate AI-generated responses for relevance, accuracy, and usefulness
- Assess how effectively personal context is incorporated into responses
- Compare and rank multiple AI responses side-by-side
- Identify grounding issues, unsupported assumptions, and incorrect personalization
- Review conversation quality, tone, and naturalness
- Write clear evaluation rationales and structured feedback
- Verify that AI systems correctly utilize available context and data sources
- Follow evaluation guidelines and maintain high-quality standards
- Manage evaluation data according to project requirements
Requirements
- Professional proficiency in written and spoken Chinese
- Bachelor's degree or equivalent experience in a relevant field
- Strong analytical and critical thinking skills
- Experience in AI evaluation, data annotation, content moderation, research, or similar analytical work is preferred
- Ability to assess nuanced and subjective quality differences in AI responses
- Strong written communication skills
- Comfortable working with AI tools and conversational systems
- Willingness to use a primary personal Google account as part of the evaluation process
Skills Required
- Professional proficiency in written and spoken Chinese
- Bachelor's degree or equivalent experience
- Strong analytical and critical thinking skills
- Ability to assess nuanced and subjective quality differences in AI responses
- Strong written communication skills
- Comfortable working with AI tools and conversational systems
- Willingness to use a primary personal Google account for evaluation
- Experience in AI evaluation, data annotation, content moderation, or similar analytical work
What We Do
Gramian Consulting Group is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong foundation in software engineering and leadership, the firm helps organizations build high-performing teams by matching them with qualified professionals. They specialize in talent augmentation and recruiting, specifically focusing on connecting engineering and data/AI talent with organizations to unlock real business value.
