Real-world data is the competitive edge in AI.
HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their researchers from real-world data creation to annotation to delivery.
We design and create datasets from scratch, recruit and manage the domain experts who evaluate model output, and run everything through our own platform, Label Studio, the open-source standard for data labeling and evaluation, used by over 1 million practitioners worldwide.
We specialize in the operationally complex: real-world data collection, multimodal pipelines, and multi-step workflows. Advanced ML and AI teams use our enterprise platform to run their own data factories, and our services team to extend their reach where in-house capacity runs out.
If you want to do work that materially shapes how the next generation of AI products gets built, we'd love to talk.
Location: Remote
Type: Part-Time Contract (20 hours per week for 4 weeks | 80 hour minimum commitment)
Duration: 4 weeks with potential for ongoing work
Compensation: Up to $25/hour for US-based candidates
You'll examine designs to understand color schemes, font choices, and other design elements. You will then review a slide within that design that seems out of place. Leveraging your graphic design expertise, along with what you learn in our training, you will edit the incorrect slide to match the rest of the overall design. This is all in an effort to train AI models that assess design effectiveness—contributing to cutting-edge technology that advances how AI understands and evaluates visual content quality.
What You'll Do- Review designs in Canva Editor to examine various design dimensions:
- Typography quality (readability, hierarchy, font pairing appropriateness)
- Visual hierarchy effectiveness (clarity, balance, focal point strength)
- Design element execution (icon quality, illustration polish, image selection)
- Color palette effectiveness (harmony, contrast, accessibility)
- Layout and composition quality (grid usage, spacing, alignment)
- Overall design effectiveness and professional polish
- Identify flaws in one slide within the overall design. These flaws may be minor, like simple font size issues. They may be major, requiring a full reconstruction of the slide
- Using the other slides in the design as reference, edit the flawed slide to build more cohesion across the entire design
- Process high volumes of design samples with sustained attention to detail
- Meet quality benchmarks for rating consistency and inter-rater reliability
- Formal training in graphic design, visual arts, or related field (degree, certificate, or bootcamp completion) OR 2-3 years of professional graphic design experience
- Fully proficient at intermediate to expert level in Canva Editor
- Strong proficiency in design principles (composition, color theory, typography, visual hierarchy)
- Ability to objectively critique and evaluate design work against industry standards
- Understanding of what constitutes effective vs. ineffective design choices
- Experience reviewing or providing design feedback
- Exceptional attention to detail and ability to maintain focus during repetitive tasks
- Comfort with highly repetitive work—ability to maintain quality and engagement while rating similar content for extended periods
- Strong visual discrimination and consistent aesthetic judgment
- Reliability and consistency in your evaluation standards
- Self-motivated with ability to work independently on repetitive tasks
- Excellent focus and concentration for sustained periods
- Strong communication skills for flagging edge cases or unclear examples
- Comfortable with feedback and re-calibration
- Minimum 20 hours per week availability for at least 4 weeks, with potential to extend
- Portfolio demonstrating diverse design work and critical eye
- Experience as a design reviewer, art director, or in quality assurance roles
- Understanding of accessibility standards (WCAG, inclusive design principles)
- Knowledge of design across multiple contexts (branding, editorial, digital, UX/UI)
- Experience teaching or mentoring design concepts
- Experience with Canva Editor
- Prior data annotation, content moderation, or quality evaluation experience
- Exposure to machine learning or computer vision concepts
- Familiarity with design systems and component libraries
- High tolerance for repetitive evaluation tasks—finds rhythm and satisfaction in systematic work
- Ability to maintain consistent standards across thousands of ratings
- Strong pattern recognition—quickly identifying quality patterns across examples
- Disciplined, methodical approach with minimal variation in judgment
- Problem-solving mindset for handling ambiguous or borderline cases
- Interest in AI/ML applications in design and creative fields
- Previous remote work experience with self-directed tasks
You thrive on consistency and repetition - You find satisfaction in systematic work and can maintain quality standards while evaluating similar content repeatedly without losing focus or accuracy.
You have a calibrated design eye - You can quickly assess design quality against objective criteria and apply the same standards consistently across hundreds or thousands of examples.
You're objective and principled - You separate personal taste from professional assessment, basing ratings on established design principles rather than subjective preference.
You maintain sustained focus - You can work through high volumes of repetitive tasks while staying sharp and attentive to subtle quality differences.
You're a clear communicator - You flag confusing examples or rating criteria early, ask clarifying questions, and document edge cases clearly.
You're curious about technology - You're genuinely interested in how AI learns to evaluate design quality and see value in contributing to machine learning training data.
Skills Required
- Formal training in graphic design or related field OR 2-3 years of professional graphic design experience
- Fully proficient at intermediate to expert level in Canva Editor
- Strong proficiency in design principles
- Experience reviewing or providing design feedback
- Exceptional attention to detail
What We Do
HumanSignal (formerly known as Heartex) enables data science teams to build AI models with their company DNA. With the emergence of generative AI, it’s more important than ever to build highly differentiated models by guiding foundation models with proprietary data and human feedback. Creators of Label Studio, the most popular open source data labeling platform, HumanSignal enables data scientists to develop high quality datasets and workflows for model training, fine tuning and continuous validation. Today, the Label Studio open source community has more than 250,000 users who have collectively annotated more than 100 million pieces of data. Label Studio Enterprise is available as a cloud service with enhanced security, automation, quality review workflows, and performance reporting, used by leading data science teams including Bombora, Geberit, Outreach, Wyze, and Zendesk.
