Real-world data is the competitive edge in AI.
HumanSignal is a human data partner for companies building AI models and products. Our customers ship better AI, faster, because we partner with their researchers from real-world data creation to annotation to delivery.
We design and create datasets from scratch, recruit and manage the domain experts who evaluate model output, and run everything through our own platform, Label Studio, the open-source standard for data labeling and evaluation, used by over 1 million practitioners worldwide.
We specialize in the operationally complex: real-world data collection, multimodal pipelines, and multi-step workflows. Advanced ML and AI teams use our enterprise platform to run their own data factories, and our services team to extend their reach where in-house capacity runs out.
If you want to do work that materially shapes how the next generation of AI products gets built, we'd love to talk.
Location: San Francisco or Austin, TX
Reports to: Head of Data Services
Compensation: $140,000 - $180,000
Every Data Services engagement succeeds or fails on two things: whether the team doing the work understands the spec, and whether we can prove the output meets it. This role owns both.
As Head of Quality & Project Enablement, you'll take each customer spec and turn it into the materials that get annotators and experts productive. You'll then design the quality pipeline and sampling methodology that verifies the work before it ships. When quality slips, you'll trace it back to its source, whether that's a gap in the spec, the training, or the review process, and fix it.
What you'll ownProject enablement
- Turn customer spec documents into project-ready enablement packages: annotator guidelines, decision trees, edge-case libraries with worked examples, and onboarding walkthroughs.
- Build qualification tests and gold-standard sets that confirm annotators understand the spec before they touch production work.
- Run pilot and calibration rounds at project kickoff to surface spec ambiguities early. Resolve those ambiguities with the customer and delivery lead before scaling up.
- Maintain versioned guidelines throughout the project. Roll out updates as new edge cases appear, and confirm the workforce has absorbed the changes.
Quality pipeline design
- Design the quality plan for every project. Choose the review structure (gold tasks, overlap/consensus, multi-stage review, expert adjudication) based on the task type, risk, and budget.
- Set the sampling methodology. Size samples to hit target confidence levels, stratify by class and difficulty, and adjust review rates for each annotator based on performance.
- Select the right metrics for each task, such as accuracy against gold, agreement measures like κ or α, or per-class error rates. Set acceptance thresholds that fit how subjective the task is.
- Configure these workflows in our labeling platform, and turn what works into reusable quality playbooks by task type.
Delivery quality control
- Own final quality sign-off. No delivery ships without meeting its agreed acceptance criteria.
- Monitor quality throughout each project. Catch drift early, run root-cause analysis on defects, and drive corrective action for individuals and for guidelines.
- Produce a clear quality report for every delivery that shows the methodology, the results, and any known limitations.
- 6+ years in quality assurance or quality operations for data labeling, human data, or ML training/evaluation data, including experience leading a quality function or team
- A proven ability to translate complex or ambiguous specs into guidelines and training that produce consistent results
- Strong applied statistics for QA: sampling design, confidence intervals, and agreement metrics, plus the ability to explain your choices to a customer
- Hands-on experience configuring review and QA workflows in an annotation platform
- Proficiency in Claude Code for quality analysis
- Crisp written communication. Your guidelines and quality reports are the product.
Nice to have: Experience with LLM, RLHF, or preference-data projects; experience with expert or domain-specialist workforces; a background in instructional design; familiarity with Label Studio; experience with model-assisted QA.
What success looks like- 90 days: A standard enablement package and quality plan template is in use on every new project. You own sign-off on all active deliveries.
- 6 months: Annotators ramp to target accuracy faster, mid-project guideline churn drops, and rework and customer-reported defects are measurably down.
- 12 months: Quality reports and methodology are a selling point for Data Services, and the playbooks are in place so the function can scale beyond you.
Skills Required
- 6+ years of experience in quality assurance or quality operations for data labeling, human data, or ML training/evaluation data
- Experience leading a quality function or team
- Ability to translate complex or ambiguous specifications into guidelines and training that produce consistent results
- Strong applied statistics for QA, including sampling design, confidence intervals, and agreement metrics
- Ability to explain statistical and quality methodology choices to customers
- Hands-on experience configuring review and QA workflows in an annotation platform
- Proficiency in Claude Code for quality analysis
- Strong written communication skills for creating guidelines and quality reports
- Experience with LLM, RLHF, or preference-data projects
- Experience with expert or domain-specialist workforces
- Background in instructional design
- Familiarity with Label Studio
- Experience with model-assisted QA
What We Do
HumanSignal (formerly known as Heartex) enables data science teams to build AI models with their company DNA. With the emergence of generative AI, it’s more important than ever to build highly differentiated models by guiding foundation models with proprietary data and human feedback. Creators of Label Studio, the most popular open source data labeling platform, HumanSignal enables data scientists to develop high quality datasets and workflows for model training, fine tuning and continuous validation. Today, the Label Studio open source community has more than 250,000 users who have collectively annotated more than 100 million pieces of data. Label Studio Enterprise is available as a cloud service with enhanced security, automation, quality review workflows, and performance reporting, used by leading data science teams including Bombora, Geberit, Outreach, Wyze, and Zendesk.
