We’re an AI-focused startup working with cutting-edge models and safety-critical use cases. A big part of our work relies on high-quality data annotation to train, evaluate, and monitor AI systems across complex domains (safety, fraud, compliance, content quality, etc.).
We’re looking for an Operations Lead to own our annotation operations end-to-end: managing projects, coordinating internal and external teams, and ensuring we deliver high-quality labeled data on time, at scale.
Role OverviewIn this role, you will:
Be the point person for all data annotation projects.
Manage a team of annotators (external vendors).
Design, refine, and enforce workflows, guidelines, and quality processes.
Partner with product, research, and engineering to turn vague requirements into clear task specs and rubrics.
You’re the kind of person who loves structure, can keep many moving pieces aligned, and cares deeply about quality, throughput, and reliability.
What You’ll DoProject & Workflow Management
Own planning and execution for multiple annotation projects at once (scope, timelines, staffing, and priorities).
Turn high-level requirements into clear task definitions, instructions, and edge-case guidance.
Build and maintain project plans including milestones, SLAs, and communication cadences with stakeholders.
Team & Vendor Management
Manage a team of annotators / reviewers (internal and/or external).
Handle capacity planning, scheduling, and task assignment to hit deadlines.
Coordinate with annotation vendors or agencies, ensuring they understand requirements and meet quality + throughput expectations.
Provide feedback, coaching, and training to improve annotator performance.
Quality, Process & Tooling
Define and iterate on rubrics, guidelines, and golden sets for consistent labeling.
Design and manage QA workflows (spot checks, double label, adjudication, calibration sessions).
Track and improve key metrics: accuracy, agreement, throughput, cost per label, and SLA adherence.
Partner with the product/engineering team to improve annotation tooling, dashboards, and automation.
Stakeholder Communication
Serve as primary contact for internal teams needing labeled data (research, product, T&S, etc.).
Provide regular status updates: progress vs plan, blockers, quality metrics, and risks.
Gather feedback on label quality, edge cases, and evolving requirements; turn those into updated guidelines and processes.
Continuous Improvement
Identify and implement process improvements to increase speed, reduce errors, and lower costs.
Run experiments to optimize task design, instructions, and QA strategies.
Help codify best practices into playbooks and documentation as we scale.
Experience
3–7+ years in operations, project management, or program management, ideally in:
Data annotation / labeling
Trust & Safety operations
Customer support operations
Or another high-volume, process-driven environment
Experience managing small to mid-sized teams and/or external vendors.
Prior work in AI / ML, data labeling, or content moderation is a strong plus.
Skills
Strong project management skills: planning, prioritizing, and keeping multiple tracks on schedule.
Excellent written communication; you can write clear guidelines and edge-case docs.
Comfortable working with metrics and dashboards (e.g., spreadsheets, BI tools) to monitor performance.
Detail-oriented and process-minded; you naturally look for ways to standardize and streamline.
Familiarity with annotation tools (e.g., Labelbox, Scale, Doccano, custom tools) is a plus, but not required.
Mindset
Ownership mentality: you feel responsible for outcomes, not just tasks.
Calm under pressure; you can navigate ambiguity and shifting priorities.
Collaborative; you work well with annotators, engineers, and leadership alike.
Bias toward action: you don’t just spot problems—you propose and test fixes.
Experience setting up calibration tests, golden sets, and inter-annotator agreement.
Background in trust & safety, content policy, or compliance.
Exposure to SQL or basic data analysis for monitoring volumes and quality trends.
Experience in a startup or early-stage environment where processes are still being built.
Top Skills
What We Do
Making the internet a safer place using AI.
