The Role
Build and operate AI-driven workflows that improve business and product review processes. Responsibilities include mapping workflows, writing prompts, creating pipelines and test sets, measuring quality, calibrating LLM judges, defining AI launch criteria, developing labeling guidelines, managing vendor labeling, monitoring model drift, and documenting reusable methods. The role owns workstreams from setup through production quality checks and partners with product, legal, policy, engineering, and subject matter experts.
Summary Generated by Built In
About the Role
We are seeking an AI Operations Specialist to help business and product teams use AI to take on review and quality work. You will map how work gets done today, build AI-driven workflows, and test them until the results can be trusted. You will also help product teams decide whether an AI feature is ready to launch. This is a hands-on role: you help build the workflow, ship it, and stay on to make sure it keeps working. The scope is expected to grow as AI capabilities evolve.
Key Responsibilities- Help prioritize projects by business impact and make the case for the ones worth pursuing.
- Partner with process owners to map current workflows, then define where AI can do the work and where people step in on hard cases.
- Build AI workflows: write prompts, set up pipelines, maintain agreed quality levels, and monitor for drift after launch.
- Build test sets, measure accuracy, and calibrate LLM judges against human reviewers.
- Help product teams define launch-readiness criteria for AI features and provide evidence for go/no-go decisions.
- Turn unclear policy into criteria reviewers can apply consistently, working with product, legal, and policy partners.
- Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
- Run labeling at scale with vendor agents, and validate their work using agreement metrics and spot checks.
- Make solutions reusable and document learnings so future teams reach production faster.
- Keep methods current as models, tools, and vendor agents change, and retire what no longer works.
- Own assigned workstreams end to end, from setup through quality checks.
Required
- 5-7 years in operations, program management, analytics, consulting, or applied AI, with accountability for results.
- Daily AI user who has built something with AI that delivered measurable impact, such as a workflow, an automation, or an analysis.
- Comfortable with metrics such as accuracy, precision, recall, and sampling.
- Experience taking a messy process and making it measurable.
- Quality experience in some form, such as review guidelines, QA, or audits.
- Clear writer, able to work with business leads, product managers, and engineers.
- Self-directed and comfortable working with limited direction.
- A technical background is not required. Prior experience designing evaluations or calibrating LLM judges is not required; training is provided.
Preferred
- Hands-on SQL, Python, or API experience.
- Trust and safety, content moderation, annotation operations, or other human-in-the-loop work at scale.
- Experience fine-tuning models or calibrating LLM judges.
- Background in a two-sided marketplace, travel, or another operations-heavy business.
- Experience setting up intake and prioritization for a new team.
- Start date: November 23, 2026
- Duration: 12 months
- Location: Remote (within the U.S.)
- Work schedule: 40 hours/week (8 hours/day, 5 days/week)
- Pay rate range: $65-$75/hour + benefits
- Work Authorization: Must be authorized to work in the U.S. without sponsorship.
- Engagement: W-2 Contract
Skills Required
- 5-7 years of experience in operations, program management, analytics, consulting, or applied AI, with accountability for results
- Daily AI user who has built an AI workflow, automation, or analysis with measurable impact
- Comfort with accuracy, precision, recall, and sampling metrics
- Experience making a messy process measurable
- Quality experience, such as review guidelines, QA, or audits
- Clear writing and ability to work with business leads, product managers, and engineers
- Self-directed and comfortable working with limited direction
- Hands-on SQL, Python, or API experience
- Experience in trust and safety, content moderation, annotation operations, or other human-in-the-loop work at scale
- Experience fine-tuning models or calibrating LLM judges
- Background in a two-sided marketplace, travel, or another operations-heavy business
- Experience setting up intake and prioritization for a new team
Similar Jobs
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Maleda Tech is a global contingent workforce solutions provider that offers Employer of Record (EOR) and Managed Service Provider (MSP) services. The company specializes in connecting highly skilled IT professionals from the US and Africa with clients, operating as a minority-owned business enterprise. They focus on building and fixing contingent workforce programs for companies across three continents, leveraging a diverse, global talent pool to meet technical staffing needs.
