Platform Engineer, RL & Eval Data | 100% remote

Posted 3 Days Ago
Be an Early Applicant
Hiring Remotely in Fully, Valais, CHE
In-Office or Remote
Senior level
Artificial Intelligence • Edtech • Machine Learning • Professional Services
The Role
Build, maintain, and scale production platforms for RL environments, LLM evaluations, annotation workflows, and human-data tooling. Responsibilities include backend and API development, distributed job orchestration, sandbox infrastructure, rollout collection, reward computation, observability, security, and internal tools. The role requires close collaboration with data teams and clients, production incident response, and reliable delivery of new platform capabilities.
Summary Generated by Built In
About the team

The Human Data Division builds training and evaluation data for frontier AI labs: RL environments, coding and STEM evaluation tasks, and expert-annotated datasets. We are a small team that moves fast and holds a high quality bar.

The role

We have built our own evaluation, RL and annotation platforms, and they are already in use. We are looking for a platform engineer to join the platform team as a core member and play a central part in their next stage: keeping them running, making them faster and more reliable, and building the features the team and our clients need next.

You will work across all of these systems and get to know them in depth. You will work closely with the people who write our tasks, review our data and work with the labs, and help turn what they need into working software.

Responsibilities
  • Maintain and improve the platforms. Learn the existing codebases in depth and keep them healthy in production: fix bugs, pay down technical debt, improve test coverage and documentation.

  • Build new features. Design and ship new capabilities across the stack, from the database and backend to the APIs and internal tools.

  • Scale the systems. Grow the platforms from today's volume to much larger workloads, with predictable performance and cost.

  • Build RL environments and their infrastructure. Build environments that agents act in, and the infrastructure that runs them at scale: sandboxes, environment images, rollouts and reward computation.

  • Keep everything reliable and secure. Monitoring, alerting, incident response, access control, safe handling of untrusted code and protection of client data.

  • Support the data teams. Build the tooling that task authors, reviewers and project leads use every day, and help produce fast-turnaround samples when a project needs them.

Skills we are looking for:

Software engineering

  • Strong backend engineering in Python: clean, tested, maintainable code.

  • Designing and building APIs and the data models behind them.

  • Relational databases (Postgres or similar): schema design, migrations, query performance.

  • Enough frontend skill (TypeScript and React) to build and maintain internal tools.

  • Comfort joining a codebase someone else wrote, understanding it quickly and improving it without breaking it.

Large-scale system design

  • Designing distributed systems that run many long-running jobs at once.

  • Job orchestration, workflow engines and queues: retries, idempotency, failure recovery.

  • Containers and container orchestration (Docker, Kubernetes or similar), and at least one major cloud provider.

  • Observability: logging, metrics, tracing and alerting.

  • Security fundamentals: secrets management, isolation, access control.

RL environments and sandbox infrastructure

  • Hands-on experience building RL environments for LLM agents: tool interfaces, stateful environments that can be seeded, reset and snapshotted, and reward computed from the final state or the agent's answer.

  • Running agent code in isolated sandboxes at scale, with sandbox platforms such as Daytona, E2B or Modal, or with your own isolation layer (gVisor, Firecracker microVMs or similar).

  • Building and caching environment images so that thousands of sandboxes can start quickly and reproducibly.

  • High-throughput rollout collection: many parallel episodes, timeouts, resource limits, and full logs and trajectories for every episode.

  • Feeding rollouts and rewards into RL post-training pipelines.

Evaluation

  • Hands-on experience building or running evaluation harnesses or benchmarks for LLMs or LLM agents.

  • Writing verifiers, graders or reward functions, and knowing how they fail.

  • Understanding of how frontier models are evaluated: pass rates, pass@k, headroom, discrimination, reward hacking, and what makes a task hard for a frontier model.

  • Working with LLM provider APIs and tracking model cost and usage.

Data annotation and human data

  • Experience building annotation, labeling or human-data tooling.

  • Contributor and reviewer workflows, quality control and calibration.

  • Turning a manual, ad-hoc process into a scaled, measured one.

Ways of working

  • Daily, careful use of AI coding agents: you move fast with them and review their output with the same rigor as a colleague's.

  • Thrives in ambiguity and ships fast; works independently and takes initiative on a small team.

  • Clear written communication with both engineers and non-engineers.

Qualifications
  • 5+ years in software, data or ML engineering, with recent work in AI/ML data, model evaluation, annotation or developer tooling.

  • A track record of running and maintaining production systems, not only building them.

  • Fluent English.

Nice to have
  • Contributions to open-source evaluation or RL frameworks.

  • Experience building tools for a large distributed contributor workforce.

Skills Required

  • 5+ years of experience in software, data, or ML engineering
  • Recent experience in AI/ML data, model evaluation, annotation, or developer tooling
  • Strong backend engineering experience with Python
  • Experience designing and building APIs and data models
  • Experience with relational databases such as Postgres, including schema design, migrations, and query performance
  • Working knowledge of TypeScript and React for internal tools
  • Experience designing distributed systems with long-running concurrent jobs
  • Experience with job orchestration, workflow engines, queues, retries, idempotency, and failure recovery
  • Experience with containers and orchestration such as Docker and Kubernetes
  • Experience with at least one major cloud provider
  • Experience with observability, including logging, metrics, tracing, and alerting
  • Knowledge of security fundamentals, secrets management, isolation, and access control
  • Hands-on experience building RL environments for LLM agents
  • Experience running agent code in isolated sandboxes at scale
  • Experience building and caching reproducible environment images
  • Experience with high-throughput rollout collection and RL post-training pipelines
  • Experience building or running LLM evaluation harnesses or benchmarks
  • Experience writing verifiers, graders, or reward functions
  • Experience with LLM provider APIs and model cost and usage tracking
  • Experience building annotation, labeling, or human-data tooling
  • Experience with contributor and reviewer workflows, quality control, and calibration
  • Track record of running and maintaining production systems
  • Fluent English
  • Contributions to open-source evaluation or RL frameworks
  • Experience building tools for a large distributed contributor workforce
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
Year Founded: 2022

What We Do

Anyone AI is an edtech startup dedicated to bridging the AI talent gap by investing in software developers from Latin America. The company provides intensive, hands-on training programs in Machine Learning and Artificial Intelligence, led by industry experts. By combining technical skill development with employability support, Anyone AI prepares professionals for global career opportunities, helping them transition into high-impact roles within the rapidly evolving AI and technology sectors.

Similar Jobs

Rain Logo Rain

Analytics Engineer

Blockchain • Fintech • Payments • Financial Services • Cryptocurrency • Web3 • Infrastructure as a Service (IaaS)
In-Office or Remote
30 Locations
100 Employees
128K-167K Annually

Rain Logo Rain

Data Engineer

Blockchain • Fintech • Payments • Financial Services • Cryptocurrency • Web3 • Infrastructure as a Service (IaaS)
Remote or Hybrid
30 Locations
100 Employees
128K-167K Annually

Rain Logo Rain

Platform Engineer

Blockchain • Fintech • Payments • Financial Services • Cryptocurrency • Web3 • Infrastructure as a Service (IaaS)
In-Office or Remote
30 Locations
100 Employees
136K-170K Annually

Pfizer Logo Pfizer

Manager Archiving and Compliance

Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Remote or Hybrid
28 Locations
121990 Employees

Similar Companies Hiring

Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
Vega Thumbnail
Artificial Intelligence • Automotive • Insurance • Transportation
US
65 Employees
VCA Animal Hospitals Thumbnail
Pet • Professional Services
Los Angeles, CA
35000 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account