Lead Data Engineer, Data Platform

Posted Yesterday
Be an Early Applicant
San Francisco, CA, USA
In-Office
Senior level
Artificial Intelligence • Software
The Role
Own and build CrewAI's end-to-end data foundation: consolidate telemetry and application data, design pipelines and warehouses, define trusted metrics and semantic models, improve instrumentation and data quality, enable self-serve analytics, and support product and GTM decisions with reliable dashboards and analyses.
Summary Generated by Built In
About CrewAI

CrewAI is the leading framework and enterprise platform for building and orchestrating multi-agent AI systems, powering 300M+ agent executions per month across thousands of companies. As the product, platform, and customer base scale, data is becoming one of the most important systems in the company: how we understand usage, reliability, activation, customer health, cost, governance, and where to invest next.

Today, we have meaningful data already, but it is spread across product telemetry, trace data, application databases, analytics tables, Cube models, Metabase dashboards, and team-specific queries. We need someone to turn that into a coherent, trusted, useful data foundation.

The Role

You’ll be CrewAI’s first dedicated data engineering hire. Your job is to own the data foundation end to end: rationalize what exists, improve the infrastructure, define trusted metrics, close instrumentation gaps, and make data accessible enough that product, growth, engineering, customer success, and leadership can actually use it.

This is a foundational role with real range. The center of gravity is data infrastructure and analytics engineering: pipelines, warehouse/lake design, semantic modeling, metric definitions, data quality, and self-serve access. You’ll also be the person who turns messy questions into clear analysis, reliable dashboards, and better product decisions.

This is not a maintenance role. It is a “make data legible and useful for the company” role.

What You’ll Do
  • Own and evolve CrewAI’s data platform across ingestion, transformation, storage, semantic modeling, BI, and operational data quality.
  • Rationalize the existing data estate: product events, execution telemetry, OpenTelemetry-derived traces, application tables, Cube models, Redshift/data-lake tables, Metabase dashboards, and team-specific reporting.
  • Establish trusted source-of-truth metrics for the business and product, including executions, active builders/users, activation, deployment health, token and cost usage, customer health, governance adoption, retention, and feature usage.
  • Build and maintain the models, pipelines, and metric layers that make those numbers consistent across teams.
  • Partner with product and engineering to improve instrumentation, event taxonomy, data contracts, and telemetry coverage for new features.
  • Make data self-serve through clear dashboards, documented datasets, reusable metric definitions, and sensible access patterns.
  • Improve reliability and trust in the stack through data quality checks, freshness monitoring, lineage, alerting, backfills, and incident/debug workflows.
  • Partner with Discovery, product, and go-to-market teams on analysis behind recommendations, customer signals, usage patterns, and roadmap decisions.
  • Keep the stack secure and cost-aware, including access control, PII handling, retention, and warehouse/query efficiency.
  • Help define how CrewAI uses data internally as the company scales.

RequirementsWhat We’re Looking For
  • Strong data engineering or analytics engineering experience, especially building data foundations in fast-moving product companies.
  • Excellent SQL and data modeling skills, with experience designing reliable datasets, fact/dimension models, and metric definitions.
  • Experience operating a warehouse or analytics store such as Redshift, Snowflake, BigQuery, Postgres, or similar.
  • Familiarity with transformation and modeling tools such as dbt, Cube, semantic layers, or equivalent systems.
  • Experience with event pipelines, product telemetry, application data, and BI tools such as Metabase, Looker, Mode, or similar.
  • Strong Python for data work, automation, validation, and operational workflows.
  • Product sense: you can turn ambiguous questions into useful metrics, and you care whether the numbers are understood correctly.
  • Pragmatism: you are comfortable inheriting messy systems, improving them incrementally, and choosing boring reliable solutions when they are right.
  • Strong communication and documentation habits. You make data easier for other people to use.
  • Comfort being the first dedicated owner in an early-stage, high-growth environment.
Bonus
  • Experience with LLM, agent, observability, trace, usage, or cost analytics.
  • Experience with OpenTelemetry, high-volume event data, or operational telemetry.
  • Experience with experimentation, causal analysis, activation/retention modeling, or customer health scoring.
  • Experience defining event taxonomies and instrumentation standards for SaaS products.
  • Familiarity with Rails/Postgres application data, background jobs, and product analytics in B2B SaaS.
  • Lightweight ML or recommendation experience, especially where it supports product or customer workflows.

Skills Required

  • Strong data engineering or analytics engineering experience in fast-moving product companies
  • Excellent SQL and data modeling skills, including fact/dimension models and metric definitions
  • Experience operating a warehouse or analytics store such as Redshift, Snowflake, BigQuery, or Postgres
  • Familiarity with transformation and modeling tools such as dbt, Cube, or semantic layers
  • Experience with event pipelines, product telemetry, application data, and BI tools (Metabase, Looker, Mode)
  • Strong Python for data work, automation, validation, and operational workflows
  • Product sense to translate ambiguous questions into useful metrics and analyses
  • Pragmatism: comfortable improving messy systems incrementally and choosing reliable solutions
  • Strong communication and documentation habits to make data accessible for other teams
  • Comfort being the first dedicated data owner in an early-stage, high-growth environment
  • Experience with LLM, agent, observability, trace, usage, or cost analytics
  • Experience with OpenTelemetry, high-volume event data, or operational telemetry
  • Experience with experimentation, causal analysis, activation/retention modeling, or customer health scoring
  • Experience defining event taxonomies and instrumentation standards for SaaS products
  • Familiarity with Rails/Postgres application data, background jobs, and B2B SaaS product analytics
  • Lightweight ML or recommendation experience supporting product or customer workflows
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Middletown, DE
7 Employees

What We Do

Most AI agent frameworks are hard to use. We provide power with simplicity. Automate your most important workflows quickly.

Similar Jobs

The Walt Disney Company Logo The Walt Disney Company

Platform Engineer

Digital Media • Gaming • News + Entertainment • Sports
In-Office
3 Locations
219548 Employees
148K-219K Annually

Teamworks Logo Teamworks

Data Engineer

Artificial Intelligence • Cloud • Information Technology • Machine Learning • Software • Sports • Analytics
In-Office or Remote
2 Locations
600 Employees
Remote or Hybrid
United States
300 Employees
100K-140K Annually

Windfall Logo Windfall

Senior Data Engineer

Artificial Intelligence • Big Data • Machine Learning • Software
Hybrid
2 Locations
81 Employees
170K-220K Annually

Similar Companies Hiring

Hanover Park Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
42 Employees
Kepler  Thumbnail
Fintech • Software
New York, New York
6 Employees
Onshore Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
60 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account