The Role
Build production data foundations for AI systems, including Databricks and Fabric lakehouses, ingestion pipelines, semantic layers, RAG and GraphRAG retrieval, evaluation datasets, and governed agent infrastructure. Partner directly with client technical and business leaders to design secure, reliable data and retrieval architectures. Own data quality, access controls, observability, model evaluation, and reusable engineering components across client engagements.
Summary Generated by Built In
About Cadre AI
The Role
What You’ll DoLakehouse and Pipeline Engineering
Compensation
Cadre AI is an AI strategy and integration firm that builds production AI systems for B2B companies in private equity, wholesale lending, real estate, and SaaS. We do not build decks about what AI could do. We ship systems that move revenue, compress costs, and automate the work that used to take entire teams.
You will build the data foundations that make our clients’ AI systems trustworthy: the lakehouses that unify their ERPs and SaaS tools, the retrieval systems that ground agents in their documents and records, the semantic layers that give every metric one definition, and the evaluation datasets that prove an answer is right. Every agent, dashboard, and workflow Cadre ships sits on top of the data layer you design.
Your week moves between architecture and execution: mapping a manufacturer’s ERP tables with their finance team one day, building Databricks medallion layers or tuning a hybrid retrieval pipeline the next. You thrive here if you keep up with retrieval research, know why a random split leaks in a time series, and can explain a GraphRAG trade-off to a CFO in plain language.
You will work in client pods with AI engineers, solution architects, and AI strategists, and shape the reusable data and retrieval blocks every engagement starts from.
- Design and build medallion (bronze, silver, gold) lakehouses on Databricks (Delta Lake, Unity Catalog, Lakeflow Spark Declarative Pipelines) and Microsoft Fabric (OneLake, Data Factory), and on Snowflake where a client already runs it.
- Own ingestion from ERPs, CRMs, and SaaS platforms using platform-native connectors (Lakeflow Connect, Fabric Data Factory), Fivetran, or dlt, including change data capture and schema drift handling.
- Model and transform data with dbt, orchestrate it with Apache Airflow, Dagster, or Lakeflow Jobs, and ship it through CI/CD with GitHub Actions and Declarative Automation Bundles (formerly Databricks Asset Bundles).
- Build data quality checks and data contracts into every pipeline so downstream agents never consume silent errors.
- Run agentic data engineering safely in client Databricks repositories: set up coding agents (Claude Code, Codex) with shared skills, policies, and guardrails, using Omnigent where it is available, and accelerate legacy-to-Databricks migrations with Genie Migrate.
- Build production RAG pipelines end to end: document parsing and OCR, chunking, embeddings, hybrid retrieval (BM25 plus dense vectors), and cross-encoder reranking.
- Design GraphRAG systems that extract entities and relationships into a knowledge graph, using frameworks such as Microsoft GraphRAG or LightRAG on graph stores such as Neo4j, for questions that span documents, systems, and multiple entity hops.
- Choose the right retrieval architecture for each use case, from hybrid search with reranking to graph-based and agentic retrieval, and justify the cost and maintenance burden of each to the client.
- Operate vector, full-text, and hybrid search infrastructure (Databricks AI Search, formerly Vector Search; Azure AI Search; OpenSearch; pgvector), including incremental indexing, permission-aware retrieval, and citations back to the source.
- Build and deploy retrieval-backed agents on Databricks Agent Bricks, using its code-first tooling (Agent Bricks CLI, Agent Runtime on Databricks Apps) and governing model access through Unity AI Gateway.
- Translate how a client defines a customer, an order, or an invoice into governed metric and entity definitions using dbt Semantic Layer (MetricFlow), Power BI semantic models, Cube, Databricks metric views, or Snowflake semantic views.
- Lead data readiness assessments: inventory source systems, map fields and business logic, and document gaps and remediation steps before engineering starts.
- Expose governed data to Claude and other agents through MCP, so every agent, dashboard, and text-to-SQL query answers from the same definitions.
- Use AI to draft documentation and definitions, then own their correctness together with the business owner.
- Build golden datasets, output and label pairs, and backtesting splits that measure whether an agent, retriever, or model is right.
- Instrument retrieval and agent pipelines with dedicated tracing and evaluation tools (Langfuse, Arize Phoenix, Braintrust), or with MLflow 3 GenAI evaluation and LLM judges on Databricks, and track recall, precision, and groundedness.
- Track experiments and version models and evaluation runs in the MLflow registry, so every change to a retriever or agent is traceable.
- Design leakage-free, time-aware splits for forecasting, scoring, and risk ranking.
- Monitor drift, freshness, and data gaps that degrade agents in production, and fix them at the data layer.
- Implement role-based access, row-level security, and masking in Unity Catalog, Microsoft Purview, or Snowflake roles, and carry those permissions into retrieval and agent context.
- Ensure every data path meets SOC 2 expectations and the client’s own AI governance policies, including the handling of PII, PHI, and payroll data.
- Work directly with client IT, finance, and operations leaders to scope work, secure access, and explain build-versus-buy trade-offs across platforms.
- Record architecture decisions as ADRs in the client repository and contribute reusable pipeline, retrieval, and semantic-layer blocks to Cadre’s internal library.
- 5+ years of data engineering experience, including at least one production lakehouse you designed and operated on Databricks or Microsoft Fabric with a medallion architecture; experience on both is a plus, since our clients run both.
- A Master’s degree or PhD in Computer Science, Mathematics, Engineering, or a related quantitative field, with the theoretical grounding to read current research and apply it to client problems.
- Hands-on experience shipping RAG pipelines to production (hybrid retrieval on Databricks AI Search or an equivalent engine, reranking, chunking strategy, and permission-aware indexing over messy enterprise documents), with command of the current state of the art: you follow the literature and can explain when hybrid, graph-based, or agentic retrieval fits a problem.
- Experience designing and running GraphRAG or knowledge-graph retrieval (Microsoft GraphRAG, LightRAG, or equivalent, on graph stores such as Neo4j), and the judgment to know when a graph earns its maintenance cost and when hybrid search is enough.
- Production experience orchestrating data pipelines with Apache Airflow or Dagster, including dependencies, backfills, retries, and alerting; you understand the trade-off between task-based and asset-based orchestration and work comfortably in platform-native schedulers such as Lakeflow Jobs and Fabric Data Factory.
- Expert SQL and strong Python and PySpark, with production experience in dbt or platform-native pipelines and in ingestion tools (Fivetran, Lakeflow Connect, Azure or Fabric Data Factory, or dlt); you use coding agents such as Claude Code daily and review everything they generate.
- Experience authoring a semantic or metrics layer (dbt Semantic Layer, Power BI semantic models, Cube, LookML) and turning business definitions into governed metrics that people and agents both rely on.
- Working ML fluency: you understand data leakage, time-based validation, class imbalance, and precision versus recall, and you have built datasets that trained or evaluated a model or retriever, evaluated with Langfuse, Arize Phoenix, Braintrust, or MLflow and versioned in the MLflow registry.
- Hands-on experience building agents on Databricks (Agent Bricks or Mosaic AI Agent Framework) and running coding agents such as Claude Code or Codex under governed harnesses with skills, policies, and guardrails; familiarity with Databricks’ newer agentic tooling (Omnigent, Genie Migrate, Agent Bricks CLI) and with how agents consume data through RAG, text-to-SQL, and MCP.
- Governance depth across Unity Catalog, Purview, or Snowflake RBAC; you treat access control as a design input for pipelines and retrieval alike.
- Client-facing consulting or forward-deployed experience and professional English fluency, because you will work daily with US executives and turn technical trade-offs into business decisions.
- Strong plus: Snowflake Cortex, Palantir Foundry ontologies, RDF/OWL knowledge graphs, DSPy prompt optimization, or LoRA fine-tuning of open-weight models.
- AI-native culture. We do not just build AI for clients. We use it to run our own operations. You will work with people who are as obsessed with the tools as you are.
- Access to the frontier. Through partnerships with Anthropic and OpenAI, you will be among the first to experiment with new models and capabilities.
- Upside. We are bootstrapped, profitable, and growing fast. Growth means new teams and new initiatives, and room to move into them.
- No bureaucracy. Small, scrappy team. Clear accountability. The best idea wins, regardless of who says it.
The base pay range for this role is Col$6,000 – Col$7,500 per year.
Skills Required
- 5+ years of data engineering experience
- Designed and operated at least one production Databricks or Microsoft Fabric lakehouse using medallion architecture
- Master's degree or PhD in Computer Science, Mathematics, Engineering, or a related quantitative field
- Production experience shipping RAG pipelines, including hybrid retrieval, reranking, chunking, and permission-aware indexing
- Experience designing and running GraphRAG or knowledge-graph retrieval systems
- Production pipeline orchestration experience with Apache Airflow or Dagster
- Expert SQL and strong Python and PySpark skills
- Production experience with dbt or platform-native data pipelines and ingestion tools such as Fivetran, Lakeflow Connect, Azure Data Factory, Fabric Data Factory, or dlt
- Experience authoring a semantic or metrics layer using dbt Semantic Layer, Power BI semantic models, Cube, or LookML
- Working ML fluency, including data leakage, time-based validation, class imbalance, and precision versus recall
- Experience building evaluation datasets and evaluating systems with Langfuse, Arize Phoenix, Braintrust, or MLflow
- Hands-on experience building agents on Databricks using Agent Bricks or Mosaic AI Agent Framework
- Experience running governed coding agents such as Claude Code or Codex
- Governance experience with Unity Catalog, Microsoft Purview, or Snowflake role-based access control
- Client-facing consulting or forward-deployed experience
- Professional English fluency
- Experience with both Databricks and Microsoft Fabric
- Experience with Snowflake Cortex, Palantir Foundry ontologies, RDF/OWL knowledge graphs, DSPy, or LoRA fine-tuning
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Cadre is an AI strategy and integration firm focused on helping organizations unlock the power of AI to drive measurable business results that grow revenue, increase EBITDA, and unlock scalability. We implement best-in-class AI tools and technologies to AI-Enable your business and create a lasting competitive advantage. Our combination of entrepreneurial experience, strong operational understanding and technical AI expertise enables us to help our clients integrate cutting-edge AI technologies to drive efficiency, productivity, and growth. As a team we take an “ownership mindset” to our clients business, asking ourselves “what moves would I make if this was my business” and are committed to transforming your company into a “future-proof” organizations by leveraging the power of AI to achieve operational excellence and enhanced profitability and putting you ahead of your competitors and industry.
.png){kind=logo}
