The Role
Lead design and ownership of ETL pipelines and OLAP-oriented data architecture on an S3-backed data lake. Define partitioning, file formats, DAG-based Airflow workflows, drive AWS data stack (Athena, EKS), enforce code quality, and lead senior engineers to deliver scalable marketing analytics pipelines.
Summary Generated by Built In
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
We are looking for a Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural decisions around partitioning strategy, file formats, schema design, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. You will design and govern ETL pipelines, define DAG-based orchestration strategies using Airflow, drive the AWS data stack including Athena and EKS, and lead a team of senior developers while enforcing code quality standards.
WHAT YOU WILL DO
- Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale.
- Make architectural decisions around partitioning, file formats, schema and data-type strategy, and near-real-time processing for large-volume, OLAP-oriented data systems built on an object-storage data lake.
- Own the design of scheduled batch workflows (DAGs) on the client's Airflow setup, defining pipeline structure, dependencies, and triggering strategies, and driving architectural discussions around them.
- Drive the use of the client's AWS data stack, including an S3-backed data lake, Athena, and EKS/Kubernetes.
- Partner directly with the client's DevOps team to clarify functional and non-functional requirements.
- Review pull requests and enforce code quality standards.
- Guide senior developers and ensure alignment with the client's engineering practices.
MUST HAVES
- 7+ years of engineering experience, with a proven track record designing and implementing ETL pipelines and making architectural decisions for large-volume data systems.
- Hands-on experience with OLAP-style analytical data architecture, with experience in Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar technologies.
- Hands-on experience designing data lakes backed by object storage such as S3 or equivalent, including partitioning strategies, file formats such as Parquet/ORC, and cost/performance tradeoffs.
- Deep familiarity with DAG-style workflow definition and triggering, with substantial experience in Airflow or comparable orchestrators such as Dagster, Prefect, Luigi, or Step Functions.
- Practical experience across the AWS data ecosystem, including S3-backed data lakes, serverless query engines such as Athena or equivalent, and EKS/Kubernetes.
- Strong backend proficiency in Python, with experience using FastAPI or Flask.
- Comfortable working with REST and GraphQL.
- Experience with Docker and PostgreSQL for transactional and application layers.
- Highly comfortable working in Mac/Linux terminal-centric environments.
- Practical, hands-on experience with AI-assisted development tools such as Claude Code, combined with the critical judgment to challenge AI-generated output when it compromises long-term maintainability.
- Leadership experience setting standards for responsible use of AI tooling, including identifying risky AI-driven shortcuts during code review.
- Strong communication and technical judgment, with the ability to defend technical decisions, challenge quick fixes with sound reasoning, and balance long-term maintainability with pragmatic delivery.
- Upper-Intermediate English level.
NICE TO HAVES
- Direct production experience with Athena.
- Working knowledge of TypeScript and React, sufficient to guide integrations and review frontend-adjacent pull requests.
- Production experience building AI features using AWS Bedrock, LangChain, Pydantic AI, or similar technologies.
- Experience with monorepo tooling such as Nx or modern package managers such as Poetry, UV, or Yarn.
- Experience with Redis and caching layers or SageMaker.
- Experience with marketing data structures, campaign management APIs, or digital advertising metrics.
PERKS AND BENEFITS
- Growth without limits: build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
- Competitive compensation: get recognition that reflects your skills and impact, with regular performance and compensation reviews
- Flexibility: work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
- Meaningful, modern projects: build impactful products using modern technologies alongside global teams and leading brands
- Collaborative culture: join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
- Well-being & support: access local well-being programs and people-focused support tailored to your location
Skills Required
- 7+ years of engineering experience
- Designing and implementing ETL pipelines at scale
- Experience with OLAP-style analytical data architecture (Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse or similar)
- Designing data lakes backed by object storage (S3), partitioning strategies, file formats such as Parquet/ORC
- DAG-style workflow definition and triggering with Airflow or comparable orchestrators (Dagster, Prefect, Luigi, Step Functions)
- Practical experience across AWS data ecosystem, including S3-backed data lakes, Athena, and EKS/Kubernetes
- Strong backend proficiency in Python with experience using FastAPI or Flask
- Experience working with REST and GraphQL APIs
- Experience with Docker and PostgreSQL
- Comfortable in Mac/Linux terminal-centric environments
- Practical, hands-on experience with AI-assisted development tools (e.g., Claude Code) and ability to judge AI-generated output
- Leadership experience: guiding senior developers, reviewing pull requests, enforcing code quality and engineering practices
- Upper-Intermediate English level
- Direct production experience with Athena
- Working knowledge of TypeScript and React
- Production experience building AI features (AWS Bedrock, LangChain, Pydantic AI)
- Experience with monorepo tooling (Nx) or modern package managers (Poetry, UV, Yarn)
- Experience with Redis and caching layers or SageMaker
- Experience with marketing data structures, campaign management APIs, or digital advertising metrics
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
AgileEngine is a privately held company established in 2010 that builds dedicated teams of designers and developers. We turn good ideas into awesome software that people actually want to use. Some of the biggest names and the hottest startups around the world chose us to build their tech.
