Senior Data Pipeline Engineer

Job Description

An experienced data pipeline engineer who uses modern data engineering practices to transform raw data into reliable, consumable data products on AWS and other cloud platforms. The role is responsible for designing, developing, testing, and deploying scalable data pipelines, data warehouses, data lakes, and data products that support business and analytics needs. As a senior member of the analytics team, you will own critical production data pipelines and shape the evolution of our customer-facing data products and metrics. You will work closely with data analysts, data scientists, and other stakeholders to ensure data quality, reliability, and availability across batch and streaming workloads. Typical activities include developing and configuring jobs for data ingestion, transformation, enrichment, efficient database and table design, and exposing curated data to downstream consumers. The role focuses on efficiency and resilience by aligning data platforms and pipelines with business goals and cloud architecture best practices. You will also influence data and platform roadmaps by providing technical leadership, setting best practices, and mentoring other engineers.

Job Duties

• Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark and cloud-native services (for example AWS Glue, EMR, Kinesis, and Lambda).
• Utilize and optimize Apache Spark (RDDs, DataFrames, Spark SQL) for distributed processing of large datasets, including both batch and near real‑time use cases.
• Implement robust ETL/ELT processes to ingest and transform data from databases, APIs, files, and event streams into curated datasets stored in S3 data lakes, data warehouses (such as Amazon Redshift), and data marts.
• Implement data quality checks, validation rules, and governance controls (including schema enforcement, profiling, and reconciliation) to ensure accuracy, completeness, and consistency.
• Develop and maintain logical and physical data models, schemas, and metadata in catalogs to support analytics, BI, and ML consumption.
• Create and manage data warehouses, data lakes, and data marts on AWS and other cloud platforms (such as Azure or GCP) following modern architectural patterns.
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and translate them into scalable pipeline and modeling solutions.
• Collaborate with DevOps, platform, security, and compliance teams to ensure secure, reliable cloud implementations and adherence to organizational standards.
• Develop cloud and data architecture documentation, including diagrams, guidelines, and best practices, to enable knowledge sharing and reuse.
• Troubleshoot and resolve data pipeline and job issues across development and production environments, ensuring minimal downtime and preserving data integrity.
• Continuously optimize data pipelines for performance, cost, reliability, and data quality using best practices in distributed data engineering and cloud resource tuning.
• Build algorithms and prototypes that combine and reconcile raw information from multiple sources, including resolving data conflicts and inconsistencies.
• Provide technical leadership for the analytics data stack, including reviewing designs, establishing standards for observability and reliability, and guiding junior engineers in delivering high-quality solutions.
• Define and manage data and cloud infrastructure using infrastructure‑as‑code tools such as Terraform (and/or AWS CDK/CloudFormation) to ensure consistent, repeatable environments across development, test, and production.
• Participate actively in agile ceremonies (backlog refinement, sprint planning, daily stand‑ups, reviews), including estimating and updating user stories, tracking progress, and collaborating closely with data product and analytics stakeholders.

Required Skills

• Bachelor’s degree in Computer Science, Engineering, Mathematics, or related field, or equivalent work experience.
• 6+ years of experience in data engineering or closely related roles, working with large, complex datasets.
• Demonstrated experience owning production-grade data pipelines end to end, from design and implementation through monitoring, incident response, and continuous improvement.
• Extensive hands-on experience with Apache Spark for large-scale data processing, including RDDs, DataFrames, and Spark SQL.
• Familiarity with big data ecosystem components such as HDFS, Hive, and HBase, and their cloud-native equivalents on AWS and other clouds.
• Experience with SQL and NoSQL databases such as MySQL, PostgreSQL, DynamoDB, or similar technologies.
• Strong proficiency in SQL and at least one programming language such as Python (preferred) for data processing, automation, and orchestration glue code.
• Experience with data pipeline orchestration and scheduling tools such as AWS Step Functions, Amazon Managed Workflows for Apache Airflow (MWAA), or Apache Airflow.
• Experience with cloud-based data platforms and services, ideally AWS (S3, Glue, EMR, Redshift, Kinesis, Lambda), with exposure to Azure or GCP as a plus.
• Experience designing and implementing data warehouses and data lakes, including partitioning, file formats, and performance optimization.
• Experience with data quality, automated data testing, and data governance methodologies and tools; familiarity with lineage, cataloging, and access controls.
• Strong analytical and problem-solving skills, high attention to detail, and clear written and verbal communication.
• Ability to work independently and collaboratively in a fast-paced, agile, and cross-functional environment.
• Experience working with a modern data catalog such as Alation, Collibra, or similar tools is a plus.
• Ability to prepare and curate data for prescriptive and predictive modeling (for example, features for ML models) is a plus.
• Hands‑on experience with infrastructure as code, preferably Terraform (and/or AWS CDK/CloudFormation), to provision and manage data and cloud resources.
• Practical experience working in an agile delivery model, including breaking down work into user stories, sizing and updating them during the sprint, and delivering incrementally.