Top Senior Data Engineer Jobs

Build and optimize scalable batch and streaming data pipelines for Comcast’s enterprise Data Lake. Develop cloud-native solutions using AWS, Databricks, PySpark, Kafka, Kubernetes, and Airflow/MWAA. Support high-volume distributed data processing, production troubleshooting, data warehousing, automation, observability, and data quality initiatives. Collaborate with product, governance, analytics, and engineering teams, participate in architecture decisions, mentor junior engineers, and improve platform reliability and performance.

Own and evolve business-critical data products, designing scalable data services, APIs, models, and batch or streaming pipelines. Lead projects through implementation, deployment, monitoring, and production support while applying AI tools throughout the software development lifecycle. Collaborate across engineering, product, data, and business teams, contribute to architecture and governance, improve reliability, and mentor earlier-career engineers.

Develop and maintain scalable data pipelines and secure infrastructure for multimodal synthetic data, AI evaluation, real-time model monitoring, and clinical research data. Responsibilities include data transformation, preprocessing, quality assurance, database management, privacy compliance, interactive dashboards, pipeline documentation, and scientific dissemination.

Designs, builds, administers, and supports cloud data architectures, databases, schemas, metadata repositories, ETL processes, and analytic models. Leads data quality, governance, troubleshooting, testing, automation, reporting, dashboards, and infrastructure design. Collaborates with Agile teams to translate business requirements into technical capabilities and supports decision-making for Audit and business management. The role is full-time onsite at an eligible Federal Reserve location and requires extensive security screening.

Build and operate scalable data pipelines and trustworthy data models supporting Layla’s AI-powered travel products. Own data quality, reliability, observability, experimentation, and model-evaluation foundations. Enable production AI/ML systems through feature and data availability, evaluation datasets, and monitoring. Partner with Product, Analytics, and ML teams to deliver dependable insights, establish engineering best practices, and mentor other engineers.

Designs, develops, and maintains data platforms, pipelines, and integration solutions across on-premises and cloud environments. Responsibilities include resolving data quality and security issues, building ELT processes, developing data scripts and relational or non-relational models, and monitoring data lake performance and scalability. The role also supports analytical and operational decision-making through reliable data solutions and collaborates with cross-functional teams.

Architect, deploy, and operate core data platform services (Airflow, Superset, Terraform). Build CI/CD pipelines, automate with Python, manage cloud infrastructure, implement security and compliance, develop dbt data models, catalog assets in OpenMetadata, and mentor junior engineers.

Build, optimize, and support scalable batch and streaming data pipelines for enterprise workforce, billing, and interaction datasets. Develop cloud-native solutions with AWS, Databricks, PySpark, Kafka, Kubernetes, and Airflow while processing tens of terabytes of data. Provide production support, performance tuning, troubleshooting, and data platform automation. Collaborate across Product, Governance, Analytics, and Engineering teams, support data warehousing and reliability practices, participate in architecture decisions, and mentor junior engineers.

Lead the architecture and development of scalable AWS-based data ingestion, transformation, and orchestration pipelines. Build reliable data infrastructure using Python, SQL, Airflow, Lambda, ECS, SQS, and Terraform. Establish data modeling, quality, lineage, monitoring, testing, and observability practices while partnering with analysts, scientists, and backend engineers. Mentor senior engineers, guide technical decisions, and provide hands-on leadership for complex data platform initiatives.

Design, build, and operate big-data and streaming platforms (Kafka, Hadoop, Dremio). Develop, deploy, and monitor data pipelines using Java, Python, Spark/Flink; collaborate on data modeling and ingestion, ensure data integrity and availability, act as a Big Data SME, and perform root-cause analysis.

Design, implement, and manage advanced data ecosystems and scalable ETL/ELT pipelines. Analyze complex problems, deliver data warehousing and governance solutions, mentor junior staff, and build client relationships while maintaining high standards and business-context awareness.

Lead design and implementation of enterprise data architecture and cloud-based data solutions. Manage large data engineering projects, develop data models and pipelines, ensure governance and security compliance, advise clients strategically, and build high-performing inclusive teams.

Design, build, and maintain scalable Spark-based ETL pipelines and computed tables in a central data lake. Integrate structured and unstructured IoT, sensor, and external data for analytics, model training, and dashboards. Collaborate with Data Science, Analytics, and ML teams to ensure reliable, high-quality customer-facing datasets.

Lead design, development, and operation of large-scale Spark-based data pipelines and lakehouse solutions. Write and review Python/PySpark and Spark SQL production code, optimize Spark workloads on EMR/Databricks, manage Apache Iceberg table operations, drive AI-assisted engineering adoption, automate reliability and CI/CD practices, and lead technical evaluations and communities of practice.

Design, build, and maintain scalable ETL/ELT pipelines and data warehouse architecture. Partner with cross-functional teams to define KPIs, ensure data quality and governance, own billing data infrastructure, and deliver dashboards and analytics for leadership while mentoring peers.

Design and deliver enterprise-scale data models and analytics-ready datasets in Snowflake; build reliable ELT/ETL pipelines (batch and near-real-time); enforce data quality, observability, and secure handling of PII; optimize performance and publish curated data to downstream systems while collaborating with product, operations, and engineering teams.

Lead the architecture, development, and productionization of scalable data pipelines and governed data products in Databricks. Transform heterogeneous source data for analytics, AI, and machine learning; establish data quality, observability, governance, and delivery standards; and support experimentation, model development, deployment, and monitoring. Lead cross-team initiatives, influence technical roadmaps, mentor engineers and data scientists, and develop reusable AI and multi-agent data capabilities.

Build and support scalable data products, pipelines, and trusted datasets for customer care and aftersales analytics, AI, reporting, and operational decision-making. Optimize distributed workloads for performance, reliability, scalability, and cost. Implement data quality, governance, observability, monitoring, and automation. Develop reusable SQL and Python frameworks, collaborate with technical and business stakeholders, and mentor engineers.

Design and build data infrastructure, pipelines, and transformations using Azure Data Factory, Azure Databricks, and Snowflake. Develop data architecture and modeling, maintain data lakes/warehouses, ensure data quality and security, optimize performance, and collaborate with clients to deliver tailored data solutions and governance.

Lead design and implementation of scalable data infrastructure, pipelines, and transformations using cloud platforms. Build and optimize data warehouses (Snowflake, Databricks), orchestrate workflows (Airflow, Hadoop), ensure data quality and security, and coach teams to deliver analytics solutions that drive tax practice insights and client outcomes.

Lead design and implementation of data infrastructure, pipelines, and warehouses using Azure Data Factory, Databricks, and Snowflake. Manage data modeling, quality, security, and analytics frameworks. Lead client accounts, mentor teams, and drive innovation to deliver data-driven business insights.

Build and maintain production data pipelines, Airflow DAGs, Snowflake infrastructure, and Data Vault 2.0 models. Ensure data quality, observability, security, governance, reliability, and cost efficiency through automated testing and CI/CD. Lead technical projects end-to-end, collaborate with stakeholders, and support SOX-relevant reporting while developing expertise in commercial insurance data and processes.

Designs and maintains scalable data pipelines, data quality frameworks, and cloud-native data solutions using Spark, Databricks, Hadoop, AWS, and advanced SQL. Implements validation, monitoring, observability, and automated remediation capabilities. Applies AI/ML, anomaly detection, and statistical modeling to improve data quality. Provides production support, resolves complex incidents, contributes to architecture and engineering standards, and collaborates with cross-functional data and engineering teams.

Build, optimize, and maintain scalable data pipelines and reusable datasets for business intelligence, marketing, and data science. Develop federated data validation frameworks, optimize queries and data models, and protect user privacy and security. The role uses Python, SQL, AWS, Airflow, PostgreSQL, Iceberg, Snowflake, Spark, and other modern data-stack technologies. It is based in New York City with in-office work three days per week.

Design, build, and maintain optimized, highly available data pipelines and the BMG data warehouse. Lead medium-to-complex data engineering initiatives, perform solution design, code reviews, QA, data integration, and collaborate across teams. Support cloud migration and ensure compliance with data policies for reporting and analytics.