Top Data Engineer Jobs

Build and operate large-scale data pipelines, web-scraping systems, and data-collection tools for AI datasets. Maintain analytical databases, optimize queries, troubleshoot pipeline and data-quality issues, manage containerized workloads across Kubernetes and Linux infrastructure, and support CI/CD workflows. The role also involves documenting technical processes, researching improvements, and developing scalable APIs in a fully remote environment.

Supports the development and maintenance of data pipelines for analytics and application development. Assists with data gathering, cleaning, transformation, architecture, workflow automation, documentation, and knowledge sharing using AWS and Databricks. Collaborates with software developers, analysts, and product teams, participates in code reviews, and explores generative AI tools to improve productivity and data workflows.

Provides technical leadership and hands-on expertise for enterprise data modernization. Designs scalable batch and real-time pipelines, modernizes legacy data stores and BI workloads, implements data quality and reconciliation controls, and enables governed data products using Azure, Databricks, Microsoft Fabric, and related platforms. Leads API- and event-based ingestion, metadata and lineage management, DevSecOps delivery, production support, architecture decisions, engineering standards, and team mentoring.

Design and modify data solutions using Databricks and Palantir Foundry; build data models and reports from ERP and legacy-system datasets; define and execute tests; and translate stakeholder reporting needs into technical specifications. The role requires SQL, Python, PySpark, database architecture, cloud architecture, software engineering practices, independent execution, documentation, and collaboration. It supports an Army client in a hybrid Arlington, Virginia environment and requires an active Secret clearance.

Leads enterprise data platform architecture, engineering standards, automation, security, governance, observability, and cloud integration. Designs scalable batch and streaming pipelines connecting SAP, legacy systems, Databricks, and enterprise data products. Guides SAP transformation data harmonization, analytics and reporting strategy, infrastructure as code, CI/CD, reliability, and platform roadmaps. Provides hands-on technical leadership, mentors senior engineers, collaborates across business and transformation teams, and supports critical incidents.

As the first in-house Data Engineer, you will own the data infrastructure, build and maintain data pipelines, and collaborate with teams to ensure quality data access and insights.

Creates and manages data connections, validates and transforms client files, integrates external APIs and FTP feeds, and normalizes portfolio data into Clearwater’s standardized format. Monitors and automates daily processes, improves auto-reconciliation, resolves data issues, and implements operational enhancements. Communicates technical workflows to stakeholders, documents procedures, and provides light on-call support.

Own the end-to-end commerce data supply chain, including third-party ingestion, schema-drift detection, entity resolution, deduplication, freshness SLAs, and retirement of stale records. Design and operate an LLM validation fleet with evaluation harnesses, golden sets, escalation paths, and defensible unit economics. Build production data systems at scale, establish machine-checkable quality metrics, and work independently with the founder in a fast-moving, in-person environment.

Lead hands-on development and operation of production data pipelines and platforms using PySpark, Python, SQL, and AWS services. Translate requirements into technical designs, data models, SLAs, and reliable pipeline architectures. Establish engineering, data quality, observability, documentation, and coding standards; conduct code reviews; mentor engineers; manage Agile delivery and releases; and use GenAI tools to improve productivity and solution quality.

Design, build, and optimize scalable AWS and Snowflake data pipelines, databases, data models, and cloud-based solutions. Implement ETL/ELT workflows, data quality and monitoring frameworks, CI/CD pipelines, automated recovery, documentation, and performance improvements. Collaborate with stakeholders and cross-functional teams in an Agile environment while conducting advanced analysis using Python and SQL.

Build and maintain scalable data platforms and systems that transform raw market data into validated analytical datasets, including bars, residuals, curves, volatility surfaces, and greeks. Collaborate with trading, research, and external vendor teams to understand datasets and workflows, implement calculations, document technical solutions, and ensure reliable delivery through strong testing and validation practices.

Develop and implement data pipelines and large-scale datasets for reporting and analytics. Analyze data sources for quality and availability, standardize data structures, document technical designs, create unit tests, and support QA testing. The internship provides supervised, practical data engineering experience, mentorship, leadership exposure, and opportunities to present project outcomes.

Build and maintain centralized, validated data platforms covering reference, fundamental, market, index, exchange, instrument, and security master datasets. Analyze vendor data, design scalable data models, implement quality controls, resolve processing issues, and partner with quantitative researchers and traders. Provide technical leadership and mentorship while maintaining testing and system design standards across the Core Data team.

Provides technical leadership for Finance, Actuarial, and Corporate Data platforms. Defines engineering strategy and standards, architects automated data pipelines, leads cloud migrations and ETL modernization, and drives AI-assisted development, DevOps, CI/CD, testing, and data quality practices. Manages senior data engineers and distributed teams, resolves complex engineering issues, governs technical delivery, and communicates strategies to executive stakeholders. Requires extensive enterprise data engineering, cloud platform, data warehousing, automation, and people leadership experience.

Designs and maintains scalable cloud data architectures, pipelines, ETL processes, data warehouses, databases, and data models. Ensures data quality, accuracy, security, availability, and consistency across sources. Develops SQL queries for analytics and reporting, and documents pipelines, schemas, architectures, and models. The role requires expertise with Python, Java, Scala, R, SQL, relational and NoSQL databases, cloud platforms, and Snowflake.

Build and maintain scalable ETL/ELT pipelines and data workflows using Python, PySpark, and SQL. Improve data quality, reliability, architecture, and governance while supporting analytics, reporting, and machine learning initiatives. Collaborate with data scientists, analysts, and business stakeholders, use Azure DevOps for CI/CD and release management, and mentor junior engineers. The role also involves API integration, cloud data platforms, testing, version control, and contributing to engineering best practices.

Build and operate high-volume telemetry and analytics data pipelines supporting real-time monitoring, product decisions, experimentation, and user experience analysis. Design event instrumentation, schemas, quality checks, deduplication, late-event handling, and identity stitching. Partner with data science, product, and SRE teams to shape requirements, and integrate AI tooling into development and data quality workflows.

Designs and delivers large-scale, governed data pipelines and infrastructure across the full data lifecycle. Responsibilities include coding in Python and SQL, optimizing Airflow, dbt, AWS, Kafka, and Iceberg/Parquet workflows, implementing data quality and observability, contributing to architecture and tool evaluations, mentoring engineers, collaborating with stakeholders, and participating in on-call support.

Develops and optimizes scalable ETL/ELT data pipelines and analytics solutions for DoD mission requirements. Processes diverse large-scale datasets, maintains pipeline operations, monitors system health, optimizes databases and costs, prototypes data capabilities, and collaborates with technical teams. Implements DoD STIGs and secure data retrieval methods across enterprise and potentially classified environments.

Design, build, and maintain large-scale ETL pipelines and data warehouses (Redshift) with strong data modeling, governance, and quality controls. Optimize systems for performance and reliability, integrate and deploy AI/ML models, troubleshoot and support Redshift, collaborate with data scientists and stakeholders, and produce technical documentation and knowledge-base articles.

Designs and manages scalable data pipelines, contracts, transformation layers, and analytics models. The role focuses on data governance, observability, lineage, configuration validation, and collaboration with product engineering teams. It requires processing large datasets, integrating cloud and on-premises industrial technologies, and implementing streaming and message-queuing solutions, ideally in cybersecurity or industrial environments.

Architects and evolves enterprise-scale batch, real-time, and event-driven data platforms and pipelines. Leads data modeling, distributed processing, data quality, governance, observability, CI/CD, security, monitoring, incident response, and cost optimization. Provides technical direction across hybrid cloud environments, mentors engineers, leads critical initiatives and code reviews, manages technical debt, evaluates new technologies, and partners with business and technical stakeholders. Participates in on-call rotations and supports distributed and offshore engineering teams.

Builds and maintains production-grade ETL/ELT pipelines across legacy on-premise systems and AWS/Azure cloud platforms, centralizing data in Snowflake for analytics, machine learning, feature stores, and generative AI. Develops vectorization and embedding pipelines, optimizes Snowflake performance and cloud costs, manages cross-platform orchestration, and implements data quality and observability frameworks. Partners with IT, security, MLOps, data science, and infrastructure teams to deliver scalable, governed data solutions.

Build, deploy, and operationalize complex data pipelines and solutions using Databricks, cloud platforms, Python, and SQL. Responsibilities include data transformation, cleansing, governance, security, quality, source analysis, testing, defect resolution, and support for AI, machine learning, and business intelligence. The role also involves cross-team collaboration, stakeholder education, and delivery of resilient data products.

Design, build, test, and maintain cloud-based data pipelines and ETL processes using Python, Databricks and Spark. Develop SQL queries and dashboards, automate workflows, integrate code with GitHub/GitLab CI/CD, troubleshoot data quality, and collaborate with stakeholders to deliver scalable, secure data solutions.