Senior Data Engineer

Senior Data Engineer

Job Description

We are looking for a Senior Data Engineer to join our technology organization. As a Senior Data Engineer, you will lead strategies for modernizing and remediating legacy data platforms by leveraging existing tools and implementing new lakehouse and cloud data platform capabilities. You will architect and build scalable data pipelines on the Azure stack, design dimensional data warehouse models, and champion engineering best practices across the team. You will also drive the adoption of AI-assisted engineering, leveraging platforms such as OpenAI and Anthropic to accelerate pipeline development, automate data validation, and deploy solutions at scale. As part of a high-performing team working on mission-critical projects with visibility across the organization, you will develop critical insight into the company and support every function of the business, taking ownership of data quality and treating data as a product.

Responsibilities

•        Architect, design, develop, and own robust, high-performance batch and streaming data pipelines and RESTful APIs serving analytics and operational workloads.

•        Lead the design and implementation of lakehouse solutions on Databricks, including Delta Lake, medallion (bronze/silver/gold) architecture, Delta Live Tables, Unity Catalog governance, and Spark performance optimization.

•        Build and orchestrate ingestion and transformation workflows using Azure Data Factory, Databricks Workflows, and dbt across a wide variety of structured, semi-structured, and unstructured data sources.

•        Design and maintain enterprise data warehouse models grounded in dimensional modeling best practices — star schemas, conformed dimensions, slowly changing dimensions, and fact table design — to support reliable, performant analytics.

•        Integrate data from internal and third-party systems by building and consuming RESTful APIs, handling authentication, pagination, rate limiting, and schema evolution.

•        Leverage AI platforms such as OpenAI and Anthropic (Claude) to accelerate pipeline development, generate and refactor transformation code, automate data quality validation and anomaly detection, enrich and classify data, and scale deployment through AI-assisted CI/CD.

•        Implement automated data quality frameworks, testing, and data observability (freshness, volume, schema drift, and lineage monitoring) to ensure trusted data across the platform.

•        Define and document cloud solution architectures, technical designs, and diagrams; contribute to architectural decisions and evaluate system implementations as a critical stakeholder.

•        Establish CI/CD pipelines and Infrastructure as Code for data workloads using GitHub Actions and/or Azure DevOps, with automated unit and integration testing.

•        Gain a thorough understanding of the business and its data strategy; assemble large, complex data sets that meet functional and non-functional requirements.

•        Mentor junior engineers, conduct code reviews, and set standards for engineering excellence, documentation, and security across the data team.

•        Troubleshoot and resolve issues pertaining to data management, articulating opportunities for continuous improvement with a strong customer focus, ownership, urgency, and drive.

The above cited duties and responsibilities describe the general nature and level of work performed by people assigned to the job. They are not intended to be an exhaustive list of all the duties and responsibilities that an incumbent may be expected or asked to perform.

Qualifications

•        B.S. in Computer Science, Engineering, or equivalent required; advanced degree a plus.

•        Minimum of 8 years of hands-on data engineering experience building and operating production data platforms.

•        Expert-level experience with Databricks, including Apache Spark/PySpark, Delta Lake, medallion architecture, Unity Catalog, Delta Live Tables, and cluster/cost optimization.

•        Proficiency with the Azure data stack, particularly Azure Data Factory, along with services such as Azure Data Lake Storage (ADLS Gen2), Azure Synapse, Event Hubs, Key Vault, and Azure Functions.

•        Very strong command of data warehousing concepts and dimensional modeling, with demonstrated experience designing star schemas, conformed dimensions, slowly changing dimensions, and ETL/ELT best practices.

•        Proven experience building, consuming, and integrating RESTful APIs for data ingestion and delivery.

•        Practical know-how leveraging AI platforms such as OpenAI and Anthropic to gain engineering efficiencies — building data pipelines faster, validating data with LLM-assisted checks, and deploying solutions at scale (prompt engineering, API integration, and responsible AI practices).

•        Deep knowledge of Python and strong experience in SQL, including query tuning and performance optimization.

•        Experience with dbt for modular, tested, and documented transformations.

•        Experience deploying Azure Infrastructure as Code (Terraform or Bicep) and building CI/CD pipelines using GitHub and/or Azure DevOps and other source control environments is required.

•        Experience with streaming and near-real-time data processing (e.g., Spark Structured Streaming, Event Hubs, or Kafka) preferred.

•        Familiarity with data governance, security, lineage, and observability tooling (e.g., Unity Catalog, Microsoft Purview, Great Expectations, or similar) preferred.

•        Azure and/or Databricks certifications (e.g., Azure Data Engineer Associate, Databricks Certified Data Engineer Professional) preferred.

•        Strong analytic skills related to working with structured, semi-structured, and unstructured datasets.

•        Excellent verbal, written, and interpersonal communication skills, with the ability to articulate technical solutions to both technical and business audiences.

•        Ability to influence and build relationships with engineering and data science teams, technology leadership, external service providers, infrastructure, and enterprise architecture teams.