Lead Azure Data Engineer

Key Responsibilities

• Data Platforms & Pipeline Engineering
• Lead end-to-end development of scalable data pipelines and orchestration frameworks using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric.
• Build robust real-time and batch data pipelines, including integration with streaming sources (e.g., Event Hubs, Kafka) and structured streaming engines.
• Design and implement Structured Streaming applications in Spark for near-real-time processing of streaming data.
• Develop and maintain ETL/ELT pipelines and transformations leveraging Spark, PySpark, SQL, and fabric orchestration capabilities.
Microsoft Fabric & Azure Ecosystem
• Architect and implement data solutions using Microsoft Fabric, including OneLake, Dataflows, warehouses, and Fabric capacity planning to support enterprise analytics.
• Collaborate on data governance, cataloging, and asset organization using Unity Catalog within Databricks and Fabric environments.
• Manage Microsoft Fabric capacity and resource utilization to optimize performance and cost efficiency for analytics workloads.
Databricks Engineering & Dashboards
• Design, deploy, and optimize Databricks dashboards and reporting artifacts for business stakeholders.
• Apply best practices for data modelling, caching, file sizing, and performance tuning of Spark and Delta Lake jobs (e.g., Z-ORDER, broadcast joins, adaptive query execution).
• Oversee governance, access controls, metadata management, and lineage using Unity Catalog.
Leadership & Collaboration
• Lead and mentor a team of data engineers, fostering best practices in development, operations, documentation, and quality.
• Work with cross-functional teams (architecture, BI, data science, DevOps) to translate business requirements into scalable data solutions.
• Partner with stakeholders to define data strategy, standards, and architectural roadmaps.
Quality, Governance & Compliance
• Establish and enforce standards for data quality, testing, monitoring, operational observability, and governance.
• Implement secure, compliant data access and lineage frameworks across cloud data platforms.
DevOps & CI/CD
• Implement CI/CD pipelines, infrastructure-as-code for data platform artifacts, and automated testing frameworks for data jobs and workflows.

Required Skills & Qualifications

• 10+ years of hands-on experience in data engineering on Azure with deep expertise in ADF, Synapse, Databricks, and Microsoft Fabric.
• Proven experience with real-time data processing, streaming architectures, and Spark Structured Streaming.
• Strong proficiency in Azure Data Factory, Spark (PySpark), SQL, Azure Synapse Analytics, Databricks Runtime, and cloud storage.
• Solid knowledge of Unity Catalog for data governance, security, and access management.
• Experience designing and managing Databricks Dashboards, performance optimization, cost controls, and data platform resource tuning.
• Expertise in building scalable, fault-tolerant, and high-throughput batch & streaming data solutions.
• Excellent leadership, cross-team collaboration, and communication skills.

Good-to-Have Skills

• Experience with DBT (Data Build Tool) for modular transformation workflows.
• Hands-on expertise in Power BI for analytics and visualization.
• Familiarity with data governance tools such as Azure Purview or equivalent.
• Certifications like Microsoft Azure Data Engineer (DP-203), Databricks Data Engineer, or Microsoft Fabric Analytics Engineer.