The Role
Design, build, and maintain scalable ETL/ELT pipelines and lakehouse architectures on Azure Databricks. Implement batch and streaming workflows, enforce data modeling and governance, optimize Spark transformations, deploy using CI/CD and IaC, mentor junior engineers, and collaborate with data scientists to productionize ML models.
Summary Generated by Built In
JOB DESCRIPTION
Senior Data Engineer
Data & Analytics Engineering
You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment.
Senior Data Engineer
Data & Analytics Engineering
About the Role
We are looking for a Senior Data Engineer to join our growing Data & Analytics team. In this role, you will design, build, and maintain scalable data pipelines and platforms that power critical business decisions. You will be a key contributor in driving our cloud-first data strategy, leveraging Azure Databricks as a core technology and Azure Data Factory (ADF) as Good to have feature.You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment.
Key Responsibilities
Data Pipeline & Architecture- Design, develop, and maintain robust ETL/ELT pipelines using Azure Databricks and Azure Data Factory (ADF).
- Architect and implement scalable data lakehouse solutions on Azure using Delta Lake.
- Build and optimize data workflows across batch and streaming workloads (Spark Structured Streaming, Event Hubs).
- Define and enforce data modeling best practices (star schema, data vault, medallion architecture).
- Develop and optimize Spark-based data transformations using PySpark and Spark SQL in Databricks.
- Manage Databricks clusters, jobs, and workspace configurations for performance and cost efficiency.
- Implement Delta Live Tables (DLT) pipelines for declarative, auto-scaling data transformations.
- Leverage Unity Catalog for data governance, lineage tracking, and access control.
- Utilize Databricks Asset Bundles (DABs) and CI/CD practices for deployment automation.
- Build, schedule, and monitor complex ADF pipelines with parameterized templates.
- Integrate ADF with Azure Key Vault, Linked Services, and Integration Runtimes (SHIR/Azure IR).
- Implement incremental load patterns, watermarking, and change data capture (CDC) strategies.
- Troubleshoot pipeline failures and optimize ADF pipeline performance and cost.
- Implement data quality frameworks and validation checks across pipelines.
- Enforce data cataloging, lineage, and metadata management practices.
- Collaborate with data governance teams to ensure compliance with data policies and regulations (GDPR, HIPAA).
- Mentor junior data engineers and conduct code reviews.
- Work closely with data scientists and ML engineers to productionize machine learning models.
- Partner with DevOps/Cloud teams on infrastructure-as-code (Terraform/Bicep) for data platform provisioning.
- Document architecture decisions, pipeline designs, and operational runbooks.
Required Qualifications
- 7+ years of hands-on experience with Azure Databricks (Spark SQL, Delta Lake, PySpark).
- Strong proficiency in SQL preferred and Python.
- Deep understanding of distributed computing principles and the Apache Spark ecosystem.
- Experience with Azure data services: ADLS Gen2, Azure Synapse, Azure SQL, Event Hubs / Kafka.
- Solid understanding of data warehousing concepts and dimensional modeling.
- Experience with version control (Git) and CI/CD tools (Azure DevOps, GitHub Actions).
- Familiarity with infrastructure-as-code tools (Terraform or ARM/Bicep).
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification.
- Microsoft Certified: Azure Data Engineer Associate (DP-203).
- Experience with Delta Live Tables (DLT) and Databricks Workflows.
- Familiarity with Power BI or other BI/reporting tools.
- Experience with Scala or Java for Spark development.
Technology Stack
Core PlatformsAzure Databricks (Preferred), Azure Data Factory (ADF) (Good to have)Data StorageAzure Data Lake Storage Gen2, Delta Lake, Azure SQL, Cosmos DBLanguagesPython (PySpark), SQLOrchestrationADF Triggers, Databricks Workflows, Apache AirflowMonitoringAzure Monitor, Log Analytics, Databricks Cluster PoliciesGovernanceUnity Catalog, Azure Purview, Azure Key Vault – (Good to have)BI / ReportingPower BI, Tableau Key Competencies
- Strong analytical and problem-solving skills with attention to detail.
- Excellent communication skills — ability to translate complex technical concepts to non-technical stakeholders.
- Self-motivated and able to manage competing priorities in an agile environment.
- Collaborative team player with a growth mindset and eagerness to mentor others.
- Proactive in identifying performance bottlenecks and proposing architectural improvements.
Skills Required
- 7+ years hands-on experience with Azure Databricks (Spark SQL, Delta Lake, PySpark).
- Proficiency in Python for Spark/PySpark development.
- Proficiency in SQL.
- Deep understanding of distributed computing and Apache Spark ecosystem.
- Experience with Azure data services: ADLS Gen2, Azure Synapse, Azure SQL, Event Hubs / Kafka.
- Solid understanding of data warehousing concepts and dimensional modeling.
- Experience with version control (Git) and CI/CD tools (Azure DevOps, GitHub Actions).
- Familiarity with infrastructure-as-code tools (Terraform or ARM/Bicep).
- Experience building and optimizing batch and streaming data workflows (Spark Structured Streaming).
- Experience with data governance, lineage, and security tools (Unity Catalog, Azure Key Vault, Azure Purview).
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Concord is a technology consultancy building connected customer experiences backed by powerful AI & analytics and underpinned by secure IT foundations. Digital Experience | Data & Analytics | Engineering & Applications
