Job
Description – Lead/Senior Databricks Engineer
Role Summary
We are looking
for a skilled Data Engineer with strong experience in Databricks and Azure Cloud to design, develop, and optimize scalable data pipelines
and modern data platforms. The ideal candidate should have expertise in big
data technologies, ETL development, and cloud-based data engineering.
Key
Responsibilities
- Design and develop scalable ETL/ELT pipelines using Databricks and PySpark.
- Build and optimize data ingestion frameworks from
multiple data sources.
- Develop data pipelines using Azure Data Factory
(ADF).
- Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing large datasets.
- Implement Delta Lake and Lakehouse
architecture concepts.
- Optimize Spark jobs for performance and scalability.
- Develop data transformations using SQL and PySpark.
- Collaborate with data architects, analysts, and
business stakeholders.
- Implement CI/CD pipelines and deployment automation.
- Ensure data quality, security, and governance
standards.
Required
Skills
- Strong hands-on experience with Databricks
- Proficiency in PySpark and Spark SQL
- SSIS, Essbase, SQL Server, Data Warehouse, Stored
Procedure, Views Triggers
- Experience with Microsoft Azure
- Good knowledge of:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Delta Lake
- Azure Synapse Analytics
- Strong SQL and data modeling skills
- Experience with Git and CI/CD processes
- Knowledge of performance tuning and optimization
Preferred
Skills
- Experience with:
- Streaming using Spark Structured
Streaming
- Kafka/Event Hub
- Data Warehousing concepts
- DevOps and Infrastructure as Code
- Knowledge of:
- Data Governance
- Data Quality frameworks
- Medallion Architecture (Bronze,
Silver, Gold)
Educational
Qualification
- Bachelor's degree in Computer Science, Information
Technology, or related field.
- Relevant Azure or Databricks certifications are
preferred.
Nice to Have
Certifications
- Databricks Certified Data Engineer Associate
- Microsoft Certified Azure Data Engineer Associate
Requirements
Required
Skills
- Strong hands-on experience with Databricks
- Proficiency in PySpark and Spark SQL
- SSIS, Essbase, SQL Server, Data Warehouse, Stored
Procedure, Views Triggers
- Experience with Microsoft Azure
- Good knowledge of:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Delta Lake
- Azure Synapse Analytics
- Strong SQL and data modeling skills
- Experience with Git and CI/CD processes
- Knowledge of performance tuning and optimization
Skills Required
- 8+ years of professional experience
- Strong hands-on experience with Databricks
- Proficiency in PySpark and Spark SQL
- Experience with SSIS, Essbase, SQL Server, data warehouses, stored procedures, views, and triggers
- Experience with Microsoft Azure
- Knowledge of Azure Data Factory, Azure Data Lake Storage, Delta Lake, and Azure Synapse Analytics
- Strong SQL and data modeling skills
- Experience with Git and CI/CD processes
- Knowledge of performance tuning and optimization
- Bachelor's degree in Computer Science, Information Technology, or a related field
- Experience with Spark Structured Streaming
- Experience with Kafka or Event Hubs
- Knowledge of data warehousing concepts
- Experience with DevOps and Infrastructure as Code
- Knowledge of data governance, data quality frameworks, and Medallion Architecture
- Relevant Azure or Databricks certifications
What We Do
DATAECONOMY is a global, cloud-first data and AI consultancy delivering enterprise-grade solutions through an innovative intellectual-property suite. Its work spans data and BI platform modernization, self-service AI, data mesh and fabric, master data management, governance, cloud enablement, digital engineering, knowledge graphs, and machine lakes supporting cybersecurity and financial-crime use cases for enterprise clients.
