The Role
Design, build, and maintain scalable Azure-based data pipelines using Databricks and PySpark. Develop ETL/ELT processes, data governance, data quality validations, and lineage tracking. Automate workflows with Python and collaborate with data scientists and stakeholders to deliver reliable datasets for analytics and AI.
Summary Generated by Built In
As a Data Engineer, you will:
- Design, build, and maintain scalable data pipelines using Databricks, PySpark, and Azure Data Factory.
- Develop and optimize data ingestion, transformation, and loading processes for structured and unstructured data sources.
- Implement data governance frameworks to ensure proper data ownership, compliance, and security standards.
- Build and maintain reliable data quality validation processes to ensure accuracy, consistency, and completeness of enterprise data.
- Establish data lineage tracking mechanisms to provide transparency on how data flows across systems and pipelines.
- Develop and maintain Python scripts and automation tools to streamline data engineering workflows.
- Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality datasets for analytics and AI initiatives.
- Optimize data pipeline performance and scalability within Azure-based data platforms.
What You Bring to the Table:
- 8+ years of experience in data engineering, data platform development, or big data environments.
- Strong hands-on experience with Databricks and PySpark for distributed data processing.
- Practical experience working with Azure Data Factory for orchestration and data pipeline development.
- Advanced Python scripting skills for automation, transformation, and integration tasks.
- Solid understanding of data governance principles, including metadata management, data cataloging, and compliance.
- Experience implementing data quality frameworks and validation rules within data pipelines.
- Hands-on experience with data lineage tracking and documentation practices.
- Experience working with large-scale data processing frameworks and cloud-based data platforms.
You should possess the ability to:
- Architect and implement end-to-end data pipelines that support large-scale data processing.
- Design efficient data models and transformation logic using PySpark and Databricks.
- Implement data governance and compliance practices within enterprise data ecosystems.
- Identify and resolve data quality issues proactively through automated validation and monitoring.
- Track and document data lineage across multiple data sources and transformation layers.
- Write clean, scalable, and maintainable Python code for data engineering workflows.
- Collaborate effectively with cross-functional teams including data scientists, analysts, and platform engineers.
- Troubleshoot and optimize data pipeline performance, reliability, and cost efficiency in Azure environments.
What We Bring to the Table:
- Opportunity to work on large-scale cloud data platforms and advanced data engineering solutions.
- Exposure to modern data stack technologies including Databricks, Azure Data Factory, and distributed processing frameworks.
- Collaborative environment working with data scientists, analytics teams, and engineering professionals.
- Challenging projects focused on building scalable and enterprise-grade data platforms.
- Opportunities to enhance expertise in data governance, data quality, and lineage frameworks
Let’s Connect
Want to discuss this opportunity in more detail? Feel free to reach out.
Recruiter: Aswin Dhanvandhar
Phone: +31 20 369 0609 ; Extn :141
LinkedIn: https://www.linkedin.com/in/aswin-dhanvandhar/
Skills Required
- 8+ years of experience in data engineering, data platform development, or big data environments.
- Hands-on experience with Databricks and PySpark for distributed data processing.
- Practical experience working with Azure Data Factory for orchestration and data pipeline development.
- Advanced Python scripting skills for automation, transformation, and integration tasks.
- Solid understanding of data governance principles, including metadata management, data cataloging, and compliance.
- Experience implementing data quality frameworks and validation rules within data pipelines.
- Hands-on experience with data lineage tracking and documentation practices.
- Experience working with large-scale data processing frameworks and cloud-based data platforms.
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
STAFIDE is a Netherlands-based niche technology talent consulting company operating across Europe. It helps organizations identify, recruit, and deploy technology professionals in areas including cybersecurity, cloud engineering, software development, data analytics, ERP, infrastructure, and digital transformation. Its services include recruitment, workforce engagement, secondment, onboarding support, workforce deployment, and workforce analytics that support technology hiring and expansion.
.jpg)
