The Role
Build and optimize production-grade Databricks data pipelines, including performance and cost tuning. Define trusted metrics and semantic layers, integrate Finance and IT data sources, and uphold engineering standards for testing, documentation, CI/CD, and governance. Partner with stakeholders and data owners to improve data quality and deliver practical solutions. Coach junior and mid-level engineers while providing hands-on technical leadership.
Summary Generated by Built In
Reporting to the Data & Analytics Lead, the Principal Data Engineer is a hands-on role, combining the building and optimization of production data pipelines with the technical leadership needed to raise engineering standards and develop the people around them.
The role functions as a key interface between business stakeholders and department heads, the wider data and analytics team, and the owners of source systems and data in Finance and IT.
Requirements
KEY RESPONSIBILITIES:
- Design and build robust, production-grade data pipelines on Databricks, making appropriate use of current platform capabilities such as Spark Declarative Pipelines (SDP).
- Work “under the hood” to optimise the platform for performance and cost tuning compute, jobs and queries,managing storage and table layout, and keeping platform spend under control.
- Define and maintain consistent business metrics and a semantic layer (for example, using Metric Views) so that reporting is built on trusted, reusable definitions.
- Integrate data from core business systems, including SAP S/4HANA, and other sources, using tools such as Fivetran (both SaaS connectors and HVR).
- Set and uphold engineering standards across the team, including code quality, testing, documentation, CI/CD and data governance.
- Coach and mentor junior and mid-level engineers, review their work and help them develop.
- Partner with business stakeholders to understand their needs, shape practical solutions and ensure the platform delivers genuine value.
- Engage data owners in Finance and IT to ensure data is well understood, valid and fit for purpose, and initiate data quality improvements where required.
REQUIRED SKILLS AND EXPERIENCE:
- At least three years of hands-on, day-to-day Databricks experience, covering both pipeline development and under-the-hood performance and cost optimisation.
- Up to date with recent platform developments, such as Spark Declarative Pipelines (SDP) and Metric Views.
- Strong proficiency in programming languages commonly used in data engineering, such as Python, SQL and Spark.
- Advanced experience with data manipulation, data modelling, database design and query optimisation.
- Experience managing or coaching junior developers.
- A track record of managing and influencing business stakeholders.
KEY COMPETENCIES:
- Combining deep, hands-on engineering skill with sound judgement about cost, performance and long-term maintainability.
- Coaching, mentoring and raising the capability of less experienced engineers.
- Collaborating, communicating confidently and influencing business stakeholders.
- Breaking down complex technical concepts and explaining them simply to non-technical audiences.
- Staying current with a fast-moving platform and bringing new capabilities into everyday practice.
- Taking ownership and driving work independently, from concept through to production.
NICE TO HAVE:
- Experience working with SAP S/4HANA data sets.
- Experience with Fivetran, including both SaaS connectors and HVR.
- Power BI semantic modelling experience.
QUALIFICATIONS:
- A relevant tertiary qualification in Computer Science, Information Systems, Engineering or a related field, or equivalent practical experience.
Skills Required
- At least three years of hands-on Databricks experience covering pipeline development and performance and cost optimization
- Current knowledge of Spark Declarative Pipelines and Metric Views
- Strong proficiency in Python, SQL, and Spark
- Advanced experience with data manipulation, data modeling, database design, and query optimization
- Experience managing or coaching junior developers
- Experience managing and influencing business stakeholders
- Relevant tertiary qualification in Computer Science, Information Systems, Engineering, or a related field, or equivalent practical experience
- Experience with SAP S/4HANA datasets
- Experience with Fivetran SaaS connectors and HVR
- Power BI semantic modeling experience
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Sabenza IT is a niche recruitment company specializing in Information Technology, SAP, Finance, and Engineering roles, with over 23 years of experience.
