The Role
Lead design and delivery of enterprise-scale cloud data platforms, architect and build scalable ETL/ELT pipelines, enforce data governance and quality, mentor engineers, collaborate with BI and stakeholders, and ensure platform reliability, performance, and security.
Summary Generated by Built In
Our client is seeking an experienced Data Engineering Lead to design, build, and lead enterprise-scale data platforms that enable business intelligence, advanced analytics, reporting, and data-driven decision-making.
The successful candidate will provide technical leadership across the full data engineering lifecycle, ensuring scalable, secure, reliable, and high-performing data solutions. This role requires close collaboration with business stakeholders, solution architects, BI teams, governance specialists, and software engineering teams to deliver trusted and analytics-ready data assets.
The ideal candidate is a hands-on technical leader with extensive experience in cloud data platforms, ETL/ELT development, data architecture, SQL optimization, and modern data engineering practices.
Key Responsibilities
Data Engineering Leadership
Lead the design, development, implementation, and continuous improvement of enterprise-grade data engineering solutions.
Architect scalable, secure, reusable, and high-performance data pipelines supporting batch and near real-time processing.
Provide technical leadership across multiple data engineering initiatives and mentor junior and intermediate engineers.
Establish and enforce enterprise data engineering standards, frameworks, and best practices.
Drive innovation and continuous improvement across the organisation's data platform capabilities.
Data Platform Architecture
Design robust ingestion frameworks for structured, semi-structured, and unstructured data.
Develop scalable storage architectures supporting data warehousing and lake house environments.
Implement efficient ETL and ELT processing frameworks.
Optimize cloud-native data architectures for performance, scalability, reliability, and cost efficiency.
Design resilient solutions supporting analytics, reporting, AI, and machine learning initiatives.
Data Engineering & Development
Build, maintain, and optimize enterprise data pipelines.
Develop complex SQL queries, stored procedures, and transformation logic.
Design and maintain reusable data integration components.
Perform data transformation, cleansing, enrichment, and validation processes.
Ensure consistent data availability and integrity across enterprise platforms.
Data Governance & Quality
Embed data quality controls throughout the engineering lifecycle.
Implement automated validation, monitoring, lineage, metadata management, and observability.
Ensure compliance with organizational governance frameworks and regulatory requirements.
Secure sensitive and confidential information through appropriate engineering controls.
Maintain technical documentation including:
Source-to-target mappings
Data dictionaries
Data lineage
Technical specifications
Data models
Pipeline documentation
Stakeholder Management
Collaborate closely with:
Business stakeholders
Business Intelligence teams
Data Analysts
Solution Architects
Application Development teams
Governance teams
Infrastructure teams
Translate business requirements into scalable technical solutions.
Communicate technical concepts clearly to both technical and non-technical audiences.
Provide regular project updates, technical recommendations, and risk assessments.
Operational Excellence
Monitor production data platforms.
Troubleshoot pipeline failures and performance issues.
Optimize processing times and infrastructure costs.
Lead root cause analysis and incident resolution.
Support production deployments and platform upgrades.
Requirements
Minimum Requirements
Education
Bachelor's Degree in one of the following:
Computer Science
Information Systems
Information Technology
Software Engineering
Data Science
Engineering
Mathematics
Statistics
Or a related discipline
Relevant postgraduate qualifications will be advantageous.
Experience
8–10+ years' experience in Data Engineering.
Minimum 3 years leading technical data engineering teams.
Experience designing enterprise-scale cloud data platforms.
Proven experience building scalable ETL/ELT solutions.
Experience working within enterprise environments.
Strong experience delivering complex data integration solutions.
Experience supporting reporting, analytics, and data science initiatives.
Experience implementing data governance frameworks.
Experience in Financial Services, Mining, Retail, Telecommunications, or Consulting environments will be advantageous.
Technical Knowledge
The successful candidate should possess strong knowledge of:
Modern Data Engineering architectures
Enterprise Data Platforms
Data Warehousing
Lake house Architecture
ETL / ELT Frameworks
Data Integration
Cloud Computing
SQL Performance Optimization
Data Modelling
Metadata Management
Data Governance
Master Data Management
Data Quality
Data Lineage
CI/CD
Infrastructure as Code
Software Development Lifecycle (SDLC)
Information Security
Data Privacy Regulations
Technical Skills
Cloud Platforms
Experience with one or more:
AWS
Microsoft Azure
Snowflake
Databricks
Programming & Query Languages
Advanced SQL
Python
Spark
PySpark
Data Engineering Technologies
Databricks
Azure Data Factory
AWS Glue
Snowflake
Apache Spark
Delta Lake
Airflow
Data Pipeline Orchestration Tools
Business Intelligence Tools
Experience with one or more:
Power BI
Qlik Sense
QlikView
Tableau
Database Technologies
SQL Server
PostgreSQL
Oracle
Snowflake
Azure SQL
Amazon Redshift
Additional Technical Skills
Git
Azure DevOps
Terraform
CI/CD Pipelines
API Integration
REST Services
Data Security
Data Encryption
Behavioural Competencies
The successful candidate should demonstrate:
Technical Leadership
Strategic Thinking
Analytical Thinking
Strong Problem-Solving Ability
Excellent Communication Skills
Stakeholder Management
Decision-Making Ability
Continuous Improvement Mindset
Innovation
Collaboration
Accountability
Attention to Detail
Adaptability
Results Orientation
Planning and Organizing
Mentoring and Coaching
Preferred Certifications
One or more of the following would be advantageous:
Microsoft
Microsoft Certified: Azure Data Engineer Associate
Azure Solutions Architect Expert
AWS
AWS Certified Data Engineer – Associate
AWS Certified Solutions Architect – Associate / Professional
Databricks
Databricks Certified Data Engineer Associate
Databricks Certified Data Engineer Professional
Snowflake
SnowPro Core Certification
Other
Certified Data Management Professional (CDMP)
Informatica Certification
TOGAF (Advantageous)
Key Success Measures
The successful candidate will be measured on:
Delivery of scalable, secure data platforms.
Data pipeline reliability and availability.
Data quality and integrity.
Platform performance and optimization.
Successful project delivery within agreed timelines.
Stakeholder satisfaction.
Reduction in operational incidents.
Engineering automation and efficiency improvements.
Compliance with enterprise governance standards.
Mentoring and development of engineering team members.
Why Join This Opportunity?
This is an exciting opportunity to lead the development of modern enterprise data platforms within a collaborative and innovative environment. You will play a key role in shaping the organisation's data engineering capability, enabling business intelligence, advanced analytics, AI initiatives, and digital transformation through scalable, secure, and high-performing data solutions.
Skills Required
- Bachelor's degree in Computer Science, Information Systems, IT, Software Engineering, Data Science, Engineering, Mathematics, Statistics or related discipline
- 8-10+ years experience in Data Engineering
- Minimum 3 years leading technical data engineering teams
- Designing enterprise-scale cloud data platforms and lakehouse/warehouse architectures
- Proven experience building scalable ETL/ELT solutions and data pipelines (batch and near-real-time)
- Strong SQL skills and SQL performance optimization experience
- Experience with cloud platforms (AWS or Microsoft Azure) and cloud-native data architectures
- Experience with Databricks, Snowflake, Apache Spark, Delta Lake, or similar data engineering technologies
- Experience implementing data governance, metadata management, data lineage, and data quality frameworks
- Experience supporting reporting, analytics, and data science initiatives (BI tool experience such as Power BI, Tableau, Qlik)
- Familiarity with orchestration and pipeline tooling (Airflow, Azure Data Factory, AWS Glue)
- Experience with CI/CD, Infrastructure as Code (Terraform), Git, and Azure DevOps or equivalent
- Knowledge of information security, data privacy regulations, and engineering controls for sensitive data
- Experience working in enterprise environments; experience in Financial Services, Mining, Retail, Telecommunications, or Consulting advantageous
- Postgraduate qualifications or industry certifications (Azure/AWS/Databricks/Snowflake/CDMP)
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Blue Pearl is a market-leading CLOUD Solutions developer with extensive knowledge and insight into the latest technologies, standardised processes, advanced technical capabilities and consulting processes available, ensuring wholistic success for our clientele. We offer professional consulting to compliment your business strategy and overall management and make it our priority to add value to any business by listening, analysing and creating a conducive solution that will empower our client. We implement a Data Analysis Process that includes inspecting, cleansing, transforming, and modelling data with the end-goal of discovering useful information, informing conclusions, and relevant information to support your decision-making. Your business cannot afford not to engage with us, allowing our data analysis to play a role in making your business decisions more scientific and helping your business achieve effective operation. Blue Pearl’s team of experts include BI strategists, BI analysts, Data Warehouse Architects, Data Scientists, Implementation and Development experts. With the use of BI, Analytics and Big Data, we effectively partner with our customers on their mission to achieve a competitive business advantage and real ROI from the structured information we collect.
