RequirementsData Engineer – Candidate Requirements
Intermediate Level
3–5 years' experience
3–5 years of hands-on experience in data engineering.
Strong proficiency in Python and/or SQL, including query optimisation.
Experience working with both relational and non-relational databases.
Experience designing and building data pipelines and data models.
Understanding and practical experience with lakehouse architectures, including the medallion pattern.
Practical experience with at least one major cloud platform, including:
Microsoft Azure
AWS
Google Cloud Platform (GCP)
Familiarity with:
Databricks
Snowflake
Delta Lake
PySpark
Understanding of data transformation frameworks such as dbt.
Experience with version control using Git.
Understanding of CI/CD practices for data workflows.
Strong analytical and problem-solving skills.
Ability to perform root-cause analysis on complex data issues.
Good communication and stakeholder engagement skills.
6–8+ years' experience
6–8+ years of hands-on experience in data engineering.
All intermediate-level technical requirements, together with demonstrable experience in:
Leading end-to-end data platform delivery.
Architecting enterprise-grade lakehouse environments.
Implementing data mesh patterns.
Infrastructure-as-code using tools such as Terraform, Bicep, AWS CDK or Pulumi.
DevOps and CI/CD pipelines.
Working effectively with cross-functional teams in a dynamic consulting environment.
Mentoring junior engineers.
Contributing to technical strategy and solution direction.
Bachelor's degree in:
Computer Science
Information Systems
Information Technology
or a related field.
Master's degree in a relevant field is advantageous.
One or more of the following certifications would be advantageous:
Microsoft Fabric Data Engineer Associate
Microsoft Azure Data Engineer Associate
Databricks Certified Data Engineer Associate
Google Professional Data Engineer
AWS Certified Data Engineer – Associate
Databricks Certified Data Engineer Professional
Languages & Frameworks
Python
PySpark
SQL
dbt
Microsoft Fabric Lakehouses
Fabric Pipelines
Fabric Semantic Models
Direct Lake
Azure Data Factory
Azure Data Lake Storage Gen2
Azure Synapse Analytics
Azure Databricks
Azure Event Hubs
BigQuery
Cloud Storage
Dataflow
Dataproc
Pub/Sub
Amazon S3
AWS Glue
Amazon Redshift
Amazon EMR
Amazon Kinesis
Databricks
Delta Lake
Unity Catalog
MLflow
Databricks Workflows
Azure SQL
Azure Cosmos DB
PostgreSQL
Snowflake
BigQuery
Amazon Redshift
Git
Azure DevOps
GitHub Actions
Terraform
Bicep
AWS CDK
CI/CD pipelines
Azure Event Hubs
Azure Stream Analytics
Apache Kafka
Amazon Kinesis
Google Pub/Sub
Microsoft Power BI
Microsoft Fabric Real-Time Dashboards
Looker / Looker Studio
Amazon QuickSight
Skills Required
- 3–5 years of hands-on data engineering experience for intermediate-level roles
- 6–8+ years of hands-on data engineering experience for senior-level roles
- Strong proficiency in Python and/or SQL, including query optimization
- Experience with relational and non-relational databases
- Experience designing and building data pipelines and data models
- Understanding and practical experience with lakehouse architectures, including the medallion pattern
- Practical experience with at least one major cloud platform: Azure, AWS, or Google Cloud Platform
- Familiarity with Databricks, Snowflake, Delta Lake, and PySpark
- Understanding of data transformation frameworks such as dbt
- Experience using Git for version control
- Understanding of CI/CD practices for data workflows
- Strong analytical and problem-solving skills
- Ability to perform root-cause analysis on complex data issues
- Good communication and stakeholder engagement skills
- Leading end-to-end data platform delivery
- Architecting enterprise-grade lakehouse environments
- Implementing data mesh patterns
- Infrastructure as code using Terraform, Bicep, AWS CDK, or Pulumi
- DevOps and CI/CD pipeline experience
- Experience working with cross-functional teams in a consulting environment
- Experience mentoring junior engineers
- Contributing to technical strategy and solution direction
- Bachelor’s degree in Computer Science, Information Systems, Information Technology, or a related field
- Master’s degree in a relevant field
- Microsoft Fabric Data Engineer Associate certification
- Microsoft Azure Data Engineer Associate certification
- Databricks Certified Data Engineer Associate certification
- Google Professional Data Engineer certification
- AWS Certified Data Engineer Associate certification
- Databricks Certified Data Engineer Professional certification
What We Do
Blue Pearl is a market-leading CLOUD Solutions developer with extensive knowledge and insight into the latest technologies, standardised processes, advanced technical capabilities and consulting processes available, ensuring wholistic success for our clientele. We offer professional consulting to compliment your business strategy and overall management and make it our priority to add value to any business by listening, analysing and creating a conducive solution that will empower our client. We implement a Data Analysis Process that includes inspecting, cleansing, transforming, and modelling data with the end-goal of discovering useful information, informing conclusions, and relevant information to support your decision-making. Your business cannot afford not to engage with us, allowing our data analysis to play a role in making your business decisions more scientific and helping your business achieve effective operation. Blue Pearl’s team of experts include BI strategists, BI analysts, Data Warehouse Architects, Data Scientists, Implementation and Development experts. With the use of BI, Analytics and Big Data, we effectively partner with our customers on their mission to achieve a competitive business advantage and real ROI from the structured information we collect.
