Data Engineer (Databricks)

Posted 9 Days Ago
Be an Early Applicant
Cape Town, Western Cape, ZAF
In-Office
Junior
Agency • Information Technology • Professional Services
The Role
Designs, builds, and optimizes production lakehouse pipelines on Databricks using Python, PySpark, Spark SQL, and Delta Lake. Responsibilities include ingestion, transformation, orchestration, data quality, performance and cost tuning, governance with Unity Catalog, CI/CD, troubleshooting, documentation, and reusable pipeline development across medallion architecture layers.
Summary Generated by Built In

Databricks Consultant – Build and Optimise Production Lakehouse Pipelines

 

We’re looking for a hands-on Databricks Consultant who designs, builds and optimises lakehouse data pipelines and solutions on the Databricks platform.

 

This is a role for someone who likes getting into the platform and making data pipelines reliable, fast and cost-efficient. You’ll work across ingestion, transformation and delivery of curated data products using PySpark, Spark SQL and Delta Lake, together with job orchestration, performance and cost tuning, data quality and platform governance.

 

What you’ll do:

·       Design, build and maintain production data pipelines for ingestion, transformation and delivery of curated data.

·       Develop and optimise PySpark and Spark SQL transformations, including incremental and batch processing patterns.

·       Design and operate Delta Lake tables, including table layout, optimisation and file-size management.

·       Contribute to platform performance and cost efficiency through job, compute and query tuning.

·       Implement data quality checks, validation, testing and clear technical documentation.

·       Build and maintain workflow orchestration, job scheduling, retries and alerting for production pipelines.

·       Apply Unity Catalog for data cataloguing, permissions and lineage where in use.

·       Follow version control and CI/CD practices for notebooks, packages, jobs and pipeline configuration.

·       Tune Spark jobs for scale and cost, including partitioning, shuffles, caching and cluster configuration.

·       Ingest and transform data from enterprise and SaaS sources into well-structured medallion (bronze, silver, gold) or equivalent layers.

·       Troubleshoot failed jobs, slow queries and data defects, and implement lasting technical corrections.

·       Package reusable pipeline components, parameters and configuration for consistent delivery.

·       Document pipeline design, table contracts, refresh behaviour and operational steps.



Requirements

What we’re looking for:

·       Minimum two years of hands-on, day-to-day Databricks experience.

·       Strong proficiency in Python, PySpark and Spark SQL.

·       Practical experience building and operationalising data pipelines on Databricks.

·       Experience working with Delta Lake, including table design and routine optimisation.

·       Solid grasp of data modelling and the ability to write efficient, well-structured SQL and Spark transformations.

·       Experience with incremental processing, orchestration and production operationalisation of pipelines.

·       Experience on at least one major cloud platform (Azure, AWS or GCP).

·       Familiarity with Databricks compute, jobs, workflows and SQL warehouses, including performance and cost awareness.

·       Experience with Git-based version control and CI/CD for data workloads.

·       Working knowledge of data quality checks, testing, technical documentation and access control.



Skills Required

  • Minimum two years of hands-on, day-to-day Databricks experience
  • Strong proficiency in Python, PySpark, and Spark SQL
  • Experience building and operationalizing data pipelines on Databricks
  • Experience with Delta Lake, including table design and routine optimization
  • Strong understanding of data modeling and efficient SQL and Spark transformations
  • Experience with incremental processing, orchestration, and production operationalization of pipelines
  • Experience with at least one major cloud platform: Azure, AWS, or GCP
  • Familiarity with Databricks compute, jobs, workflows, SQL warehouses, performance tuning, and cost management
  • Experience with Git-based version control and CI/CD for data workloads
  • Working knowledge of data quality checks, testing, technical documentation, and access control
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
Year Founded: 2013

What We Do

Sabenza IT is a niche recruitment company specializing in Information Technology, SAP, Finance, and Engineering roles, with over 23 years of experience.

Similar Jobs

Sabenza IT Logo Sabenza IT

Data Engineer

Agency • Information Technology • Professional Services
In-Office
Cape Town, Western Cape, ZAF

CDW Logo CDW

Senior Reward Analyst - SA/UK

Artificial Intelligence • Cloud • Information Technology • Consulting • Cybersecurity • Infrastructure as a Service (IaaS)
Hybrid
Cape Town, Western Cape, ZAF
15100 Employees

Hewlett Packard Enterprise Logo Hewlett Packard Enterprise

Software Engineer

Artificial Intelligence • Cloud • Information Technology • Consulting
In-Office
Cape Town, Western Cape, ZAF
85422 Employees

CDW Logo CDW

Accountant

Information Technology
Hybrid
Cape Town, Western Cape, ZAF
15100 Employees

Similar Companies Hiring

Cravath, Swaine & Moore LLP Thumbnail
Legal Tech • Other • Professional Services
New York, NY
1200 Employees
VCA Animal Hospitals Thumbnail
Pet • Professional Services
Los Angeles, CA
35000 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account