The Role
Design and implement scalable cloud-native data architectures, build and optimise ETL and big-data pipelines using Databricks/Spark/Delta Lake, ensure data quality and governance, automate workflows with Python/Scala/SQL, and collaborate with data scientists and analysts to enable AI-driven insights.
Summary Generated by Built In
Our client is looking for a Sr. Data Engineer who thrives on solving complex data challenges, designing robust architectures, and optimising big data pipelines. You’ll work with cutting-edge cloud and AI solutions, ensuring that data is accessible, scalable, and analytics-ready.
This role blends deep technical expertise, strategic problem-solving, and hands-on implementation. You’ll build and optimise ETL pipelines, data lakes, and data platforms while collaborating with cross-functional teams to drive AI-powered insights.
- Build scalable data architectures – Design and implement high-performance data pipelines and cloud-native solutions.
- Optimise big data processing – Develop efficient and cost-effective data solutions using Databricks, Apache Spark™, and Delta Lake.
- Ensure data quality and governance – Implement best practices for data validation, lineage, and security.
- Drive automation and efficiency – Use Python, Scala, or SQL to automate data workflows and streamline processes.
- Collaborate with cross-functional teams – Work with data scientists, analysts, and engineers to deliver data-driven solutions.
- Stay ahead of the curve – Continuously explore new data engineering trends, AI advancements, and cloud technologies.
Qualifications
- 3+ years of experience in data engineering, big data, or cloud data solutions.
- Proficiency in Python, Scala, or SQL for building scalable data pipelines.
- Experience with Databricks, Apache Spark™, and Delta Lake.
- Familiarity with cloud platforms (AWS, Azure) and modern data architectures.
- Understanding of ETL processes, data lakes, and warehouse solutions.
- Excellent problem-solving skills – Able to tackle complex data challenges with innovative solutions.
- A passion for automation, AI, and optimising data workflows.
Preferred Qualifications
- Databricks Certification (e.g., Databricks Certified Data Engineer or ML Associate).
- Experience with real-time streaming architectures (Kafka, Kinesis, or similar).
- Knowledge of AI/ML pipelines and MLOps best practices.
- Prior experience in a consulting or client-facing role.
Benefits
- Be part of an AI-native company – We live and breathe cutting-edge AI and data solutions.
- Work on high-impact projects – We partner with industry leaders to solve some of the toughest data challenges.
- Shape the future of data engineering – Be at the forefront of AI-driven transformation.
- Competitive compensation – We offer a generous salary and performance-based bonuses.
- Flexible work culture – Hybrid and remote work options to fit your lifestyle.
- Grow with us – Access top-tier learning resources, training, and leadership development programs.
The base pay range for this role is A$125,000 – A$175,000 per year.
Skills Required
- 3+ years of experience in data engineering, big data, or cloud data solutions
- Proficiency in Python, Scala, or SQL for building scalable data pipelines
- Experience with Databricks, Apache Spark, and Delta Lake
- Familiarity with cloud platforms (AWS, Azure) and modern data architectures
- Understanding of ETL processes, data lakes, and data warehouse solutions
- Implement data quality, validation, lineage, and security best practices
- Strong problem-solving skills and passion for automation and AI
- Databricks Certification (e.g., Databricks Certified Data Engineer or ML Associate)
- Experience with real-time streaming architectures (Kafka, Kinesis, or similar)
- Knowledge of AI/ML pipelines and MLOps best practices
- Prior experience in a consulting or client-facing role
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
MatchDB is a specialized talent marketplace that connects Databricks consultants with organizations utilizing the Databricks Lakehouse Platform. It facilitates the hiring process for both permanent and contract roles, serving as a dedicated network for Databricks expertise. The platform provides tools for companies to post jobs and for consultants to find opportunities, streamlining the recruitment process within this specific technical ecosystem.
