Data Engineer

The Data Engineer designs, builds, and maintains scalable data architecture, pipelines, and platforms that support analytics, reporting, and AI/ML solutions across the organization. This role is responsible for preparing, integrating, governing, and optimizing data from multiple sources to ensure reliable, accessible, and high-quality data assets for business and technical use. The Data Engineer works closely with data scientists, software engineers, analysts, and business stakeholders to deliver cloud-based data solutions that support operational goals, innovation, and emerging AI capabilities.

What You Will Do:

• Design, develop, and maintain scalable data pipelines, data models, and lakehouse or data warehouse structures to support analytics, reporting, and AI/ML workloads.

• · Build, test, and optimize ETL/ELT workflows that ingest, transform, and deliver data from multiple structured, semi-structured, and unstructured sources.

• · Support AI/ML initiatives by preparing and managing datasets for feature engineering, model training, inference, and other AI-enabled applications.

• · Implement and maintain data quality controls, validation processes, monitoring, and observability practices to improve reliability, accuracy, and performance of data pipelines.

• · Manage data integration processes across internal and external systems to ensure timely, secure, and efficient data availability.

• · Support data governance practices, including lineage, cataloging, access management, and documentation, to promote trusted and well-controlled data assets.

• · Develop and deploy cloud-based data engineering solutions using modern platforms, distributed processing tools, and containerized environments where applicable.

• · Build and maintain batch and near real-time data pipelines to meet business, reporting, and application requirements.

• · Collaborate with cross-functional teams to gather requirements, translate business needs into technical solutions, and support enterprise data initiatives.

• · Troubleshoot, enhance, and continuously improve data infrastructure, workflows, and related tools to increase scalability, efficiency, and business value.

• · Remain current with developments in data engineering, cloud platforms, and AI-enabling technologies and apply practical improvements where beneficial.

• · Comply with applicable ABS Health, Safety, Quality, and Environmental Management System requirements and other internal policies and procedures.

What You Will Need:

Education and Experience

• Degree in a technical field or equivalent combination of education and experience
• 8+ years of relevant experience in data engineering or a closely related discipline, including significant experience building and maintaining cloud-based data pipelines and architectures.
• Databricks certification (preferred)

Knowledge, Skills, and Abilities

• Strong knowledge of data engineering concepts, including data modeling, data integration, ETL/ELT design, and pipeline orchestration.

• · Strong programming skills in Python and SQL, with the ability to build, test, and maintain production-grade data workflows.

• · Experience working with cloud-based data platforms such as AWS, Azure, or Google Cloud.

• · Experience designing, implementing, and supporting data warehouse or lakehouse architectures such as Databricks, Snowflake, Redshift, or similar technologies.

• · Knowledge of distributed data processing frameworks and tools, such as Spark or equivalent technologies.

• · Ability to work with structured, semi-structured, and unstructured data from multiple data sources.

• · Experience implementing data quality checks, validation frameworks, monitoring, and performance optimization practices.

• · Knowledge of data governance, lineage, metadata management, and cataloging concepts and tools.

• · Familiarity with data infrastructure requirements that support AI/ML workflows, including feature engineering, data preparation, and embedding pipelines.

• · Experience with real-time or streaming data technologies such as Kafka, Kinesis, or similar tools is preferred.

• · Familiarity with containerization and deployment tools such as Docker, Kubernetes, or similar technologies is preferred.

• · Ability to analyze technical requirements, solve complex data problems, and deliver practical solutions in a fast-paced environment.

• · Strong collaboration and communication skills, with the ability to work effectively across technical and business teams.

• · Ability to manage multiple priorities, meet deadlines, and maintain a high standard of quality and accuracy.

• · Working knowledge of the ABS Health, Safety, Quality, and Environmental Management System.

Reporting Relationships:

Reports directly to the Chief Data Scientist in Global Technology or other management or executive level position.

Working Conditions

Work is primarily sedentary; exerting up to 10 pounds of force occasionally and/or a negligible amount of force frequently or constantly to lift, carry, push, pull or otherwise move object.