The Role
Design, build, and optimize end-to-end data ingestion and ETL pipelines on AWS into governed data lake zones. Implement data quality frameworks, error handling, and metadata/catalog management. Use IaC (Terraform/AWS CDK), write tests, support UAT, produce runbooks/documentation, and participate in Agile delivery.
Summary Generated by Built In
About OneByZero
OneByZero is a Frontier Systems Integrator building agentic AI systems for leading banks, telcos, insurers, and retailers across Asia Pacific. We work on mission-critical problems for organizations operating the region's financial and digital infrastructure. Operating exclusively on AWS, we combine deep technical expertise with strong partnership programs that enable innovation, scale, and delivery excellence. We build with people who are technically serious, outcome-focused, and passionate about solving complex enterprise challenges.
The Role
We are seeking a highly skilled Data Pipeline Engineer to design, build, and optimize scalable data ingestion and transformation pipelines on AWS. The ideal candidate will be responsible for developing end-to-end data pipelines, implementing data quality frameworks, and enabling reliable data delivery into a governed data lake environment.
This role requires strong expertise in AWS data services, ETL development, Infrastructure as Code (IaC), and modern data engineering best practices.
- Develop and implement end-to-end data ingestion pipelines from source systems through AWS S3, AWS Glue ETL, and AWS Lake Formation curated zones.
- Design and implement ETL transformations based on defined data mapping specifications.
- Build and maintain data quality validation frameworks to ensure completeness, schema conformance, and referential integrity.
- Configure error handling, exception management, and dead-letter queue patterns for failed ingestion records.
- Register and manage datasets within AWS Glue Data Catalog, ensuring standardized metadata tagging and governance compliance.
- Develop and maintain Infrastructure as Code (IaC) using Terraform or AWS CDK for data platform resources.
- Execute unit testing and integration testing to validate pipeline functionality and data accuracy.
- Support User Acceptance Testing (UAT) by validating output datasets, schemas, and record counts with business stakeholders.
- Create and maintain technical documentation including pipeline configurations, runbooks, and data flow diagrams.
- Participate in Agile ceremonies including sprint planning, stand-ups, demos, and code reviews.
RequirementsRequired Skills & Experience
- 5+ years of experience in Data Engineering or Data Platform Development.
- Strong hands-on experience with AWS Glue, AWS Lake Formation, Amazon S3, and Amazon Athena.
- Proficiency in Python, PySpark, and SQL.
- Experience building ETL/ELT pipelines in cloud-native environments.
- Hands-on experience with Infrastructure as Code (IaC) tools such as Terraform or AWS CDK.
- Strong understanding of data modeling, data governance, and metadata management.
- Experience with automated testing and deployment practices.
- Experience with Great Expectations or similar data quality frameworks.
- Exposure to SHIP-HATS CI/CD pipelines.
- Knowledge of enterprise data cataloging and governance solutions.
- Prior experience working with Government Agencies or Public Sector projects.
Education
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
Benefits
What We Offer
Competitive compensation benchmarked to market. The opportunity to work on AI transformation programmes that matter to the institutions millions of people in Asia Pacific depend on. A team that spans 10 markets, brings together some of the most experienced AI, data, and enterprise technology practitioners in the region, and moves fast without losing quality. Access to the AWS partnership programmes and frontier AI tooling that most firms in this region do not have. A firm that is building something genuinely new, with a leadership team that has scaled and delivered before.
OneByZero is an equal opportunity employer. We are committed to building a diverse and inclusive team and welcome applications from candidates of all backgrounds.
Why OneByZero
Agentic AI in regulated enterprise is the hardest version of this problem. The clients are the institutions that run the financial infrastructure of Asia Pacific. The constraints are real. The stakes are high.
OBZ is one of the few firms in the region doing this work at production scale in regulated industries, with an AWS partnership that puts you at the frontier of what is deployable. You will build systems that matter, work in squads small enough that your contribution is visible, and use AI coworkers as a genuine force multiplier.
If you are at the start of your career and want to be shaped by the hardest agentic engineering problems in the region, this is where that happens. If you are experienced and want to work on problems that are still genuinely unsolved, this is also where that happens.
Note - Only candidates with existing unrestricted work authorization in Singapore will be considered, as visa sponsorship is not available for this position
Skills Required
- 5+ years of experience in Data Engineering or Data Platform Development
- Hands-on experience with AWS Glue, AWS Lake Formation, Amazon S3, and Amazon Athena
- Proficiency in Python, PySpark, and SQL
- Experience building ETL/ELT pipelines in cloud-native environments
- Hands-on experience with Infrastructure as Code (Terraform or AWS CDK)
- Strong understanding of data modeling, data governance, and metadata management
- Experience with automated testing and deployment practices
- Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline
- Existing unrestricted work authorization in Singapore (visa sponsorship not available)
- Experience with Great Expectations or similar data quality frameworks
- Exposure to SHIP-HATS CI/CD pipelines
- Knowledge of enterprise data cataloging and governance solutions
- Prior experience working with Government Agencies or Public Sector projects
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
OneByZero is a consulting company specializing in artificial intelligence and data solutions, helping businesses transform and thrive in the digital age with AI-powered co-workers and data-driven consulting.
