The Role
Design and develop a big-data processing engine and heavy ETL pipelines (including geo-location analysis) across cloud and on-prem. Build distributed, partitioned pipeline architectures using Spark/PySpark, Scala, Airflow, Kubernetes, and cloud platforms; follow Agile, clean-code, and software design best practices while supporting deployments and client-facing interactions.
Summary Generated by Built In
Description
We are looking for a Big Data / Data Engineer to join our growing team. In this role, you will be responsible for the design and development of our big data processing engine a heavy ETL pipeline handling complex logic including geo-location analysis, running across both cloud and on-prem environments.
Requirements:
- 4+ years of experience in Data Engineering
- 2+ years of hands-on experience with Scala (must) or Java (optional)
- 2+ years of experience with SQL and NoSQL DBs
- Hands-on experience with Spark & PySpark (must)
- Experience with Airflow (must)
- Experience with AWS and/or GCP
- Experience with Kubernetes (k8s) and on-prem deployments
- Solid understanding of distributed systems design, pipeline architecture, and partitioning strategies
- Deep understanding of Agile principles, practices and values
- Solid understanding of design patterns, software development techniques, and clean code practices
Advantage:
- Experience with Apache Iceberg, Helm Charts, or data catalog tools
- Experience with Spring Framework
- Experience with microservice architecture
- Experience working with DevOps and CI/CD practices
- Customer-facing experience or comfort working directly with clients
- Bachelor's, BSc or Master's degree in Computer Science or Engineering
Skills Required
- 4+ years of experience in Data Engineering
- 2+ years hands-on experience with Scala
- 2+ years hands-on experience with Java
- 2+ years of experience with SQL and NoSQL databases
- Hands-on experience with Spark and PySpark
- Experience with Airflow
- Experience with AWS and/or GCP
- Experience with Kubernetes and on-prem deployments
- Solid understanding of distributed systems design, pipeline architecture, and partitioning strategies
- Deep understanding of Agile principles, practices and values
- Solid understanding of design patterns, software development techniques, and clean code practices
- Experience with Apache Iceberg
- Experience with Helm Charts
- Experience with data catalog tools
- Experience with Spring Framework
- Experience with microservice architecture
- Experience working with DevOps and CI/CD practices
- Customer-facing experience or comfort working directly with clients
- Bachelor's, BSc or Master's degree in Computer Science or Engineering
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
