We are looking for a Senior Big Data Infra Engineer skilled in developing and scaling data processing systems, with expertise in public cloud technologies and DevOps practices.
Summary Generated by Built In
About the Role:
We are seeking a highly skilled and experienced Senior Big Data Infra Engineer to join our dynamic team. The ideal candidate will have a strong background in developing and scaling both stream and batch processing systems, and a solid understanding of public cloud technologies. This role involves working in a remote environment, requiring excellent communication skills and the ability to solve complex problems independently and creatively.
What you will be doing:
Implementing automation/DevOps best practices for CI/CD, IaC, Containerization, etc to Build a reusable infra structure for stream and batch processing systems at scale.
Create automation, whether that is building DevOps pipelines, scripting or creating Infrastructure as Code in Terraform
Participating in work sessions with clients
Completing technical documentation
Requirements:
Experience in Developing and Scaling data Processing Systems This includes working with technologies like Pub/Sub, Kafka, Kinesis, DataFlow, Flink, Hadoop, Pig, Hive, and Spark.
Expertise in public cloud services, Azure/AWS or GCP.
Experience with Cloud managed services and understanding of cloud-based messaging/stream processing systems are critical.
Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
Has knowledge in containerization technologies such as Docker and Kubernetes to enhance the scalability and efficiency of applications.
Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals.
Proven experience in engineering stream/batch processing systems at scale.
Strong programming abilities in Java and Python.
Hands-on experience in public cloud platforms. Additional experience with other cloud technologies is advantageous.
Must Have:
Cloud Engineer Certification or other Cloud Professional level certification
4+ years of experience in customer-facing software/technology or consulting
4+ years of experience with “on-premises to cloud” migrations or IT transformations
4+ years of experience building, and operating solutions built on GCP or AWS/Azure
Technical degree: Computer Science, software engineering or related
