Senior Software Engineer - Airflow

Business Area:

Seniority Level:

Job Description: 

At Cloudera, we empower people to transform complex data into clear and actionable insights. With as much data under management as the hyperscalers, we're the preferred data partner for the top companies in almost every industry.  Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world’s largest enterprises.

As a Senior Airflow Engineer at Cloudera, you will be a key architect and builder of the orchestration layer for the Cloudera Data Platform (CDP). This isn't just about writing DAGs; you will be optimizing the Airflow infrastructure that powers mission-critical data pipelines for the world's largest enterprises.

You will bridge the gap between core platform engineering and data workflow automation, ensuring scalability, security, and high availability across hybrid and multi-cloud environments.

• Engineer the Platform: Design, deploy, and manage large-scale Airflow environments on-prem and on Kubernetes (K8s) using YARN or Kubernetes executors.

• Develop Custom Providers: Build bespoke Airflow operators, hooks, and sensors to integrate seamlessly with CDP services like Ozone, Hive, and Impala.

• Optimize & Scale: Tune Airflow schedulers and workers to handle thousands of concurrent tasks with minimal latency.

• Secure the Workflow: Implement robust IAM roles, Kerberos integration, and secret management to meet rigorous enterprise security standards.

• Drive CI/CD & DevOps: Develop automated testing frameworks for DAGs and maintain Infrastructure-as-Code (IaC) for Airflow deployments.

• Mentor & Lead: Conduct high-impact code reviews and provide technical guidance on best practices for complex workflow orchestration.

• Deep Orchestration Expertise: Advanced knowledge of Apache Airflow, including XComs, Task Groups, and Dynamic Task Mapping.

• Expert Programming Skills: Expert-level Python and strong SQL skills (PostgreSQL/MySQL) for managing Airflow metadata.

• Cloud & Infra Foundations: Proven experience with Kubernetes (K8s), Docker, and familiarity with AWS, Azure, or GCP.

• Data Ecosystem Knowledge: Hands-on experience with Spark, Hive, Impala, or Kafka.

• DevOps Proficiency: Experience with Git, Jenkins/GitHub Actions, and Terraform.

• An "Owner's" Mindset: The autonomy and drive to make technical decisions that impact an entire global customer base.

• Open Source Contributions: Experience contributing directly to the Apache Airflow project.

• Legacy Migration Experience: A background in migrating complex legacy Oozie workflows to Airflow.

• Data Governance Tools: Knowledge of Data Quality frameworks (e.g., Great Expectations) and Data Lineage (e.g., OpenLineage).

What you can expect from us:

• Generous PTO Policy 

• Support work life balance with Unplugged Days

• Flexible WFH Policy 

• Mental & Physical Wellness programs 

• Phone and Internet Reimbursement program 

• Access to Continued Career Development 

• Comprehensive Benefits and Competitive Packages 

• Paid Volunteer Time

• Employee Resource Groups

EEO/VEVRAA

#LI-AB1

#LI-Hybrid