Lead Software Engineer - AWS/Airflow/DevOps

Sorry, this job was removed at 04:40 p.m. (CST) on Tuesday, Sep 10, 2024
2 Locations
Remote
Artificial Intelligence • Big Data • Cloud • Information Technology • Machine Learning
The Role

Egen is a fast-growing and entrepreneurial company with a data-first mindset. We bring together the best engineering talent working with the most advanced technology platforms, including Google Cloud and Salesforce, to help clients drive action and impact through data and insights. We are committed to being a place where the best people choose to work so they can apply their engineering and technology expertise to envision what is next for how data and platforms can change the world for the better. We are dedicated to learning, thrive on solving tough problems, and continually innovate to achieve fast, effective results.


As a Lead Engineer at Egen you will deliver a key component of a large modernization project by migrating legacy Control M (xml) jobs to run seamlessly through Airflow DAGs. You will design and scope delivery of the new MWAA (Managed Workflows for Apache Airflow) environment on AWS. The Staff Engineer will also be responsible for leading the onboarding and integration of multiple product teams onto the new Airflow platform. This role will also be responsible for creating Platform Run books and user guides as part of the product team onboard. This role will require significant hands-on work as you identify and implement custom Airflow operators with AWS.

Required Experience:

  • Airflow expert with experience in setting up and managing MWAA on AWS.
  • In-depth knowledge of orchestration concepts, including DAG (Directed Acyclic Graph) creation, task definition, operators, and best practices for workflow management.
  • Proficiency in Python programming, with a focus on developing Airflow DAGs and custom operators.
  • Strong understanding of Terraform, including the ability to write Terraform code for infrastructure as code (IaC) management.
  • Familiarity with AWS services such as S3, Lambda, CloudWatch, Fargate, VPC, security groups, and other related technologies.
  • Understanding of AWS Postgres and experience working with databases in the cloud environment.
  • Basic-level knowledge of Unix shell scripting for basic automation and maintenance tasks.

Nice to have's (but not required):

  • Experience with Python/Golang and cloud SDKs like AWS SDK, Kubernetes clients library, Helm
  • Experience with Mysql, PostgreSQL, Bitbucket, GitHub
  • Knowledge of service mesh frameworks like Istio, Linkerd, and Hashicorp Consul Connect
  • Knowledge of IAM and JWT authorization, SSO/OAuth (Azure AD, Okta)
  • Knowledgeable of 12-factor app methodology and how they should be applied 
The Company
HQ: Naperville, IL
240 Employees
Hybrid Workplace
Year Founded: 2000

What We Do

Egen is a data engineering and cloud modernization firm partnering with leading Chicagoland companies to launch, scale, and modernize industry-changing technologies. We are catalysts for change who create digital breakthroughs at warp speed. Our team of cloud and data engineering experts are trusted by top clients in pursuit of the extraordinary.

Our mission is to be an enabler of amazing possibilities for companies looking to use the power of cloud and data. We want to stand shoulder to shoulder with clients, as true technology partners, and make sure they succeed at what they have set out to do. We want to be disruptors, game-changers, and innovators who have played an important part in moving the world forward.

Similar Jobs

Remote
United States
630 Employees

Redflag AI Logo Redflag AI

Infrastructure Engineer

Artificial Intelligence • Machine Learning • Software
Remote
California, USA
20 Employees

Red Cell Partners Logo Red Cell Partners

Senior DevSecOps Engineer

Fintech • Payments • Financial Services
Remote
USA
64 Employees

DEFCON AI Logo DEFCON AI

Senior DevSecOps Engineer

Logistics • Transportation
Remote
USA
31 Employees

Similar Companies Hiring

Air Space Intelligence Thumbnail
Transportation • Software • Machine Learning • Logistics • Defense • Artificial Intelligence • Aerospace
Boston , Massachusetts
109 Employees
HERE Technologies Thumbnail
Software • Logistics • Internet of Things • Information Technology • Computer Vision • Automotive • Artificial Intelligence
Amsterdam, NL
6000 Employees
True Anomaly Thumbnail
Software • Machine Learning • Hardware • Defense • Artificial Intelligence • Aerospace
Colorado Springs, CO
131 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account