Data Engineer
Q2 is seeking a Data Engineer to join our fast-growing Data Products organization. As part of the Data Engineering team, you will build and maintain the infrastructure for developing and deploying data analytic products and core data transformation jobs. Our team collaborates closely with data scientists and other product engineering teams. Our goal is to enable Data Scientists to focus on creating data products through discovery and development of transformation jobs and machine-learning models, while ensuring their quick deployment to production. To achieve this, we created a scalable architecture for writing and executing ETL batch and Pyspark jobs which is currently supporting the execution of greater than 20,000 daily jobs. Additionally, we authored and maintain a real-time model serving and prediction service.
RESPONSIBILITIES:
Designs, implements, integrates, and tests complex infrastructure and software solutions.
Plans, leads, and coordinates development projects including activities of other team members.
Provides technical leadership and mentorship for other team members. Reviews and critiques designs and implementations.
Facilitates collaboration of working team and often engages with stakeholders and product owners. Ensures development commitments and objectives are achieved.
Communicates technical content effectively and drives technical discussions.
Typically requires a Bachelor’s degree in (relevant degree) and a minimum of 5 years of related experience; or an advanced degree with 3+ years of experience; or equivalent related work experience.
Knowledge in multiple areas of:
- data transformations, data pipelines, workflow automation, and scheduling systems
- software engineering and distributed software design
- data-centric system architecture
- database systems, data warehouses, distributed file storage and compute platforms
Experience with several of the technologies we currently use:
- Cloud Providers: Azure, AWS
- Containerization: Kubernetes, Docker
- Data Pipelines: Apache Airflow
- Data Tools: Pyspark, Pandas, Databricks, SageMaker, Athena
- Databases: SqlServer, Postgresql
- Languages: Python, Golang, Bash, SQL
- SCM: Git, Github, Gitlab
- CI/CD: Azure DevOps, Gitlab
At Q2, our goal is to be a diverse and inclusive workforce that fosters mutual respect for our employees and the communities we serve. Q2 is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status.