Lead Data Engineer
Job Summary:A Lead DS&E Engineer will create and guide the team in developing data and reporting solutions using batch-oriented and real-time streaming data pipelines inside AWS and other cloud environments. A person in this role will create new data workloads/applications. Implement, troubleshoots, and optimizes distributed solutions based on modern big data technologies like SQL, S3, Glue, MSK, Starburst Presto, Snowflake, Redshift. Solve large scale processing problems in a Cloud DW environment. Design syndicated and curated data views that will be used for reporting, analytics, and advanced applications. Experience with an agile / scrum environment. You will report to the Decision Science and Engineering manager
Responsibilities:
- Provide technical guidance and act as a point of escalation for projects
- Strong ability to mentor junior engineers
- Develop production grade syndicated and curated data views, assist with architecture decisions, deployment and feedback.
- Develop reusable patterns and frameworks as part of the engineering process
- Develop data pipelines in the AWS cloud environment
- Debug and troubleshoot across broad technical stack
Basic Qualifications:
- 5+ years of experience with Python development.
- 5+ years of experience with SQL.
- 5+ years of experience working in large Data Warehousing environments
- 2+ years of experience with AWS S3, Glue/Spark, MSK, Redshift, Snowflake, etc.
- 2+ years of experience in software development in an agile environment.
- Production experience with DevOps Tools like: Terraform, Docker, Jenkins.
- Queuing Technology – Kafka, RabbitMQ, Redis, SQS, Kinesis Streams, Kinesis Firehose
- Understanding of security at rest and in transit for big data pipelines.
- Experience with NoSQL databases, and ability to obtain insights from unstructured data formats.
- Strong systems and data analysis experience with testing in data space.
- Experience with architecture, design, data models, event based infrastructure, code repositories and implementation details
Required Education
- Bachelor's Degree in Computer Science, or related field or 10 years proven industry experience
Preferred Education
- Master’s Degree in Computer Science or related field