The Role
Designs, develops, and optimizes scalable batch and near-real-time data pipelines and platforms using Snowflake, AWS, Python, DataStage, and Control-M. Builds data models, ensures data quality and governance, tunes performance and costs, supports production operations, and enables analytics and AI/ML use cases. Collaborates with engineering, data science, BI, platform, and business teams in Agile environments.
Summary Generated by Built In
We are unable to sponsor or take over sponsorship of an employment visa at this time.
Job Title: Senior Data Engineer
Role Summary
We are looking for a Senior Data Engineer to design, build, and optimize scalable data pipelines and data platforms supporting analytics, reporting, and AI/ML use cases. The ideal candidate has strong hands-on experience with Snowflake on AWS, Python-based ETL/ELT development, and enterprise scheduling/orchestration tools like Control-M, along with legacy/enterprise ETL experience in IBM DataStage. You will collaborate across engineering, analytics, and business teams in an Agile delivery model.
Key Responsibilities
- Design, develop, and maintain end-to-end data pipelines (batch and near real-time) using Snowflake, AWS services, and Python.
- Build and optimize data models in Snowflake (e.g., dimensional modeling, data vault, or curated data marts) for analytics and downstream consumption.
- Develop and maintain ETL/ELT workflows using Python and IBM DataStage; migrate/modernize workloads where applicable.
- Implement job scheduling, monitoring, and operational support using Control-M (alerting, retries, SLAs, and dependency management).
- Ensure data quality, governance, lineage, and documentation standards are met across pipelines.
- Perform performance tuning and cost optimization across Snowflake and AWS (query optimization, clustering, warehouse sizing, storage management).
- Partner with stakeholders (Data Science/AI, BI, Product, and Platform teams) to enable data products and AI-ready datasets.
- Participate in Agile ceremonies, contribute to estimation, planning, and sprint execution; follow SDLC and change management processes.
- Troubleshoot production issues, perform root-cause analysis, and drive preventative improvements.
Required Technical Skills
- Snowflake: Strong expertise in Snowflake architecture, SQL development, performance tuning, security/roles, data loading/unloading, and best practices.
- AWS: Hands-on experience with AWS data ecosystem (commonly S3, IAM, CloudWatch; plus services such as Glue, Lambda, EC2, Step Functions, EMR, or Kinesis as applicable).
- Python: Strong Python programming for data engineering (ETL/ELT frameworks, API ingestion, automation, unit testing, logging).
- Control-M: Experience designing and managing enterprise job scheduling, dependencies, calendars, SLAs, monitoring, and incident handling.
- IBM DataStage: Solid experience building and maintaining DataStage jobs, handling complex transformations, and supporting production workloads.
- SQL: Advanced SQL skills for transformations, optimization, and data validation across large datasets.
- CI/CD & Version Control: Experience with Git and CI/CD practices for data pipelines (tools may vary).
- Operational Excellence: Monitoring, alerting, and production support experience in a 24x7 or business-critical environment.
Good to Have
- AI/ML exposure: Experience enabling AI/ML pipelines or feature datasets; familiarity with ML lifecycle concepts, feature engineering, or MLOps tools/processes.
- Experience with data governance/metadata tools and practices (catalog, lineage, data quality frameworks).
- Exposure to streaming or event-driven architectures.
Required Soft Skills
- Strong experience working in Agile/Scrum teams and delivering within structured SDLC processes.
- Excellent communication skills (technical and non-technical) with the ability to explain complex data concepts clearly.
- Proven ability to coordinate across multiple teams (Data Engineering, Data Science, DevOps, Security, BI, and business stakeholders).
- Strong ownership mindset, problem-solving ability, and attention to detail.
Qualifications (Typical)
- Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent practical experience).
- 9+ years of data engineering experience, including enterprise-grade data platform delivery and production support.
Kaleidoscope, an Infosys Company, is an equal opportunity employer, and all qualified applicants will receive consideration without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, spouse of protected veteran, or disability.
Skills Required
- Strong expertise with Snowflake architecture, SQL development, performance tuning, security, data loading, and unloading
- Hands-on experience with AWS data services, including S3, IAM, and CloudWatch
- Strong Python programming experience for data engineering, ETL/ELT, automation, testing, and logging
- Experience designing and managing Control-M job schedules, dependencies, SLAs, monitoring, and incident handling
- Solid experience building and maintaining IBM DataStage jobs and complex transformations
- Advanced SQL skills for data transformations, optimization, and validation
- Experience with Git and CI/CD practices for data pipelines
- Monitoring, alerting, and production support experience in a 24x7 or business-critical environment
- Experience working in Agile/Scrum teams and structured SDLC processes
- Excellent technical and non-technical communication skills
- Ability to coordinate across data engineering, data science, DevOps, security, BI, and business teams
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience
- 9+ years of data engineering experience, including enterprise data platform delivery and production support
- Experience enabling AI/ML pipelines or feature datasets and familiarity with ML lifecycle concepts or MLOps
- Experience with data governance and metadata tools and practices
- Exposure to streaming or event-driven architectures
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
When clients come to us for product design and development, they get a full range of technical expertise and laboratory resources, but they also get a team that’s relentless when it comes to solving problems and creating designs that are the ideal combination of function and form. What does that mean? We build tools that save lives. For surgeons, it means they have tools that not only address but anticipate their needs, which helps improve their patients’ outcomes. We create products that save money and ensure safety. For people juggling family, work, and other demands, it means they have products that provide real help in managing their everyday lives. And for businesses, it means their bottom line is supported with durable tools that improve efficiency and protect their teams. From concept to production plan, or at any phase in between, we are a partner that can extend your product development know-how or provide additional mindpower to free up in-house teams.
