The Role
Design, develop, optimize, and maintain scalable data pipelines using Databricks, PySpark, Delta Lake, SQL, and Azure services. Apply ETL, data modeling, governance, and DevOps practices while troubleshooting data issues, conducting root-cause analysis, and improving pipeline performance. Collaborate within an agile scrum team, independently deliver production-ready code, and contribute to architecture and design discussions.
Summary Generated by Built In
Key Responsibilities:
• Design, develop, and optimize data pipelines using Databricks, PySpark, Delta Lake, and related Azure components.
• Work across pipeline development, data modeling, and production code optimization, ensuring scalability and performance.
• Build new pipelines from scratch as well as enhance and maintain existing ones.
• Apply strong understanding of ETL design principles, data modeling concepts, and data governance standards.
• Collaborate within a scrum team, taking ownership of assigned stories while independently delivering high-quality, production-ready code.
• Demonstrate proficiency across PySpark, SQL, Delta Lake, Unity Catalog, and Databricks Workflows, with solid understanding of logic and data flow.
• Work in Azure environments, leveraging tools like ADLS, ADF, and Synapse (as applicable).
• Contribute inputs to architecture and design discussions where appropriate, while primarily focusing on hands-on development.
• Troubleshoot data issues, perform root-cause analysis, and optimize performance for existing pipelines.
Requirements
Desired Skills and Experience:
• 2-3 years of experience in data engineering, preferably with Azure Databricks.
• Strong technical expertise in PySpark, SQL, and Delta Lake.
• Familiarity with Unity Catalog, data governance, and DevOps practices (Git, CI/CD).
• Ability to work independently as well as collaboratively within an agile delivery team.
• Excellent problem-solving, debugging, and communication skills.
Skills Required
- 2-3 years of experience in data engineering, preferably with Azure Databricks
- Strong technical expertise in PySpark, SQL, and Delta Lake
- Familiarity with Unity Catalog, data governance, and DevOps practices including Git and CI/CD
- Ability to work independently and collaboratively within an agile delivery team
- Excellent problem-solving, debugging, and communication skills
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
ScatterPie Analytics is a data analytics and business intelligence company that modernizes enterprise decision-making. It brings together open-source technologies and enterprise tools to create powerful, cost-effective, and flexible intelligence solutions. The company delivers end-to-end analytics, including data engineering, cloud data platforms, business intelligence, AI, decision intelligence, and industry-focused analytics solutions, transforming raw data into actionable insights for evidence-based decisions across modern organizations.
