The Role
Design, develop, optimize, and maintain scalable data pipelines using Databricks, PySpark, Delta Lake, SQL, and Azure services. Responsibilities include data modeling, ETL development, governance, troubleshooting, root-cause analysis, production-code optimization, and performance improvement. The role collaborates within an agile Scrum team, independently delivers production-ready code, contributes to architecture discussions, and uses DevOps practices such as Git and CI/CD.
Summary Generated by Built In
Key Responsibilities:
• Design, develop, and optimize data pipelines using Databricks, PySpark, Delta Lake, and related Azure components.
• Work across pipeline development, data modeling, and production code optimization, ensuring scalability and performance.
• Build new pipelines from scratch as well as enhance and maintain existing ones.
• Apply strong understanding of ETL design principles, data modeling concepts, and data governance standards.
• Collaborate within a scrum team, taking ownership of assigned stories while independently delivering high-quality, production-ready code.
• Demonstrate proficiency across PySpark, SQL, Delta Lake, Unity Catalog, and Databricks Workflows, with solid understanding of logic and data flow.
• Work in Azure environments, leveraging tools like ADLS, ADF, and Synapse (as applicable).
• Contribute inputs to architecture and design discussions where appropriate, while primarily focusing on hands-on development.
• Troubleshoot data issues, perform root-cause analysis, and optimize performance for existing pipelines.
Requirements
Desired Skills and Experience:
• 3–5 years of experience in data engineering, preferably with Azure Databricks.
• Strong technical expertise in PySpark, SQL, and Delta Lake.
• Familiarity with Unity Catalog, data governance, and DevOps practices (Git, CI/CD).
• Ability to work independently as well as collaboratively within an agile delivery team.
• Excellent problem-solving, debugging, and communication skills.
Skills Required
- 3-5 years of experience in data engineering, preferably with Azure Databricks
- Strong technical expertise in PySpark, SQL, and Delta Lake
- Familiarity with Unity Catalog, data governance, and DevOps practices including Git and CI/CD
- Ability to work independently and collaboratively within an agile delivery team
- Excellent problem-solving, debugging, and communication skills
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
ScatterPie Analytics is a data analytics and business intelligence company that modernizes enterprise decision-making. It brings together open-source technologies and enterprise tools to create powerful, cost-effective, and flexible intelligence solutions. The company delivers end-to-end analytics, including data engineering, cloud data platforms, business intelligence, AI, decision intelligence, and industry-focused analytics solutions, transforming raw data into actionable insights for evidence-based decisions across modern organizations.
