Location: Hyderabad / Mumbai / Bangalore
Education: Bachelor's Degree
Experience: 5+ Years
We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, Microsoft Fabric, Data Engineering, Automation, and Data Quality Engineering. The ideal candidate will play a key role in designing, building, automating, and governing enterprise data platforms while ensuring the reliability, accuracy, and security of analytics solutions.
The role will support strategic data modernization initiatives across multiple clients and industry domains, focusing on scalable data pipelines, data quality frameworks, automation, and analytics platform engineering.
Key Responsibilities- Design, develop, and maintain scalable data solutions using Databricks, Microsoft Fabric, SQL, Spark, and Delta Lake.
- Build and optimize data ingestion, transformation, and orchestration pipelines for enterprise data platforms.
- Develop and implement automation frameworks for data validation, testing, monitoring, deployment, and operational support.
- Lead data engineering activities across the full lifecycle, including requirements, design, development, testing, deployment, and support.
- Build and maintain automated QA and reconciliation frameworks for analytical and reporting solutions.
- Validate data quality, completeness, consistency, and business rule compliance across enterprise datasets.
- Develop and maintain semantic models, lakehouses, warehouses, and data marts within Microsoft Fabric.
- Implement and validate security controls, including Row-Level Security (RLS) and data governance standards.
- Collaborate with business stakeholders, architects, analytics teams, and platform engineers to deliver enterprise-grade solutions.
- Support CI/CD, DevOps, Infrastructure-as-Code, and deployment automation initiatives.
- Drive performance optimization and cost efficiency across Databricks and Fabric environments.
- Establish engineering best practices, coding standards, reusable frameworks, and governance processes.
- 5+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Development.
- Strong hands-on expertise in:
- Databricks
- Apache Spark (PySpark)
- SQL
- Microsoft Fabric
- Delta Lake
- Data Warehousing
- Experience designing and implementing:
- Data Pipelines
- ETL/ELT Solutions
- Lakehouse Architectures
- Data Integration Frameworks
- Strong experience with Databricks Workflows, Jobs, Automation, and Orchestration.
- Experience developing automated testing and validation frameworks for data platforms.
- Hands-on experience implementing CI/CD pipelines using Azure DevOps, GitHub, or equivalent tools.
- Strong understanding of data quality, reconciliation, metadata management, and governance practices.
- Experience with performance tuning and optimization of Spark workloads.
- Experience working with Microsoft Fabric components, including:
- Data Factory
- Lakehouse
- Warehouse
- Semantic Models
- Power BI
- Experience with Infrastructure as Code (Terraform, Bicep, ARM Templates).
- Experience with DataOps and MLOps processes.
- Knowledge of Power BI semantic models, DAX, and analytical reporting solutions.
- Experience with enterprise data governance and security frameworks.
- Familiarity with cloud platforms such as Azure, AWS, or GCP.
- Experience leading technical discussions, mentoring engineers, and driving engineering best practices.
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Ability to lead initiatives independently and work effectively in cross-functional teams.
- Strategic mindset with the ability to align technical solutions with business objectives.
Skills Required
- Bachelor's degree
- 5+ years of experience in data engineering, analytics engineering, or data platform development
- Hands-on expertise with Databricks
- Hands-on expertise with Apache Spark and PySpark
- Hands-on expertise with SQL
- Hands-on expertise with Microsoft Fabric
- Hands-on expertise with Delta Lake
- Experience with data warehousing
- Experience designing data pipelines, ETL/ELT solutions, lakehouse architectures, and data integration frameworks
- Experience with Databricks Workflows, Jobs, automation, and orchestration
- Experience developing automated testing and validation frameworks for data platforms
- Experience implementing CI/CD pipelines using Azure DevOps, GitHub, or equivalent tools
- Understanding of data quality, reconciliation, metadata management, and governance practices
- Experience with Spark performance tuning and optimization
- Experience with Microsoft Fabric Data Factory, Lakehouse, Warehouse, Semantic Models, and Power BI
- Experience with Infrastructure as Code using Terraform, Bicep, or ARM Templates
- Experience with DataOps and MLOps processes
- Knowledge of Power BI semantic models, DAX, and analytical reporting solutions
- Experience with enterprise data governance and security frameworks
- Familiarity with Azure, AWS, or GCP
- Experience leading technical discussions, mentoring engineers, and driving engineering best practices
What We Do
A premiere data services company serving clients in North America, Datavail has 1,000 data professionals, data engineers, developers, project managers, consultants, and business experts, supported by industry-leading automation and intellectual property. For more than 17 years, Datavail has worked with thousands of companies spanning different industries and sizes. At Datavail, we look for more than smarts, experience and proficiency. On top of those requirements, we seek people who mesh with our corporate values. We seek brilliance without bravado and know-how without a know-it-all attitude. We hold low ego in high regard, embrace problem-solving as a passion and welcome every day as a new opportunity to learn. We’re flexible and hard working. We’re committed to our clients and colleagues. We help our people grow so they can help our clients grow. That makes us grow so we can help even more customers leverage organizational data for business value. Our Core Values: 1. We desire to serve. 2. We embody flexibility for availability 3. We exemplify low ego. 4. We work hard. 5. We strive for continuous improvement. 6. We are growth-oriented.
