We are looking for an accomplished Lead Data Engineer to drive the architectural design, development, and optimization of our enterprise-scale data ecosystem. In this senior role, you will spearhead the build-out of high‑performance data pipelines, enforce rigorous data quality and validation frameworks, and ensure end‑to‑end reliability, scalability, and integrity of data flows across the organization. You will play a pivotal role in shaping our data engineering strategy, enabling advanced analytics and mission‑critical, data‑driven decision-making within our insurance-focused business domains.
Responsibilities- Lead the design and implementation of scalable data architectures on Azure, leveraging Databricks, Delta Lake, and related Azure data services.
- Build and optimize high‑volume ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, and Delta Live Tables.
- Define and enforce best practices for data engineering across notebooks, jobs, CI/CD, Unity Catalog, security, and workspace governance.
- Integrate and orchestrate data pipelines using Azure Data Factory, Azure Synapse pipelines, or Azure Databricks Workflows.
- Drive performance tuning for Spark jobs, cluster configurations, and data storage layers to balance speed and cost efficiency.
- Implement robust data quality and validation frameworks using tools like Great Expectations, DLT expectations, or custom PySpark checks.
- Ensure compliance, governance, and lineage tracking through Unity Catalog, Purview integration, and RBAC/ABAC policies.
- Architect end‑to‑end data solutions supporting analytics, ML, actuarial models, and business-critical reporting workloads.
- Evaluate new Databricks features, MLflow enhancements, Photon execution, serverless compute, and recommend adoption strategies.
- Familiar with working on Agile methodologies - scrum, sprint planning, backlog refinement etc.
- 8-12 years experience on Data Engineering role working with Databricks & Azure Cloud technologies.
- Bachelor’s degree in computer science, Information Technology, or related field.
- Strong proficiency in PySpark, Python, SQL.
- Strong experience in data modeling, ETL/ELT pipeline development, and automation
- Hands-on experience with performance tuning of data pipelines and workflows
- Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc.
- Experience with data modeling, ETL processes, Delta Lake and data warehousing.
- Experience on Delta Live Tables, Autoloader & Unity Catalog.
- Preferred - Knowledge of the insurance industry and its data requirements.
- Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
- Excellent communication and problem-solving skills to work effectively with diverse teams
- Excellent problem-solving skills and ability to work under tight deadlines.
Skills Required
- 8-12 years experience in Data Engineering working with Databricks and Azure Cloud technologies
- Bachelor's degree in Computer Science, Information Technology, or related field
- Strong proficiency in PySpark, Python, and SQL
- Experience designing and implementing scalable data architectures on Azure and Databricks
- Experience building and optimizing ETL/ELT pipelines, data modeling, and automation
- Hands-on experience with performance tuning of Spark jobs, cluster configuration, and data storage
- Proficient with Azure Data Factory, Azure Synapse pipelines, Azure Data Lake, and Azure Databricks
- Experience with Delta Lake, Delta Live Tables, Autoloader, and Databricks Workflows
- Experience implementing data quality and validation frameworks (e.g., Great Expectations, DLT expectations, custom PySpark checks)
- Experience with governance, lineage, Unity Catalog, Purview integration, and RBAC/ABAC policies
- Familiarity with Agile methodologies (Scrum, sprint planning, backlog refinement)
- Excellent communication, analytical, and problem-solving skills
- Knowledge of the insurance industry and its data requirements
What We Do
Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations. Whether your goals are scaling the use of AI and digital, redesign operating models, or driving better and faster decisions, we’re here to partner with you to help you gain—and maintain—competitive advantage with efficient, sustainable models at scale. Our expertise in transformation, data science, and change management helps make your business more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource- and time-intensive platform designs or migrations, we look deeper at your entire value chain to integrate strategies with impact. We use our specialization in analytics, digital interventions, and operations management—alongside deep industry expertise — to deliver solutions that help you outperform the competition. At EXL, it’s all about outcomes—your outcomes—and delivering success on your terms. Share your goals with us and together, we’ll optimize how you leverage data to drive your business forward. For more information, visit www.exlservice.com.
