EXL Service is seeking an accomplished Senior Databricks Data Engineer with 10–12 years of experience designing, developing, and optimizing enterprise data platforms and large-scale ETL solutions. The ideal candidate brings deep expertise in Databricks, Spark, Delta Lake, Azure Data Platform, and modern data engineering practices.
This role is responsible for building scalable data pipelines, implementing cloud-native data solutions, improving platform performance, and delivering reliable data products for analytics, reporting, and AI/ML initiatives.
Base Compensation Range: 65,000 – 87,000
The posted range is the hiring range for this role — a subset of the broader range available to employees over time — and reflects base salary across our national hiring scale. Final offers are based on several factors, including the candidate's skills and experience, internal pay equity, work location, market conditions for the role, and the specific scope and responsibilities of the position. The top of the range is reserved for candidates who notably exceed the requirements; the lower end applies to those with less experience or fewer preferred qualifications. For positions based in higher-cost zones (e.g., California, New York, New Jersey), actual compensation may exceed the posted range; your recruiter will share specifics during the process.
For more information on benefits and what we offer please visit us at https://www.exlservice.com/us-careers-and-benefits
Responsibilities
Key Responsibilities
Data Engineering & Platform Development
- Design, develop, and optimize enterprise-scale data pipelines using Databricks, Spark, Delta Lake, and Azure Data Lake Storage.
- Build batch and incremental ETL/ELT pipelines, real-time streaming architectures, and reusable data integration frameworks.
- Develop scalable ingestion frameworks supporting structured, semi-structured, and API-based data sources.
- Build reusable data engineering frameworks for ingestion, transformation, validation, reconciliation, and publishing.
- Develop Delta Lake solutions using partitioning, optimization, Z-Ordering, Liquid Clustering, and performance tuning techniques.
- Implement CDC, incremental processing, merge strategies, and data synchronization across enterprise platforms.
- Develop Databricks Workflows, notebooks, SQL Jobs, and automation for production workloads.
- Integrate external REST APIs and process JSON/XML data into analytics-ready datasets.
- Implement robust data quality checks, audit frameworks, monitoring, and error handling.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
Databricks & Azure Development
- Develop solutions using Azure Data Lake Storage Gen2, Databricks, Unity Catalog, Azure Key Vault, Azure DevOps, and Azure Synapse.
- Build secure data pipelines using Unity Catalog, RBAC, service principals, and managed identities.
- Develop reusable notebook frameworks using PySpark and Spark SQL.
- Implement CI/CD deployment pipelines using Azure DevOps.
- Manage environment promotion across Development, QA, UAT, and Production.
- Troubleshoot production issues and optimize workloads for reliability and scalability.
Data Integration & Analytics
- Design enterprise data models supporting reporting, analytics, and downstream applications.
- Develop healthcare and financial data integration pipelines supporting multiple source systems.
- Build reusable metadata-driven ETL frameworks.
- Integrate third-party APIs including NLP, terminology normalization, and identity resolution services.
- Support data governance, lineage, and metadata management initiatives.
Qualifications
Required Skills & Experience
Technical Expertise
- 10-12 years of experience in Data Engineering, ETL Development, and Data Warehousing.
- Hands-on Databricks development experience.
- Strong experience with Spark, Delta Lake, Unity Catalog, Databricks Workflows, and SQL Warehouses.
- Strong experience with PySpark, Spark SQL, SQL, and Python.
- Experience building enterprise ETL/ELT pipelines using Databricks and Azure Data Platform.
- Experience with Azure Data Lake Storage (ADLS Gen2), Azure Synapse, Azure Key Vault, and Azure DevOps.
- Experience implementing CDC, SCD, incremental loading, and data quality frameworks.
- Experience integrating REST APIs and processing JSON/XML datasets.
- Experience with Git, CI/CD, release management, and deployment automation.
- Strong knowledge of performance tuning, partitioning, caching, broadcast joins, and Spark optimization.
- Experience working with healthcare data platforms is preferred.
- Microsoft Azure certifications (e.g., DP-203 Azure Data Engineer Associate) or Databricks certifications (Data Engineer Associate/Professional).
- Experience with Ab Initio suite of products is also preferred.
Skills Required
- 10-12 years of experience in Data Engineering, ETL Development, and Data Warehousing.
- Hands-on Databricks development experience.
- Strong experience with Spark, Delta Lake, Unity Catalog, Databricks Workflows, and SQL Warehouses.
- Strong experience with PySpark, Spark SQL, SQL, and Python.
- Experience building enterprise ETL/ELT pipelines using Databricks and Azure Data Platform.
- Experience with Azure Data Lake Storage Gen2, Azure Synapse, Azure Key Vault, and Azure DevOps.
- Experience implementing CDC, SCD, incremental loading, and data quality frameworks.
- Experience integrating REST APIs and processing JSON/XML datasets.
- Experience with Git, CI/CD, release management, and deployment automation.
- Strong knowledge of performance tuning, partitioning, caching, broadcast joins, and Spark optimization.
- Experience working with healthcare data platforms.
- Microsoft Azure or Databricks data engineering certifications (e.g., DP-203, Databricks Data Engineer Associate/Professional).
- Experience with Ab Initio suite of products.
What We Do
Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations. Whether your goals are scaling the use of AI and digital, redesign operating models, or driving better and faster decisions, we’re here to partner with you to help you gain—and maintain—competitive advantage with efficient, sustainable models at scale. Our expertise in transformation, data science, and change management helps make your business more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource- and time-intensive platform designs or migrations, we look deeper at your entire value chain to integrate strategies with impact. We use our specialization in analytics, digital interventions, and operations management—alongside deep industry expertise — to deliver solutions that help you outperform the competition. At EXL, it’s all about outcomes—your outcomes—and delivering success on your terms. Share your goals with us and together, we’ll optimize how you leverage data to drive your business forward. For more information, visit www.exlservice.com.
