Build and operate the data pipelines that feed the Entity Hub. This role lands all six in-scope sources into Fabric, implements standardization and transformation logic, and maintains the data quality checks and monitoring that the entity resolution engine depends on. Reliable, observable ingestion is the foundation the entire programme rests on.
Responsibilities- Data Pipeline Development: Design, develop, and maintain scalable batch and streaming ETL pipelines using Databricks, PySpark, Python, and SQL.
- Lakehouse Implementation: Build and manage data solutions leveraging Delta Lake and medallion architecture principles to support trusted and reusable data products.
- Data Integration: Integrate data from enterprise applications, databases, APIs, files, and cloud platforms to enable unified analytics and reporting.
- Data Modeling & Transformation: Develop efficient data models and transformation frameworks that support business intelligence, advanced analytics, and AI/ML use cases.
- Platform Optimization: Monitor and optimize Databricks workloads, cluster configurations, and data processing performance to ensure scalability and cost efficiency.
- Data Quality & Governance: Implement data quality checks, monitoring, lineage, security controls, and governance practices in alignment with enterprise standards.
- DevOps & Automation: Contribute to CI/CD implementation, automated deployments, testing frameworks, and release management for data engineering solutions.
- Cross-functional Collaboration: Partner with business stakeholders, data architects, analytics teams, and AI engineers to deliver data products that support business outcomes.
- AI & Analytics Enablement: Enable downstream AI, machine learning, and Generative AI initiatives by building reliable, high-quality, and governed data foundations.
- Continuous Improvement & Innovation: Stay current with advancements in Databricks, cloud technologies, and modern data engineering practices to continuously enhance platform capabilities.
- Experience: 4 - 6 years of experience in data engineering, ETL development, and cloud-based data platforms.
- Technical Skills:
- Proficiency in Python, PySpark and SQL
- Strong expertise in Databricks Data Engineering capabilities, including: Databricks Lakehouse Architecture, ETL/ELT Design and Development, Delta Lake, Data Modeling, Batch and Streaming Data Pipelines, Spark Optimization and Performance Tuning
- Experience in designing and implementing: Medallion Architecture, Data Quality Frameworks, Data Governance Controls, Metadata and Lineage Solutions
- Experience with orchestration and workflow management tools such as Databricks Workflows, Azure Data Factory, or equivalent.
- Experience implementing CI/CD pipelines, Git-based source control, automated testing, and deployment automation.
- Strong understanding of data warehousing, dimensional modeling, and modern Lakehouse architectures.
- Exposure to Databricks Mosaic AI, Vector Search, MLflow, feature engineering, or GenAI data preparation workflows (Preferred).
- Familiarity with data preparation for RAG, LLM, and AI/ML use cases (Preferred).
- Experience with infrastructure-as-code and containerization technologies (Preferred).
- Certifications: Relevant certifications in data engineering, such as Databricks Certified Data Engineer Associate/Professional, or similar. (Preferred not mandatory)
- Problem-Solving: Strong analytical and problem-solving skills, with a proven ability to design scalable data solutions and resolve complex data engineering challenges.
- Communication: Excellent communication skills with experience working directly with clients and business stakeholders to deliver data and analytics solutions.
- Stakeholder Management: Experience engaging with business stakeholders to understand data and analytics requirements and translate them into scalable data engineering solutions.
Skills Required
- 4–6 years of experience in data engineering, ETL development, and cloud-based data platforms
- Proficiency in Python, PySpark, and SQL
- Expertise in Databricks Lakehouse Architecture, ETL/ELT development, Delta Lake, data modeling, batch and streaming pipelines, and Spark optimization
- Experience implementing medallion architecture, data quality frameworks, data governance controls, metadata, and data lineage solutions
- Experience with orchestration and workflow tools such as Databricks Workflows, Azure Data Factory, or equivalent
- Experience implementing CI/CD pipelines, Git-based source control, automated testing, and deployment automation
- Strong understanding of data warehousing, dimensional modeling, and modern lakehouse architectures
- Strong analytical and problem-solving skills
- Excellent communication skills and experience working with clients and business stakeholders
- Experience translating stakeholder data and analytics requirements into scalable data engineering solutions
- Exposure to Databricks Mosaic AI, Vector Search, MLflow, feature engineering, or generative AI data preparation workflows
- Familiarity with data preparation for RAG, LLM, and AI/ML use cases
- Experience with infrastructure-as-code and containerization technologies
- Databricks Certified Data Engineer Associate, Professional, or similar certification
EXL Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about EXL and has not been reviewed or approved by EXL.
-
Healthcare Strength — Medical plan options include HSA, HRA, and PPO choices, with additional health-related programs such as disability coverage and an employee assistance program. Feedback suggests these options provide meaningful flexibility and coverage breadth for many roles.
-
Leave & Time Off Breadth — Up to 20 days of paid time off, plus federal holidays and paid sick and safe time, are highlighted. Feedback suggests this leave structure is a notable benefit alongside core compensation.
-
Retirement Support — A traditional and Roth 401(k) with up to a 3% company match and full vesting after one year is described. Feedback suggests this match and vesting timeline help bolster long-term savings.
EXL Insights
What We Do
Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations. Whether your goals are scaling the use of AI and digital, redesign operating models, or driving better and faster decisions, we’re here to partner with you to help you gain—and maintain—competitive advantage with efficient, sustainable models at scale. Our expertise in transformation, data science, and change management helps make your business more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource- and time-intensive platform designs or migrations, we look deeper at your entire value chain to integrate strategies with impact. We use our specialization in analytics, digital interventions, and operations management—alongside deep industry expertise — to deliver solutions that help you outperform the competition. At EXL, it’s all about outcomes—your outcomes—and delivering success on your terms. Share your goals with us and together, we’ll optimize how you leverage data to drive your business forward. For more information, visit www.exlservice.com.
