Principal Engineer, Data Platform (Databricks)
About the Role
We are building our next-generation data platform on Databricks. The Principal Engineer sets the technical direction for how the bank ingests, models, governs and serves data. This is a hands-on role: you will design the detailed lakehouse implementation, set engineering standards for SQL and pipelines, lead the migration from legacy platforms, and mentor engineers across several teams. You will work within the target-state architecture owned by the Solution Architect.
Key Responsibilities
- Lakehouse design: Own the detailed design of the Databricks lakehouse: ingestion, bronze/silver/gold layers, Delta Lake design, Unity Catalog setup, and workspace and cluster configuration.
- Migration: Lead the hands-on migration of legacy Hadoop/Hive/Spark and warehouse workloads to Databricks, including re-platforming HQL and SQL, validating data parity, and safe cutover.
- Engineering standards: Set standards for SQL, PySpark and Delta Live Tables/Lakeflow pipelines, covering code review, testing, CI/CD and observability.
- Performance and cost: Tune performance and cost: partitioning, liquid clustering/Z-ordering, file compaction, join and shuffle tuning, Photon, cluster right-sizing and DBU spend governance.
- Data modeling: Design canonical, auditable entity models (applications, accounts, customers, transactions) across source systems.
- Security and compliance: Implement PII protection, encryption and tokenization, row/column-level security, lineage, retention and audit readiness with Security, Risk, Compliance and Data Governance.
- Design reviews: Lead technical design reviews, write architecture decision records, and resolve cross-team technical trade-offs.
- Mentoring: Mentor senior and mid-level engineers and raise engineering quality across the data organization.
- Data products: Work with Analytics, Data Science/ML, Risk, Fraud and Finance to turn business needs into reliable, documented data products.
- Evaluation: Evaluate new Databricks capabilities and tools and recommend adoption where there is clear value.
Must-Have Skills
- Experience: 12+ years in data or software engineering, with at least 5 years in a technical lead, staff or principal role.
- SQL: Expert-level SQL: complex analytical queries, window functions, CTEs, query plan analysis and tuning on very large datasets (billions of rows).
- Spark and big data: Deep experience with Apache Spark (PySpark and/or Scala) and the ecosystem: Hadoop, Hive, Kafka, Parquet/ORC/Delta.
- Databricks in production: Hands-on production use of Delta Lake, Unity Catalog, Workflows/Jobs, Databricks SQL, and cluster and cost management.
- Pipelines: Proven design and operation of large-scale batch and streaming pipelines, including Structured Streaming, Delta Live Tables and Kafka, on a major cloud.
- Migration: Experience migrating on-premises or EMR-based Hadoop/Hive platforms to Databricks.
- Performance and cost: Proven Spark and Databricks performance tuning and cost optimization, including cluster policies.
- Data modeling: Dimensional, entity-centric, Data Vault or medallion modeling, including slowly changing dimensions, incremental/CDC loads and soft deletes.
- Orchestration and CI/CD: Airflow, Databricks Workflows or Oozie; Git, Terraform or Databricks Asset Bundles, and automated testing and data quality checks.
- Data security: Implementation of PII protection: encryption, tokenization, and row/column-level security.
- Leadership and communication: Mentoring engineers, and explaining technical trade-offs clearly to engineers and executives.
Good-to-Have Skills
- Banking, lending or other regulated financial services experience
- Data modeling for Marketing (campaign, attribution, funnel) and Collections (delinquency, contact, payments)
- Observability and monitoring frameworks for data pipelines
- Databricks Certified Data Engineer Professional
.
Responsibilities- Lakehouse design: Own the detailed design of the Databricks lakehouse: ingestion, bronze/silver/gold layers, Delta Lake design, Unity Catalog setup, and workspace and cluster configuration.
- Migration: Lead the hands-on migration of legacy Hadoop/Hive/Spark and warehouse workloads to Databricks, including re-platforming HQL and SQL, validating data parity, and safe cutover.
- Engineering standards: Set standards for SQL, PySpark and Delta Live Tables/Lakeflow pipelines, covering code review, testing, CI/CD and observability.
- Performance and cost: Tune performance and cost: partitioning, liquid clustering/Z-ordering, file compaction, join and shuffle tuning, Photon, cluster right-sizing and DBU spend governance.
- Data modeling: Design canonical, auditable entity models (applications, accounts, customers, transactions) across source systems.
- Security and compliance: Implement PII protection, encryption and tokenization, row/column-level security, lineage, retention and audit readiness with Security, Risk, Compliance and Data Governance.
- Design reviews: Lead technical design reviews, write architecture decision records, and resolve cross-team technical trade-offs.
- Mentoring: Mentor senior and mid-level engineers and raise engineering quality across the data organization.
- Data products: Work with Analytics, Data Science/ML, Risk, Fraud and Finance to turn business needs into reliable, documented data products.
- Evaluation: Evaluate new Databricks capabilities and tools and recommend adoption where there is clear value.
Graduate in Computer Science, Data Science, or a related field. 12+ years of experience in data modelling or data analysis.
Skills Required
- Graduate degree in Computer Science, Data Science, or a related field
- 12+ years of experience in data or software engineering, data modeling, or data analysis
- At least 5 years in a technical lead, staff, or principal role
- Expert-level SQL, including analytical queries, window functions, CTEs, query plan analysis, and tuning large datasets
- Deep Apache Spark experience with PySpark and/or Scala
- Experience with Hadoop, Hive, Kafka, Parquet, ORC, and Delta
- Production experience with Delta Lake, Unity Catalog, Databricks Workflows or Jobs, Databricks SQL, and cluster and cost management
- Experience designing and operating large-scale batch and streaming pipelines using Structured Streaming, Delta Live Tables, and Kafka on a major cloud platform
- Experience migrating on-premises or EMR-based Hadoop and Hive platforms to Databricks
- Experience with Spark and Databricks performance tuning, cost optimization, and cluster policies
- Experience with dimensional, entity-centric, Data Vault, or medallion data modeling, including slowly changing dimensions, incremental or CDC loads, and soft deletes
- Experience with Airflow, Databricks Workflows, or Oozie
- Experience with Git, Terraform, or Databricks Asset Bundles, plus automated testing and data quality checks
- Experience implementing PII protection, encryption, tokenization, and row- or column-level security
- Ability to mentor engineers and communicate technical trade-offs clearly to engineers and executives
- Banking, lending, or other regulated financial services experience
- Data modeling experience for marketing and collections domains
- Experience with observability and monitoring frameworks for data pipelines
- Databricks Certified Data Engineer Professional certification
EXL Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about EXL and has not been reviewed or approved by EXL.
-
Healthcare Strength — Medical plan options include HSA, HRA, and PPO choices, with additional health-related programs such as disability coverage and an employee assistance program. Feedback suggests these options provide meaningful flexibility and coverage breadth for many roles.
-
Leave & Time Off Breadth — Up to 20 days of paid time off, plus federal holidays and paid sick and safe time, are highlighted. Feedback suggests this leave structure is a notable benefit alongside core compensation.
-
Retirement Support — A traditional and Roth 401(k) with up to a 3% company match and full vesting after one year is described. Feedback suggests this match and vesting timeline help bolster long-term savings.
EXL Insights
What We Do
Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations. Whether your goals are scaling the use of AI and digital, redesign operating models, or driving better and faster decisions, we’re here to partner with you to help you gain—and maintain—competitive advantage with efficient, sustainable models at scale. Our expertise in transformation, data science, and change management helps make your business more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource- and time-intensive platform designs or migrations, we look deeper at your entire value chain to integrate strategies with impact. We use our specialization in analytics, digital interventions, and operations management—alongside deep industry expertise — to deliver solutions that help you outperform the competition. At EXL, it’s all about outcomes—your outcomes—and delivering success on your terms. Share your goals with us and together, we’ll optimize how you leverage data to drive your business forward. For more information, visit www.exlservice.com.
