Databricks Platform Engineering ● Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments. ● Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage. ● Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance. ● Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager). ● Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management. Delta Lake & Lakehouse Architecture ● Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM). ● Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization. ● Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations. ● Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing. ● Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS). EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com Data Pipeline Development (PySpark / SQL) ● Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake. ● Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables. ● Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning. ● Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity. ● Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines. MLflow & AI/ML Workloads ● Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks. ● Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances. ● Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features. ● Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI). ● Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints. Cloud Integration & DevOps ● Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS). ● Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI. ● Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources. ● Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte). ● Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools. Governance, Security & Optimization ● Implement row-level security, column masking, and dynamic data views using Unity Catalog policies. ● Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations. ● Conduct performance tuning — query plan analysis, caching strategies, Photon engine enablement. ● Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog. ● Document architecture decisions, runbooks, and operational guides for Databricks workloads.
ResponsibilitiesDatabricks Platform Engineering ● Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments. ● Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage. ● Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance. ● Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager). ● Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management. Delta Lake & Lakehouse Architecture ● Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM). ● Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization. ● Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations. ● Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing. ● Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS). EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com Data Pipeline Development (PySpark / SQL) ● Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake. ● Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables. ● Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning. ● Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity. ● Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines. MLflow & AI/ML Workloads ● Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks. ● Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances. ● Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features. ● Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI). ● Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints. Cloud Integration & DevOps ● Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS). ● Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI. ● Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources. ● Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte). ● Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools. Governance, Security & Optimization ● Implement row-level security, column masking, and dynamic data views using Unity Catalog policies. ● Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations. ● Conduct performance tuning — query plan analysis, caching strategies, Photon engine enablement. ● Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog. ● Document architecture decisions, runbooks, and operational guides for Databricks workloads.
QualificationsDatabricks Platform Engineering ● Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments. ● Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage. ● Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance. ● Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager). ● Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management. Delta Lake & Lakehouse Architecture ● Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM). ● Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization. ● Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations. ● Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing. ● Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS). EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com Data Pipeline Development (PySpark / SQL) ● Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake. ● Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables. ● Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning. ● Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity. ● Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines. MLflow & AI/ML Workloads ● Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks. ● Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances. ● Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features. ● Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI). ● Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints. Cloud Integration & DevOps ● Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS). ● Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI. ● Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources. ● Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte). ● Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools. Governance, Security & Optimization ● Implement row-level security, column masking, and dynamic data views using Unity Catalog policies. ● Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations. ● Conduct performance tuning — query plan analysis, caching strategies, Photon engine enablement. ● Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog. ● Document architecture decisions, runbooks, and operational guides for Databricks workloads.
About UsSkills Required
- Databricks platform engineering, including workspace, cluster, compute pool, job, and workflow management
- Unity Catalog governance, access control, permissions, data lineage, row-level security, and column masking
- Delta Lake and lakehouse architecture, including Medallion architecture, partitioning, Z-ordering, OPTIMIZE, and VACUUM
- Delta Live Tables pipelines and data quality expectations
- PySpark, Spark SQL, and Delta Lake pipeline development
- Batch and streaming data pipelines using Kafka, Event Hubs, Kinesis, ADLS, S3, or GCS
- MLflow, Databricks Feature Store, MLOps, model serving, and AI/ML workload support
- CI/CD for Databricks using Azure DevOps, GitHub Actions, or GitLab CI
- Infrastructure as code using Databricks Asset Bundles or Terraform
- Data ingestion using Auto Loader, COPY INTO, Fivetran, dbt, or Airbyte
- Databricks performance tuning, cost optimization, monitoring, and operational documentation
What We Do
Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations. Whether your goals are scaling the use of AI and digital, redesign operating models, or driving better and faster decisions, we’re here to partner with you to help you gain—and maintain—competitive advantage with efficient, sustainable models at scale. Our expertise in transformation, data science, and change management helps make your business more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource- and time-intensive platform designs or migrations, we look deeper at your entire value chain to integrate strategies with impact. We use our specialization in analytics, digital interventions, and operations management—alongside deep industry expertise — to deliver solutions that help you outperform the competition. At EXL, it’s all about outcomes—your outcomes—and delivering success on your terms. Share your goals with us and together, we’ll optimize how you leverage data to drive your business forward. For more information, visit www.exlservice.com.
.png){kind=logo}
