Tech Lead – Databricks / Data Platform

Role: Tech Lead – Databricks / Data Platform
Location: Remote
Employment Type: Full-Time
Experience: 10–12 Years

JOB SUMMARY

We are looking for an experienced Tech Lead – Databricks / Data Platform to lead the design, development, and optimization of a scalable, secure, and reliable cloud-based data platform. The role will provide technical leadership across Databricks, Azure, CI/CD, Infrastructure as Code, orchestration, data security, FinOps, and platform automation.

The ideal candidate will be hands-on, technically strong, and experienced in leading engineering teams while partnering closely with Data Engineers, Analytics Engineers, Architects, Security, and Cloud teams to deliver enterprise-grade data solutions.

KEY RESPONSIBILITIES

• Lead the architecture, design, and implementation of Databricks-based data platforms and cloud-native data pipelines.
  
• Design and implement robust CI/CD pipelines for data ingestion, transformation, dbt, SQL, notebooks, and Databricks workloads.
  
• Lead end-to-end orchestration of Databricks Jobs and Workflows, including ingestion, transformation, data quality checks, and dependencies.
  
• Define and implement automated testing and quality gates within CI/CD pipelines, including schema, data model, and contract validation.
  
• Drive Infrastructure as Code (IaC) practices using Terraform and reusable modules for Databricks and Azure infrastructure.
  
• Automate Databricks platform operations, including workspace and cluster provisioning, runtime and library management, job deployment, configuration, and environment management.
  
• Establish and maintain Identity and Access Management (IAM) across Databricks and Azure, including RBAC, TBAC, service principals, groups, roles, and workspace/cluster/table-level access controls.
  
• Work closely with the EDP Architect and Security teams to implement secure and scalable Unity Catalog and data governance patterns.
  
• Develop reusable Terraform modules, Databricks job templates, Airflow DAG patterns, and engineering frameworks to accelerate onboarding of new projects.
  
• Define and govern the code promotion and release management process across development, QA, and production environments.
  
• Lead end-to-end orchestration using managed Airflow, ensuring reliable scheduling, dependency management, monitoring, and recovery.
  
• Define and track platform SLAs, SLOs, and SLIs, and lead incident triage, root cause analysis, and corrective actions.
  
• Implement FinOps best practices for monitoring, optimizing, and allocating Databricks and cloud infrastructure costs.
  
• Continuously evaluate and improve platform tooling, architecture, reliability, security, performance, and developer productivity.
  
• Provide technical guidance and mentorship to engineers and establish engineering best practices, standards, and reusable patterns.
  
• Partner with Data Engineering, Analytics, Architecture, Security, and Infrastructure teams to ensure seamless delivery of enterprise data solutions.
  
• Contribute to cloud infrastructure, cybersecurity, disaster recovery, monitoring, logging, and operational readiness.
  
• Drive performance optimization and reliability improvements for production data workloads.
  

REQUIRED QUALIFICATIONS

• 10–12 years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE, or related technical disciplines.
  
• Strong hands-on experience with Databricks in production environments, including workspace and cluster management, Jobs/Workflows, Unity Catalog, and integration with orchestration platforms.
  
• Strong experience with Microsoft Azure and cloud-native data platforms.
  
• Strong knowledge of CI/CD and Git-based development workflows, using tools such as Azure DevOps, GitHub Actions, GitLab CI, or similar.
  
• Strong experience with Terraform / Infrastructure as Code and cloud infrastructure automation.
  
• Hands-on experience with Apache Airflow or managed Airflow for data pipeline orchestration.
  
• Strong programming/scripting skills in Python, Bash, or PowerShell.
  
• Experience implementing automated testing, validation, and quality gates for data pipelines and data models.
  
• Experience with production workload management, including monitoring, logging, troubleshooting, performance tuning, and incident management.
  
• Strong understanding of cloud security, IAM, RBAC, data access controls, and governance.
  
• Experience with FinOps, cloud cost optimization, and resource utilization monitoring.
  
• Ability to provide technical leadership, mentor engineers, and drive engineering standards across teams.
  
• Strong communication and stakeholder management skills with the ability to work effectively with technical and business stakeholders.
  

PREFERRED QUALIFICATIONS

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent professional experience.
  
• Experience with dbt, SQL, Delta Lake, and modern data engineering practices.
  
• Experience with Unity Catalog and enterprise data governance.
  
• Experience designing reusable platform frameworks and accelerators.
  
• Experience in CPG, retail, manufacturing, or distribution environments.
  
• Experience with disaster recovery, business continuity, and highly available cloud architectures.
  
• Databricks or Azure certifications are a plus.