Data Architect

Data Architect — Databricks

Data Engineering & Pipelines  |  Mid-Level  |  Full-Time

Experience

5 – 8 Years

Level

Mid-Level 

Employment Type

Full-Time

Location

Pune - Hybrid

Primary Stack

Databricks, Apache Spark, Delta Lake, SQL

Domain

Data Engineering & Pipelines

We are looking for a hands-on Data Architect with deep expertise in Databricks to design, build, and optimise enterprise-scale data platforms. You will own the end-to-end data engineering lifecycle — from ingestion and transformation to serving — while ensuring reliability, scalability, and governance across our lakehouse architecture.

You will collaborate closely with data engineers, analytics engineers, and product teams to translate business requirements into robust, reusable data solutions on the Databricks Lakehouse Platform.

Data Architecture & Design

•      Design and maintain the organisation's lakehouse architecture using Databricks and Delta Lake.

•      Define data modelling standards (dimensional, Data Vault 2.0, or medallion architecture) across Bronze, Silver, and Gold layers.

•      Architect scalable ingestion frameworks using structured and unstructured data sources (Kafka, JDBC, REST APIs, cloud storage).

•      Own schema evolution strategy and ensure backward-compatibility across data assets.

Pipeline Development & Optimisation

•      Build and maintain production-grade ETL/ELT pipelines using PySpark, Spark SQL, and Databricks Workflows.

•      Implement Delta Live Tables (DLT) for declarative, auto-scaling pipeline development.

•      Optimise Spark jobs for performance — partitioning, Z-ordering, caching, and cluster right-sizing.

•      Establish CI/CD practices for data pipelines using tools such as GitHub Actions, Azure DevOps, or Databricks Asset Bundles.

Data Governance & Quality

•      Implement Unity Catalog for data discovery, lineage tracking, fine-grained access control, and compliance.

•      Define and enforce data quality rules using Great Expectations, DLT expectations, or equivalent frameworks.

•      Work with data governance teams to document metadata, business glossary, and data contracts.

Platform & Infrastructure

•      Manage Databricks workspace configuration: clusters, pools, secrets, and access policies.

•      Collaborate with cloud and DevOps teams on infrastructure-as-code (Terraform) for Databricks on Azure / AWS / GCP.

•      Monitor platform health, SLAs, and cost using Databricks system tables and cloud-native monitoring tools.

Collaboration & Mentorship

•      Partner with data consumers (analysts, data scientists, ML engineers) to define SLAs and publish clean, well-documented data products.

•      Review code and provide architectural guidance to junior engineers.

•      Contribute to and champion internal data engineering best practices, runbooks, and documentation.

Core Databricks & Spark

•      4+ years of hands-on experience with Databricks (Unified Data Analytics Platform).

•      Strong proficiency in PySpark and Spark SQL for large-scale data transformation.

•      Deep knowledge of Delta Lake — ACID transactions, time travel, OPTIMIZE, VACUUM.

•      Experience with Databricks Workflows, Jobs, and Delta Live Tables (DLT).

•      Familiarity with Unity Catalog and Databricks governance features.

Data Engineering Fundamentals

•      Solid understanding of data modelling paradigms: dimensional modelling, Data Vault, or medallion architecture.

•      Experience designing and operating streaming pipelines (Structured Streaming, Kafka, Event Hubs, or Kinesis).

•      Proficiency in SQL; experience with dbt is a strong plus.

•      Hands-on experience with cloud platforms: Azure (ADLS, ADF), AWS (S3, Glue), or GCP (BigQuery, GCS).

Software Engineering Practices

•      Version control with Git; experience with branching strategies and code review workflows.

•      Ability to write testable, modular pipeline code with unit and integration tests.

•      Familiarity with CI/CD pipelines and infrastructure-as-code (Terraform preferred).

•      Databricks Certified Data Engineer Associate or Professional certification.

•      Experience with data mesh or data product frameworks.

•      Exposure to ML pipelines, MLflow, or Feature Store on Databricks.

•      Knowledge of data cataloguing tools (Alation, Collibra, or Databricks Unity Catalog).

•      Experience with Apache Iceberg or Apache Hudi as alternative table formats.

•      Familiarity with real-time analytics or OLAP systems (Druid, ClickHouse, Redshift).

•      Competitive salary with performance-linked bonus.

•      Flexible / hybrid working arrangements.

•      Access to Databricks training and certification budget.

•      Collaborative, engineering-first data culture with modern tooling.

•      Clear career progression path to Senior Data Architect or Data Platform Lead.

•      Comprehensive health, wellness, and retirement benefits.