Senior Data Engineer -Databricks

Job Responsibilities:

Design, develop, and maintain scalable ETL/ELT pipelines using Databricks,
PySpark, and SQL.
● Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
● Build data pipelines with Databricks Unity Catalog.
● Implement business logic, data transformations, and dimensional data models.
● Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
● Design and manage Delta Lake tables using Medallion Architecture (Bronze, Silver,Gold).
● Ensure data quality through validations, error handling, logging, and monitoring.
● Optimize Spark workloads for performance, scalability, and reliability.
● Collaborate with cross-functional teams to deliver production-ready data solutions.

Basic Qualification:

Strong expertise in Python, PySpark, and Advanced SQL.
● Hands-on experience with the Databricks Lakehouse Platform.
● Good understanding of Unity Catalog, Delta Lake, Databricks Workflows/Jobs,
Clusters, Notebooks, Repos, and Medallion Architecture.
● Experience integrating with REST APIs for data ingestion and data export.
● Strong knowledge of ETL/ELT development, batch processing, incremental loading, and data transformation.
Data Governance experience would be an added advantage.
● Experience with data modeling (Star Schema, Snowflake Schema, Fact & Dimension tables, SCD concepts).
● Understanding of data warehousing concepts and best practices.
● Experience working with structured and semi-structured data (CSV, JSON, Parquet, Delta).
● Knowledge of partitioning, file optimization, Spark performance tuning, and query optimization.
● Experience with Git and CI/CD best practices

Preferred Qualifications:

4+ years of experience in Data Engineering with 2+ years of hands-on Databricks
experience. Data Governance experience would be an added advantage.
● Experience with Auto Loader, Spark Declarative pipelines, Kafka, Airflow, or dbt is a plus.
● Databricks certification is an added advantage.