Senior Data Engineer

Reposted 11 Days Ago
Be an Early Applicant
IT Park, Talawade, Pune, Maharashtra
In-Office
Senior level
Fintech • Software • Financial Services
The Role
The Senior Data Engineer will build and optimize ETL/ELT data pipelines using Azure Databricks and Apache Spark, ensure data governance, and collaborate with stakeholders to translate business needs into data engineering solutions.
Summary Generated by Built In
Job Description:

Responsibilities:

  • Develop & Optimize Data Pipelines
    • Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).
    • Optimize performance and cost-efficiency of Spark jobs.
    • Ensure data quality through validation, monitoring, and alerting mechanisms.
    • Understand cluster types, configuration, and use-case for serverless
  • Implement Unity Catalog for Data Governance
    • Design and enforce access control policies using Unity Catalog.
    • Manage data lineage, auditing, and metadata governance.
    • Enable secure data sharing across teams and external stakeholders.
  • Integrate with Cloud Data Platforms
    • Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams.
    • Implement Delta Lake for scalable, ACID-compliant storage.
  • Automate & Orchestrate Workflows
    • Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory.
    • Monitor and troubleshoot failures in job execution and cluster performance.
  • Collaborate with Stakeholders
    • Work with Data Analysts, Scientists, and Business Teams to understand requirements.
    • Translate business needs into scalable data engineering solutions.
  • API expertise
    • Ability to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience:

  • Azure Databricks & Apache Spark (PySpark) – Strong experience in building distributed data pipelines.
  • Python – Proficiency in writing optimized and maintainable Python code for data engineering.
  • Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking.
  • SQL – Strong knowledge of SQL for data transformations and optimizations.
  • Delta Lake – Understanding of time travel, schema evolution, and performance tuning.
  • Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory.
  • CI/CD & Infrastructure as Code (IaC) – Familiarity with  Databricks CLI, Databricks DABs, and DevOps principles.
  • Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption.

Preferred Qualifications:

  • Experience with MLflow for model tracking & deployment in Databricks.
  • Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).
  • Hands-on experience with dbt (Data Build Tool) for modular ETL development.
  • Certification in Databricks, Azure  is a plus.
  • Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
  • Experience with Azure Synapse Link for Dynamics, dataverse
  • Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc

Soft Skills:

  • Strong problem-solving and debugging skills.
  • Ability to work independently and in teams.
  • Excellent communication and documentation skills.

Top Skills

Spark
Azure Data Factory
Azure Databricks
Ci/Cd
Delta Lake
Pyspark
Python
SQL
Unity Catalog
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: New York, New York
1,656 Employees

What We Do

Toppan Merrill is a global leader committed to simplifying the complexity of regulatory disclosure and regulated communications. We are an innovative and trusted partner for the corporate, legal, financial and health plan markets.

Through consultative technology, expert knowledge and service excellence, Toppan Merrill is continuously improving the process of creating compliant communications for capital markets transactions, regulatory disclosure filings, shareholder and member communications and sustainability reporting.

Toppan Merrill is part of Toppan Holdings a leading and diversified provider of sustainable, integrated solutions.

Learn more at toppanmerrill.com

Similar Jobs

Hybrid
Mumbai, Maharashtra, IND

World Wide Technology Logo World Wide Technology

Senior Data Engineer

Big Data • Cloud • Hardware • Software • App development
In-Office
5 Locations
5-5

Data Axle India Logo Data Axle India

Senior Data Engineer

Artificial Intelligence • Information Technology • Software
In-Office
Pune, Maharashtra, IND

NowVertical Group Inc. Logo NowVertical Group Inc.

Senior Data Engineer

Artificial Intelligence • Information Technology • Software • Database • Analytics
In-Office or Remote
Mumbai, Maharashtra, IND
6-8

Similar Companies Hiring

Standard Template Labs Thumbnail
Software • Information Technology • Artificial Intelligence
New York, NY
10 Employees
PRIMA Thumbnail
Travel • Software • Marketing Tech • Hospitality • eCommerce
US
15 Employees
Rain Thumbnail
Web3 • Payments • Infrastructure as a Service (IaaS) • Fintech • Financial Services • Cryptocurrency • Blockchain
New York, NY
40 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account