Senior Data engineer

Posted Yesterday
Be an Early Applicant
Chennai, Tamil Nadu, IND
In-Office
Senior level
Information Technology • Consulting
The Role
Designs and maintains scalable Azure Databricks data pipelines using Python, Scala, PySpark, Delta Lake, and SQL. Develops ETL/ELT workflows, optimizes processing performance, ensures data quality and governance, manages Azure Data Lake and Unity Catalog integrations, and supports cloud architecture, migration, modernization, troubleshooting, documentation, and cross-functional delivery.
Summary Generated by Built In


Key Responsibilities

 

Data Pipeline Development


• Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark.

• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.

• Develop and maintain data processing workflows using Delta Lake architecture.

• Ensure efficient data ingestion, transformation, and loading processes.


Data Management & Optimisation

• Optimise performance of data pipelines and processing workloads.

• Ensure data quality, reliability, and consistency across data platforms.

• Write and optimise complex SQL queries for data processing and validation.

• Implement data governance and access controls.


Cloud & Data Platform Integration

• Work with Azure Data Lake for storing and managing large datasets.

• Manage data cataloging and governance using Unity Catalog.

• Support scalable and secure cloud-based data architectures.


Collaboration & Migration Projects

• Work closely with cross-functional teams to deliver data-driven solutions.

• Participate in data migration and modernisation initiatives.

• Support troubleshooting and resolution of data pipeline issues.

• Document data architecture, pipelines, and processes.


 Required Skills & Experience 

·        Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks with good communication skills.

·        Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake.

·        Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL).

·        Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.

·        Optimise performance of data processing and ensure data quality standards.

·        Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines.

·        Experience with orchestration tools like Airflow or Azure Data Factory is a plus.

·        Have working experience on a migration project. 

·        Preferable to have experience in Informatica (although not mandatory)

  

Relevant Experience Preferred 

• Hands-on experience developing data pipelines using Azure Databricks and PySpark.

• Experience implementing ETL/ELT solutions for enterprise data platforms.

• Experience optimizing data processing performance and data workflows.

• Experience working with Azure Data Lake and cloud-based data architectures.

• Experience working on data migration or modernization projects.



Skills Required

  • At least 7 years of IT experience
  • At least 5 years of hands-on Databricks experience
  • Experience designing and developing scalable data pipelines with Azure Databricks, Scala, Python, PySpark, and Delta Lake
  • Understanding of relational database management systems and proficiency writing complex SQL
  • Oracle or PL/SQL experience
  • Experience implementing ETL/ELT workflows for structured, semi-structured, and unstructured data
  • Experience optimizing data processing performance and ensuring data quality
  • Experience with Azure Data Lake, Unity Catalog, and CI/CD pipelines
  • Experience with orchestration tools such as Airflow or Azure Data Factory
  • Experience working on a data migration project
  • Informatica experience
  • Good communication skills
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Toronto, ON
1,078 Employees
Year Founded: 1990

What We Do

Kumaran Systems is an IT Services Company with imprints in three countries. The past two decades has seen us provide our global clientele with high-end IT services that include migration support, system integration and infrastructure management solutions, providing one-stop-solution to all your IT demands. Our expertise and in-depth knowledge of businesses help us cater to a variety of industries. Our team strives to know your industry better, by observing current trends and the way it works, to tailor-make our solutions to your needs.With over 20 years of customer orientation, over 2000 engagements across Telecom, Education and Banking & Financial services spread across the globe, Kumaran stands as a key advisor to some of the largest Fortune 500 companies in their business driven technology enablement drive. Kumaran System's Customer Orientation is driven by a global delivery business model giving its customers to choose between an Onshore-Offshore mix. The delivery models enable multilevel touch points between the client, partner networks and Kumaran thereby enabling business driven customer sensitivity and agility. ENGAGE, EMERGE AND EXCEL Involving ourselves in a purposeful action, using our abilities to its maximum, helps us feel positive about ourselves. Such positivity gives raise to positive thinking and positive ideas. Ideas evolve into vision and set our aim! Our aspirations engage us with the incentive of sweet success! Every endeavor has its own pitfalls. Perseverance helps us emerge a successful entrepreneur after any deterrent. In our attempt to achieve our aim, we keep upgrading our knowledge and skill, to surface successful! A genius mind does not work to excel, but excels in work! We challenge ourselves with every next step! We change every deterrent to a stepping stone to success. We essentially, engage, emerge and excel

Similar Jobs

Kyndryl Logo Kyndryl

Senior Data Engineer

Cloud • Information Technology • Consulting
In-Office
2 Locations
46070 Employees

InFynd Logo InFynd

Senior Data Engineer

Information Technology • Software • Analytics
In-Office
Coimbatore South, Coimbatore, Tamil Nadu, IND
174 Employees

ShyftLabs Logo ShyftLabs

Senior Data Engineer

Blockchain • Database • Analytics
Hybrid
Coimbatore, Tamil Nadu, IND
110 Employees

Nexaminds Logo Nexaminds

Senior Data Engineer

Artificial Intelligence • Cloud • Information Technology • Machine Learning • Natural Language Processing • Consulting
In-Office
Coimbatore, Tamil Nadu, IND
80 Employees

Similar Companies Hiring

NODA AI Thumbnail
Artificial Intelligence • Information Technology • Software • Cybersecurity • Defense • Defense Technology
Austin, TX
54 Employees
Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
567 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account