Key Responsibilities
Data Pipeline Development
• Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark.
• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop and maintain data processing workflows using Delta Lake architecture.
• Ensure efficient data ingestion, transformation, and loading processes.
Data Management & Optimisation
• Optimise performance of data pipelines and processing workloads.
• Ensure data quality, reliability, and consistency across data platforms.
• Write and optimise complex SQL queries for data processing and validation.
• Implement data governance and access controls.
Cloud & Data Platform Integration
• Work with Azure Data Lake for storing and managing large datasets.
• Manage data cataloging and governance using Unity Catalog.
• Support scalable and secure cloud-based data architectures.
Collaboration & Migration Projects
• Work closely with cross-functional teams to deliver data-driven solutions.
• Participate in data migration and modernisation initiatives.
• Support troubleshooting and resolution of data pipeline issues.
• Document data architecture, pipelines, and processes.
Required Skills & Experience
· Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks with good communication skills.
· Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake.
· Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL).
· Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
· Optimise performance of data processing and ensure data quality standards.
· Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines.
· Experience with orchestration tools like Airflow or Azure Data Factory is a plus.
· Have working experience on a migration project.
· Preferable to have experience in Informatica (although not mandatory)
Relevant Experience Preferred
• Hands-on experience developing data pipelines using Azure Databricks and PySpark.
• Experience implementing ETL/ELT solutions for enterprise data platforms.
• Experience optimizing data processing performance and data workflows.
• Experience working with Azure Data Lake and cloud-based data architectures.
• Experience working on data migration or modernization projects.
Skills Required
- At least 7 years of IT experience
- At least 5 years of hands-on Databricks experience
- Experience designing and developing scalable data pipelines with Azure Databricks, Scala, Python, PySpark, and Delta Lake
- Understanding of relational database management systems and proficiency writing complex SQL
- Oracle or PL/SQL experience
- Experience implementing ETL/ELT workflows for structured, semi-structured, and unstructured data
- Experience optimizing data processing performance and ensuring data quality
- Experience with Azure Data Lake, Unity Catalog, and CI/CD pipelines
- Experience with orchestration tools such as Airflow or Azure Data Factory
- Experience working on a data migration project
- Informatica experience
- Good communication skills
What We Do
Kumaran Systems is an IT Services Company with imprints in three countries. The past two decades has seen us provide our global clientele with high-end IT services that include migration support, system integration and infrastructure management solutions, providing one-stop-solution to all your IT demands. Our expertise and in-depth knowledge of businesses help us cater to a variety of industries. Our team strives to know your industry better, by observing current trends and the way it works, to tailor-make our solutions to your needs.With over 20 years of customer orientation, over 2000 engagements across Telecom, Education and Banking & Financial services spread across the globe, Kumaran stands as a key advisor to some of the largest Fortune 500 companies in their business driven technology enablement drive. Kumaran System's Customer Orientation is driven by a global delivery business model giving its customers to choose between an Onshore-Offshore mix. The delivery models enable multilevel touch points between the client, partner networks and Kumaran thereby enabling business driven customer sensitivity and agility. ENGAGE, EMERGE AND EXCEL Involving ourselves in a purposeful action, using our abilities to its maximum, helps us feel positive about ourselves. Such positivity gives raise to positive thinking and positive ideas. Ideas evolve into vision and set our aim! Our aspirations engage us with the incentive of sweet success! Every endeavor has its own pitfalls. Perseverance helps us emerge a successful entrepreneur after any deterrent. In our attempt to achieve our aim, we keep upgrading our knowledge and skill, to surface successful! A genius mind does not work to excel, but excels in work! We challenge ourselves with every next step! We change every deterrent to a stepping stone to success. We essentially, engage, emerge and excel









