Experience
4–8 years
As per business requirement
Full-time
We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.
- Design, develop, and maintain scalable data pipelines using Databricks.
- Build ETL/ELT workflows for batch and streaming data processing.
- Develop solutions using PySpark, Spark SQL, and Delta Lake.
- Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
- Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
- Implement CI/CD pipelines and deployment automation for Databricks workloads.
- Ensure data quality, security, governance, and compliance.
- Monitor, troubleshoot, and optimize production data pipelines.
- Document technical solutions and follow engineering best practices.
Core Technologies
- Databricks Lakehouse Platform
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
- Microsoft Azure (preferred)
- AWS
- Google Cloud Platform
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS Gen2)
- Azure Synapse Analytics
- Azure Key Vault
- Azure DevOps
- Data Warehousing
- Data Modeling
- ETL/ELT Development
- Batch Processing
- Streaming (Kafka/Event Hubs)
- Data Lake Architecture
- Git
- Azure DevOps / GitHub
- CI/CD Pipelines
- Experience with Unity Catalog.
- Knowledge of Databricks Workflows and Jobs.
- Hands-on experience with Delta Live Tables (DLT).
- Exposure to MLflow is an added advantage.
- Experience with data governance and security best practices.
- Familiarity with Infrastructure as Code (Terraform) is a plus.
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Azure Fundamentals (AZ-900)
- Experience with real-time analytics.
- Knowledge of Lakehouse architecture.
- Experience with Agile/Scrum methodologies.
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Databricks
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
- Azure/AWS/GCP (at least one cloud platform)
- ETL/ELT Development
- Data Lake Architecture
- Unity Catalog
- Delta Live Tables (DLT)
- MLflow
- Kafka/Event Hubs
- Azure Data Factory
- Terraform
- Azure DevOps/GitHub Actions
Skills Required
- Databricks Lakehouse Platform
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
- Experience with one or more cloud platforms (Azure, AWS, or GCP)
- ETL/ELT Development (batch and streaming)
- Data Lake Architecture
- Git (version control)
- CI/CD Pipelines / Azure DevOps or GitHub Actions
- 4-8 years of relevant experience
- Bachelor's or Master's degree in CS, IT, Engineering or related field
- Unity Catalog
- Delta Live Tables (DLT)
- Databricks Workflows and Jobs
- MLflow
- Terraform (IaC)
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS Gen2)
- Azure Synapse Analytics
- Azure Key Vault
- Kafka or Azure Event Hubs
- Databricks Certified Data Engineer (Associate or Professional)
- Microsoft Azure certifications (DP-203, AZ-900)
What We Do
iLink Digital, a CMMI Level 3 and ISO 9001:2015 Certified Global Software Solution Provider and Systems Integrator, delivers next-generation technology solutions to help clients solve complex business challenges, improve organizational effectiveness, increase business productivity, realize sustainable enterprise value and transform your business inside-out. iLink integrates software systems and develops custom applications, components, and frameworks on the latest platforms for IT departments, commercial accounts, application services providers (ASP) and independent software vendors (ISV). iLink solutions are used in a broad range of industries and functions, including healthcare, telecom, government, oil and gas, education, and life sciences. iLink’s expertise includes Cloud Computing & Application Modernization, Data Management & Analytics, Enterprise Mobility, Portal, collaboration & Social Employee Engagement, Embedded Systems and User Experience design etc. What makes iLink Digital's offerings unique is the fact that we use pre-created frameworks, designed to accelerate software development and implementation of business processes for our clients. iLink has over 60 frameworks (solution accelerators), both industry-specific and horizontal, that can be easily customized and enhanced to meet your current business challenges.
