The Role
Designs, develops, and supports scalable Azure data platforms using Databricks, ADLS Gen2, PySpark, SQL, Delta Lake, and Azure Data Factory. Builds batch, incremental, CDC, and near-real-time pipelines; optimizes Spark workloads, storage, and cloud costs; implements data quality, governance, security, CI/CD, and monitoring. Provides production L2/L3 support, resolves incidents, maintains documentation, and collaborates with cross-functional engineering and architecture teams.
Summary Generated by Built In
PositionSenior Databricks / Azure Data Lake EngineerExperience6+ YearsLocationHyderabad, TelanganaEmployment TypeFull-Time
Role OverviewWe are looking for an experienced Senior Databricks / Azure Data Lake Engineer with 6+ years of experience in designing, developing, implementing and supporting scalable data engineering solutions on Microsoft Azure.The candidate will be responsible for building robust data pipelines, data lake architectures and transformation frameworks using Azure Databricks, Azure Data Lake Storage (ADLS Gen2), PySpark and SQL.The role requires close collaboration with Data Engineering, Application Engineering, Business Intelligence, Architecture, DevOps and Data Governance teams to deliver secure, scalable and high-quality data solutions.The ideal candidate should have strong hands-on experience with Databricks, Delta Lake, PySpark, Azure Data Lake and Azure data services, along with good understanding of data architecture and performance optimisation.
Key ResponsibilitiesAzure DatabricksDesign, develop and maintain scalable data engineering solutions using Azure Databricks.Develop PySpark notebooks and production-grade data processing jobs.Build reusable frameworks for batch and incremental data processing.Configure and optimise Databricks clusters and compute resources.Implement appropriate cluster policies, job configurations and access controls.Troubleshoot Databricks jobs, performance issues and production failures.Implement Databricks Workflows/Jobs for scheduling and orchestration.Azure Data LakeDesign and manage data solutions using Azure Data Lake Storage Gen2 (ADLS).Implement appropriate data lake folder structures and data organisation strategies.Develop ingestion frameworks for structured and semi-structured data.Implement incremental, full-load and CDC-based ingestion patterns.
Work with data formats such as:ParquetDeltaJSONCSVAvroImplement data partitioning and lifecycle strategies.Ensure data quality, availability and reliability across data lake environments.Delta LakeStrong understanding of Delta Lake architecture and capabilities.Implement ACID-compliant data processing using Delta tables.Work with:MERGEUPDATEDELETETime TravelSchema EvolutionOptimistic ConcurrencyImplement incremental processing and CDC pipelines.Optimise Delta tables using appropriate partitioning and optimisation techniques.Troubleshoot data consistency and performance issues.PySpark & Data EngineeringDevelop scalable data transformation pipelines using PySpark.Write efficient Spark SQL and DataFrame transformations.Optimise Spark jobs for large-volume datasets.Analyse and resolve:Data skewShuffle issuesPartitioning problemsMemory issuesLong-running jobsImplement appropriate caching, repartitioning and broadcast strategies.Develop reusable PySpark libraries and frameworks.Perform data validation and reconciliation across source and target systems.Data Ingestion & IntegrationBuild data pipelines integrating data from multiple sources, including:REST APIsRelational databasesKafkaFilesCloud storageEnterprise applicationsImplement batch and near-real-time data ingestion.
Work with CDC technologies and incremental data processing.Design resilient ingestion frameworks with appropriate error handling and retry mechanisms.Azure Data ServicesHands-on experience with relevant Azure services such as:Azure Data Lake Storage Gen2Azure DatabricksAzure Data FactoryAzure Key VaultAzure Synapse AnalyticsAzure Event HubsAzure MonitorMicrosoft Entra IDExperience integrating these services into enterprise data platforms is highly desirable.Azure Data Factory / OrchestrationDevelop and maintain Azure Data Factory pipelines.Implement pipeline orchestration between ADF and Databricks.Develop parameterised and reusable pipelines.Implement dependency management and error handling.Configure monitoring, alerts and retry mechanisms.Manage production scheduling and operational support.Data Quality & GovernanceImplement data validation and reconciliation frameworks.Identify and resolve data quality issues.Implement data quality checks at ingestion and transformation stages.Follow enterprise data governance and security standards.Maintain metadata, data lineage and technical documentation.Implement appropriate access controls and data protection mechanisms.Performance OptimisationAnalyse and optimise Databricks/Spark workloads.Optimise SQL queries and data transformations.Tune Spark configurations and cluster sizing.Identify bottlenecks in data pipelines.Optimise storage and compute costs.Implement appropriate partitioning, caching and file-size management strategies.CI/CD & DevOpsIntegrate Databricks development with CI/CD pipelines.Experience with Azure DevOps and/or GitHub Actions.Implement source control for notebooks, code and configuration.Automate deployment across:Development → QA → UAT → ProductionImplement environment-specific configuration management.Follow enterprise release and change-management processes.SecurityImplement secure access to Azure Data Lake and Databricks.Strong understanding of Azure RBAC and Microsoft Entra ID.Manage secrets using Azure Key Vault.Implement secure service-to-service authentication.Follow enterprise security and compliance requirements.Experience with Databricks access controls and Unity Catalog is highly desirable.Production SupportProvide L2/L3 support for production data pipelines.Monitor scheduled jobs and resolve failures within agreed SLAs.Perform root-cause analysis for production incidents.Implement permanent fixes and preventive actions.Participate in on-call/support rotations where required.Maintain operational runbooks and technical documentation.Required Technical SkillsMust Have6+ years of experience in Data Engineering.Strong hands-on experience with Azure Databricks.Strong expertise in PySpark.Strong SQL skills.Hands-on experience with Azure Data Lake Storage Gen2.Strong understanding of Delta Lake.Experience developing large-scale ETL/ELT pipelines.Experience with Azure Data Factory.Experience with batch and incremental data processing.Experience with data partitioning and performance optimisation.Strong understanding of data engineering principles.Experience with Git and CI/CD.Experience supporting production data platforms.Good to HaveDatabricks certification.Unity Catalog experience.Azure Synapse experience.Kafka / Event Hubs experience.CDC implementation experience.Azure DevOps / GitHub Actions.Terraform / Infrastructure as Code.Python development experience.Experience with data governance and lineage.Experience in FinTech / Banking / Financial Services.Experience handling large-scale transactional datasets.
Preferred Technical StackCloud : Microsoft AzureData Platform : Azure DatabricksData Lake : ADLS Gen2Processing : Apache Spark / PySparkStorage Format : Delta Lake / ParquetOrchestration : Azure Data Factory / Databricks WorkflowsDatabase : SQL Server / PostgreSQL / Azure SQL / SynapseStreaming : Kafka / Azure Event HubsSecurity : Entra ID / RBAC / Key VaultCI/CD : Azure DevOps / GitHub ActionsIaC : TerraformSource Control : Git
Key DeliverablesThe successful candidate will be expected to:Build highly scalable and reliable Azure data pipelines.Develop reusable Databricks/PySpark frameworks.Improve data pipeline performance and reliability.Implement robust data quality and reconciliation mechanisms.Reduce pipeline processing time and cloud infrastructure costs.Maintain high availability of production data pipelines.Ensure compliance with enterprise security and governance standards.Improve automation across data engineering processes.Maintain high-quality technical documentation.Collaborate effectively with architects, developers, analysts and business stakeholders.Experience in Large-Scale Data PlatformsCandidates should ideally have experience working with:Large-volume transactional data.Multiple source systems.Complex ETL/ELT pipelines.Batch and near-real-time processing.Production-critical data platforms.Multiple development teams and distributed engineering teams.Enterprise data governance and security standards.Soft SkillsStrong analytical and problem-solving skills.Excellent communication and collaboration skills.Ability to work with cross-functional engineering teams.Strong ownership of production systems.Ability to troubleshoot complex data and performance problems.Good understanding of software engineering best practices.Strong documentation and knowledge-sharing capabilities.Ability to mentor junior and mid-level data engineers.EducationBachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science or a related discipline.
Skills Required
- 6+ years of experience in data engineering
- Strong hands-on experience with Azure Databricks
- Strong expertise in PySpark
- Strong SQL skills
- Hands-on experience with Azure Data Lake Storage Gen2
- Strong understanding of Delta Lake
- Experience developing large-scale ETL/ELT pipelines
- Experience with Azure Data Factory
- Experience with batch and incremental data processing
- Experience with data partitioning and performance optimization
- Strong understanding of data engineering principles
- Experience with Git and CI/CD
- Experience supporting production data platforms
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline
- Databricks certification
- Unity Catalog experience
- Azure Synapse experience
- Kafka or Azure Event Hubs experience
- CDC implementation experience
- Azure DevOps or GitHub Actions experience
- Terraform or infrastructure-as-code experience
- Python development experience
- Experience with data governance and lineage
- Experience in FinTech, banking, or financial services
- Experience handling large-scale transactional datasets
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Proclink is a digital and AI transformation company that helps regulated and operationally intensive enterprises modernize systems, workflows, and decision environments. It combines strategy and consulting, data engineering, artificial intelligence, analytics, implementation and integration, managed services, and technology services. The company serves manufacturing, financial services, life sciences, and other regulated industries, delivering connected enterprise intelligence and measurable operational performance for clients.
