The Role
Designs, develops, deploys, and supports scalable Azure data pipelines using Azure Data Factory, Databricks, Spark, PySpark, Python, SQL, and Delta Lake. Responsibilities include data ingestion and transformation, quality validation, performance optimization, monitoring, CI/CD, production troubleshooting, technical documentation, and collaboration with architects, analysts, BI developers, and engineering teams.
Summary Generated by Built In
- Independently understand business and technical requirements and translate them into scalable data engineering solutions.
- Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
- Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
- Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
- Develop data transformation and processing logic using Python, PySpark and SQL.
- Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
- Design and implement batch and, where required, near-real-time data processing solutions.
- Develop optimized Databricks workloads using Apache Spark and Delta Lake.
- Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
- Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
- Troubleshoot data pipeline failures, performance issues and data discrepancies independently.
- Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
- Create reusable components and frameworks to improve development efficiency across projects.
- Participate in solution design and technical discussions and provide recommendations on implementation approaches.
- Implement CI/CD processes for data engineering components using Azure DevOps.
- Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
- Prepare appropriate technical design, data mapping and operational documentation.
- Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
- Support production deployments and troubleshoot post-production issues when required.
Requirements
Strong hands-on experience with:
Azure Data Platform
- Azure Data Factory (ADF) – pipelines, datasets, linked services, triggers, parameterization and reusable frameworks
- Azure Databricks
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Azure Synapse Analytics / Azure SQL
- Azure Key Vault
- Azure Monitor / Log Analytics or equivalent monitoring capabilities
- Azure Event Hubs – exposure/experience is preferred
Databricks & Data Processing
- Strong hands-on experience with Databricks and Apache Spark
- Strong PySpark development skills
- Experience with Delta Lake / Delta tables
- Understanding of Medallion or similar layered data architectures
- Experience implementing incremental and full-load processing patterns
- Ability to troubleshoot and optimize Spark workloads
- Experience with Databricks Workflows/Jobs
- Knowledge of Unity Catalog is preferred
SQL & Python
- Strong SQL development skills
- Ability to write and optimize complex SQL queries
- Experience with joins, CTEs, window functions and analytical queries
- Strong working knowledge of Python
- Ability to develop reusable Python/PySpark modules and utilities
DevOps / CI-CD
- Hands-on experience with Azure DevOps
- Git-based source control
- Branching and code-management practices
- CI/CD implementation for Azure Data Factory and Databricks
- Experience deploying solutions across multiple environments
Qualifications & Experience
- 4–6 years of overall experience in Data Engineering / Data Platform development.
- Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
- Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
- Bachelor's degree in Computer Science, Information Technology, Engineering or a related discipline.
- Experience delivering at least one or more end-to-end Azure data engineering projects.
- Good understanding of data warehousing, data lake and modern data platform concepts.
- Strong analytical and problem-solving skills.
- Ability to work independently and manage multiple technical activities.
- Good written and verbal communication skills.
Preferred Skills
- Microsoft Azure / Databricks certifications.
- Experience with Unity Catalog and Databricks governance.
- Experience with REST API-based data ingestion.
- Knowledge of dimensional modelling and data warehouse concepts.
- Experience working with Power BI or downstream analytics platforms.
- Exposure to streaming/event-driven data processing.
- Experience working in enterprise-scale Azure environments.
Skills Required
- 4-6 years of overall experience in data engineering or data platform development
- Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies
- Strong hands-on experience with Azure Data Factory, Azure Databricks, SQL, Python, and PySpark
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline
- Experience delivering at least one end-to-end Azure data engineering project
- Strong hands-on experience with Apache Spark and Delta Lake
- Experience with data warehousing, data lakes, and modern data platform concepts
- Hands-on experience with Azure DevOps, Git, branching, and CI/CD implementation
- Strong SQL development skills, including complex queries, joins, CTEs, window functions, and analytical queries
- Ability to develop reusable Python and PySpark modules and utilities
- Strong analytical and problem-solving skills
- Ability to work independently and manage multiple technical activities
- Good written and verbal communication skills
- Microsoft Azure or Databricks certifications
- Experience with Unity Catalog and Databricks governance
- Experience with REST API-based data ingestion
- Knowledge of dimensional modeling and data warehouse concepts
- Experience with Power BI or downstream analytics platforms
- Exposure to streaming or event-driven data processing
- Experience working in enterprise-scale Azure environments
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
TechCarrot is a global IT services provider and digital-transformation partner, delivering cost-effective, customer-centric digital solutions to clients worldwide. Its 200-plus technology specialists provide services across multiple industries, aiming to create seamless customer experiences and support client growth. Founded in 2016 and headquartered in Dubai, the company serves the Middle East and Africa through UAE operations, APAC through India, and has also expanded into the United States and the Netherlands.
