The Role
Design, build, and optimize scalable ETL/ELT pipelines and distributed data platforms using Databricks, Apache Spark, Python, Java, and SQL. Develop ingestion, transformation, validation, monitoring, and data quality frameworks for large datasets. Tune Spark jobs and SQL queries, troubleshoot production issues, support cloud-based architectures, and collaborate with cross-functional teams. The role also involves mentoring junior engineers and contributing to engineering standards, governance, security, and operational excellence.
Summary Generated by Built In
Job Role: Senior Data Engineer – Databricks
Location: Hyderabad, India (Hybrid)
Experience: 6+ Years
Number of Positions: 6
Role Overview
Anblicks is looking for experienced Senior Data Engineers to design, build, and optimize scalable data platforms and high-performance ETL/ELT pipelines using Databricks, Apache Spark, Python, Java, and SQL.
The ideal candidate will have strong hands-on experience in building enterprise-grade data solutions, processing large-scale datasets, optimizing distributed data workloads, and implementing reliable data ingestion and transformation frameworks. The role will involve working closely with data, cloud, analytics, and engineering teams to deliver scalable and high-quality data solutions.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark.
- Build robust data ingestion, transformation, processing, and validation frameworks for large-scale datasets.
- Develop and optimize data processing workloads using Python, Java, Spark, and SQL.
- Work extensively with Databricks to develop scalable data engineering solutions and production-grade pipelines.
- Implement efficient data transformation and storage solutions using modern data engineering practices.
- Write complex and optimized SQL queries for large-volume data processing and analytics.
- Perform performance tuning and optimization of Spark jobs, SQL queries, pipelines, and data processing workloads.
- Implement data quality checks, validation frameworks, monitoring, logging, and error-handling mechanisms.
- Work with stakeholders, data analysts, architects, and engineering teams to understand requirements and deliver effective data solutions.
- Contribute to the design and implementation of cloud-based data platforms and modern data architectures.
- Troubleshoot production issues and ensure the reliability, scalability, and performance of data pipelines.
- Follow data engineering best practices related to security, governance, data quality, and operational excellence.
- Mentor junior engineers and contribute to technical standards, reusable frameworks, and engineering best practices.
Required Qualifications
- 6 - 10 years of professional experience in software development and/or data engineering, with proven experience delivering production-grade data solutions.
- Strong hands-on experience in Data Engineering and ETL/ELT development.
- Strong experience with Databricks and Apache Spark.
- Strong programming skills in Python and Java.
- Advanced proficiency in SQL, including query optimization and working with large datasets.
- Experience designing and developing scalable data pipelines and distributed data processing solutions.
- Strong understanding of data warehousing, data modeling, and large-scale data processing.
- Hands-on experience with data ingestion, transformation, data quality, and pipeline optimization.
- Strong troubleshooting, debugging, and performance tuning capabilities.
- Experience working in Agile development environments and collaborating with cross-functional teams.
Preferred Qualifications
- Experience working with AWS cloud services and cloud-based data platforms.
- Experience with Delta Lake and modern Databricks data engineering capabilities.
- Knowledge of Databricks Workflows, Jobs, notebooks, clusters, and deployment practices.
- Experience with CI/CD, Git, and DevOps practices for data engineering.
- Exposure to AdTech, MarTech, Customer Data Platforms (CDP), or customer analytics environments.
- Experience working with high-volume, complex, and diverse datasets.
Education
Bachelor’s degree in Computer Science, Computer Engineering, Software Engineering, or a related technical field, or equivalent professional experience.
Skills Required
- 6–10 years of professional experience in software development and/or data engineering
- Production experience delivering enterprise-grade data solutions
- Strong hands-on experience in data engineering and ETL/ELT development
- Strong experience with Databricks and Apache Spark
- Strong programming skills in Python and Java
- Advanced SQL proficiency, including query optimization and large datasets
- Experience designing scalable data pipelines and distributed data processing solutions
- Understanding of data warehousing, data modeling, and large-scale data processing
- Hands-on experience with data ingestion, transformation, data quality, and pipeline optimization
- Strong troubleshooting, debugging, and performance tuning capabilities
- Experience working in Agile development environments and collaborating cross-functionally
- Bachelor’s degree in Computer Science, Computer Engineering, Software Engineering, a related technical field, or equivalent professional experience
- Experience with AWS cloud services and cloud-based data platforms
- Experience with Delta Lake and modern Databricks capabilities
- Knowledge of Databricks Workflows, Jobs, notebooks, clusters, and deployment practices
- Experience with CI/CD, Git, and DevOps practices for data engineering
- Exposure to AdTech, MarTech, Customer Data Platforms, or customer analytics environments
- Experience working with high-volume, complex, and diverse datasets
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Anblicks is a Cloud Data Analytics Company based out of Dallas, TX, with offices in USA, India, and Australia. Since 2004, Anblicks has been helping customers by bringing value to their data and implementing modern data architecture and advanced analytics solutions in the cloud.
