The Role
Designs and maintains scalable data pipelines and lakehouse architectures across AWS, Databricks, GCP, and Azure. Builds batch and real-time processing solutions with Spark, PySpark, and SQL; develops ETL/ELT workflows; optimizes distributed workloads; and implements data quality, security, governance, orchestration, monitoring, and automation. Collaborates on architecture, technical design, code reviews, and performance improvements while delivering reliable, cost-efficient data platforms.
Summary Generated by Built In
Job Description – Data Engineer (AWS | Multi-Cloud | Databricks)
Experience: 5+ Years
Role: Senior Data Engineer
Primary Cloud Platform: AWS
Secondary Cloud Platforms: GCP and Microsoft Azure
Core Technology: Databricks, Apache Spark, PySpark, SQL, Data Engineering
Job Overview
We are looking for an experienced Senior Data Engineer with 5+ years of hands-on experience in designing, developing, and implementing scalable data engineering solutions across cloud platforms.
The ideal candidate should have strong expertise in AWS as the primary cloud platform, along with working knowledge and exposure to data engineering services across Google Cloud Platform (GCP) and Microsoft Azure.
The candidate must possess excellent data engineering fundamentals, strong programming and problem-solving skills, and very good hands-on experience with Databricks and Apache Spark.
The role involves building modern data pipelines, implementing data lakes and lakehouse architectures, optimizing large-scale data processing workloads, and developing reliable, scalable, and cost-efficient data platforms.
Key Responsibilities
- Design, develop, and maintain scalable end-to-end data pipelines using AWS and Databricks.
- Build and optimize batch and real-time data processing solutions using Apache Spark, PySpark, and SQL.
- Design and implement modern data lake and lakehouse architectures using S3, Delta Lake, Apache Iceberg, and Databricks.
- Work with AWS data engineering services for data ingestion, transformation, orchestration, storage, cataloging, and analytics.
- Develop ETL/ELT workflows integrating structured, semi-structured, and unstructured data from multiple sources.
- Implement data transformation, data quality, validation, and reconciliation frameworks.
- Optimize Spark jobs, SQL queries, cluster configurations, and distributed processing workloads.
- Implement workflow orchestration using Airflow, AWS Step Functions, Databricks Workflows, or equivalent services.
- Work with streaming technologies and services for real-time data ingestion and processing.
- Develop reusable frameworks, automation scripts, and standardized data engineering components.
- Implement security, access control, encryption, monitoring, logging, and governance across data platforms.
- Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable technical solutions.
- Participate in architecture discussions, technical design, code reviews, and performance optimization initiatives.
Required Experience & Qualifications
- 5+ years of experience in Data Engineering.
- Strong hands-on experience with AWS data engineering services.
- Very good practical experience with Databricks and Apache Spark.
- Strong programming skills in Python, PySpark, and SQL.
- Practical exposure to at least one of GCP or Azure, with familiarity with the data engineering ecosystem of both.
- Experience designing and implementing end-to-end data pipelines.
- Understanding of modern data lakehouse architecture and distributed data processing.
- Experience working with large-scale datasets and performance optimization.
Good to Have
- Experience working on multi-cloud data engineering architectures.
- Exposure to AWS, Azure, and GCP migrations.
- Experience with Apache Kafka, Amazon MSK, or other streaming platforms.
- Knowledge of Apache Iceberg, Delta Lake, and open table formats.
- Exposure to data governance, lineage, and metadata management.
- Knowledge of DevOps practices and CI/CD automation.
- Experience with Infrastructure as Code (Terraform / CloudFormation).
- Exposure to Generative AI data pipelines and vector databases.
Ideal Candidate Profile
The ideal candidate should be a hands-on Data Engineer who can independently design, develop, troubleshoot, and optimize complex data engineering pipelines.
We are specifically looking for someone who:
- Has strong practical expertise in AWS data engineering services.
- Has very good hands-on experience with Databricks and PySpark.
- Understands data engineering ecosystems across AWS, GCP, and Azure.
- Has strong fundamentals in distributed data processing, SQL, Python, and data architecture.
- Can work independently on end-to-end data engineering implementations.
- Can contribute to technical architecture and engineering best practices.
Skills Required
- 5+ years of experience in data engineering
- Strong hands-on experience with AWS data engineering services
- Practical experience with Databricks and Apache Spark
- Strong programming skills in Python, PySpark, and SQL
- Practical exposure to at least one of GCP or Azure, with familiarity with both ecosystems
- Experience designing and implementing end-to-end data pipelines
- Understanding of modern data lakehouse architecture and distributed data processing
- Experience working with large-scale datasets and performance optimization
- Experience with multi-cloud data engineering architectures
- Experience with AWS, Azure, and GCP migrations
- Experience with Apache Kafka, Amazon MSK, or other streaming platforms
- Knowledge of Apache Iceberg, Delta Lake, and open table formats
- Exposure to data governance, lineage, and metadata management
- Knowledge of DevOps practices and CI/CD automation
- Experience with Terraform or AWS CloudFormation
- Exposure to generative AI data pipelines and vector databases
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
CloudThat is the first company in India to offer Cloud Training & Cloud Consultancy services for mid-market & enterprise clients from across the globe. Founded by Bhavesh Goswami, an ex-Amazonian with more than 17 years of experience in the Cloud Computing space, CloudThat has been empowering tech professionals with the best cloud training and cloud consulting services. Our journey in the Cloud space has won us many accolades, such as the prestigious Finalist of Microsoft Learning Partner of the Year Awards 2022 and 2020. CloudThat is also a proud AWS Advanced Consulting Partner, VMware Authorized Training Reseller, Google Cloud Platform Partner, Microsoft Gold Partner and Databricks Partner. Since our inception in 2012, we have trained over 500K IT professionals from fortune 500 companies in different types of cloud computing technologies such as Microsoft Azure, Amazon Web Services, Artificial Intelligence, Machine Learning, Google Cloud Platform, IoT, OpenStack, and OpenShift, DevOps, MongoDB, Big Data and more. We have also delivered over 200 AWS consulting services to top companies in need of cloud solutions. We have a global presence with offices in Bengaluru (India), the UK, and the USA. Our on-site and pre-scheduled public batches are available in different IT-centric cities in India and Overseas.
.jpeg)
.png)
.jpg)
