The Role
Design, build, and scale batch and real-time ETL/ELT pipelines and data platforms on AWS. Implement lakehouse architectures, optimize performance and cost, enforce governance and security, maintain observability, automate deployments with IaC/CI-CD, and collaborate with stakeholders to deliver scalable data solutions.
Summary Generated by Built In
Deep expertise in AWS data, streaming, and compute services, including Kinesis, Flink, Elasticsearch, PostgreSQL, Lambda / Fargate, Step Functions, S3, Apache Iceberg, Glue, Datadog, and Redshift Spectrum Strong experience in real-time and batch data processing architectures, including event-driven and microservices-based data platforms Advanced proficiency in SQL, Python, and Spark for large-scale data processing, transformations, and analytics Hands-on experience with CI/CD pipelines, Git, and Infrastructure-as-Code (Terraform / CloudFormation) for automated deployments Proven expertise in workflow orchestration tools (Airflow, Dagster, Step Functions) and pipeline scheduling Strong understanding of data modeling techniques (dimensional, normalized, and lakehouse architectures) Deep knowledge of partitioning strategies, indexing, and query performance optimization Experience with data lake and lakehouse architectures (S3 + Iceberg / Delta-like patterns) Familiarity with observability and monitoring tools (Datadog, CloudWatch) for pipeline health and performance tracking Knowledge of data governance, lineage, cataloging, and metadata management frameworks Understanding of security and access control mechanisms (IAM roles, RBAC, encryption, data masking) Experience with cost optimization strategies in AWS (storage tiering, compute optimization, efficient query design) Roles and Responsibilities 1. Data Platform Engineering Design, build, and scale robust ETL/ELT pipelines for structured and unstructured data sources Implement both batch and real-time data ingestion frameworks Ensure pipelines are fault-tolerant, reusable, and scalable 2. Data Architecture & Design Define and implement end-to-end data architecture across ingestion, processing, storage, and consumption layers Drive adoption of lakehouse architecture and modern data patterns Establish data contracts and schema evolution strategies 3. Performance & Optimization Optimize pipelines and queries for high performance, cost efficiency, and scalability Analyze and resolve bottlenecks in processing, storage, and query execution Implement efficient partitioning, indexing, and caching strategies 4. Data Governance, Security & Compliance Implementation of data governance frameworks, including:
Data quality checks
Data lineage and traceability
Metadata management
Ensure compliance with security standards and regulatory requirements Implement fine-grained access control, encryption, and auditing mechanisms 5. Observability & Reliability Engineering Build and maintain monitoring, logging, and alerting frameworks Define and track SLAs / SLIs / SLOs for data pipelines Ensure high availability, fault tolerance, and incident response readiness 6. Automation & DevOps Practices Develop and maintain CI/CD pipelines for data applications Enable infrastructure automation using Terraform / CloudFormation Promote DevOps best practices across data engineering workflows 7. Stakeholder Collaboration & Solutioning Collaborate with business stakeholders to translate requirements into scalable technical solutions Partner with analytics, risk, and reporting teams to ensure data usability and accessibility Provide technical guidance and feasibility analysis for new use cases
Data quality checks
Data lineage and traceability
Metadata management
Ensure compliance with security standards and regulatory requirements Implement fine-grained access control, encryption, and auditing mechanisms 5. Observability & Reliability Engineering Build and maintain monitoring, logging, and alerting frameworks Define and track SLAs / SLIs / SLOs for data pipelines Ensure high availability, fault tolerance, and incident response readiness 6. Automation & DevOps Practices Develop and maintain CI/CD pipelines for data applications Enable infrastructure automation using Terraform / CloudFormation Promote DevOps best practices across data engineering workflows 7. Stakeholder Collaboration & Solutioning Collaborate with business stakeholders to translate requirements into scalable technical solutions Partner with analytics, risk, and reporting teams to ensure data usability and accessibility Provide technical guidance and feasibility analysis for new use cases
Skills Required
- Deep expertise with AWS data, streaming, and compute services (Kinesis, Lambda, Fargate, Step Functions, Glue, Redshift Spectrum)
- Experience with Apache Flink for streaming processing
- Experience with Apache Iceberg (lakehouse patterns) and Delta-like architectures
- Proficiency with SQL, Python, and Apache Spark for large-scale data processing
- Experience building batch and real-time ETL/ELT pipelines and event-driven data platforms
- Hands-on experience with CI/CD, Git, and Infrastructure-as-Code (Terraform or CloudFormation)
- Proven expertise with workflow orchestration tools (Airflow, Dagster, Step Functions)
- Strong knowledge of data modeling techniques and lakehouse/datalake storage patterns (S3 + Iceberg/Delta)
- Experience optimizing partitioning, indexing, query performance, and cost optimization strategies in AWS
- Familiarity with observability and monitoring tools (Datadog, CloudWatch) and defining SLAs/SLIs/SLOs
- Knowledge of data governance, lineage, metadata/catalog management, and data quality practices
- Understanding of security and access control (IAM roles, RBAC), encryption, and data masking/auditing
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
AlgoLeap specializes in AI-powered software solutions, digital product engineering, and IT consulting services, focusing on digital transformation and AI-driven innovation.
