The Role
Lead design, development, and delivery of real-time Apache Flink ETL pipelines to sync PostgreSQL to StarRocks. Build CDC solutions with Flink CDC/Debezium, optimize performance, ensure reliability, run Flink on GKE with Helm and Harness CI/CD, monitor with Prometheus/Grafana, mentor engineers, and maintain operational runbooks and data quality processes.
Summary Generated by Built In
Be at the Forefront of the Agentic AI Revolution
At Skan AI, you'll be part of the team pioneering the context engine for human and agentic execution, bringing context from enterprise operators, systems, and processes to power how the world's largest organizations execute their most complex, mission-critical work.
Why Join Skan AI
We're in hyper-growth mode at exactly the right moment in history. As enterprises race to adopt agentic AI, we're uniquely positioned to deliver the clear signal they desperately need: a platform that trains and grounds AI Agents in trillions of real execution signals, enabling reliable, compliant automation of their most complex processes.
Backed by Dell Technologies Capital and other leading investors, we're the only company that can bridge the gap between AI's promise and enterprise reality, making us perfectly positioned to define the agentic era for modern enterprises.
Our diverse, collaborative team of 250+ innovators is solving category-defining challenges at the intersection of AI, process intelligence, and enterprise work. Diverse perspectives fuel breakthrough thinking, cross-functional collaboration is the norm, and our work directly transforms how Fortune 500 companies operate. We are shaping the future of work itself.
Role Overview
We are looking for an experienced Apache Flink ETL Lead to own the design, development, and delivery of our real-time data pipeline infrastructure. You will lead data engineering efforts responsible for synchronising data from PostgreSQL to StarRocks using Apache Flink's CDC and streaming pipeline capabilities. You will be hands-on in development while providing technical leadership and driving best practices across the engineering organisation.
This is an individual contributor and lead role. We value hands-on engineers who can also mentor and drive delivery. If you love debugging Flink checkpoints as much as you enjoy growing a team, we want to hear from you.
Key Responsibilities
Pipeline Development and Architecture
- Design, build, and maintain high-performance Apache Flink ETL/ELT pipelines for real-time data synchronisation from PostgreSQL to StarRocks.
- Architect robust CDC (Change Data Capture) solutions using Flink CDC connectors and Debezium for PostgreSQL source ingestion.
- Implement Flink SQL and DataStream API pipelines for complex transformation logic, aggregations, and data enrichment.
- Develop and maintain custom Flink connectors and sinks for StarRocks integration using the StarRocks Flink Connector.
- Design fault-tolerant, exactly-once or at-least-once pipelines with appropriate checkpointing and state management strategies.
- Evaluate and implement schema evolution strategies to handle upstream PostgreSQL schema changes gracefully.
Performance Optimisation
- Profile and tune Flink job performance: parallelism settings, task manager memory, operator chaining, and back-pressure management.
- Optimise StarRocks loading strategies (Stream Load vs. Routine Load) for high-throughput ingestion.
- Monitor pipeline latency and throughput SLAs; proactively identify and resolve bottlenecks.
- Implement efficient watermarking and windowing strategies for time-sensitive data flows.
- Manage Flink state backends (RocksDB / heap) and configure appropriate TTLs to control state size.
Troubleshooting and Reliability
- Own end-to-end pipeline reliability: diagnose and resolve issues including data lag, job failures, checkpoint timeouts, and OOM errors.
- Establish alerting and observability for pipeline health using Flink metrics, Prometheus, and Grafana (or equivalent).
- Define and implement data quality checks, reconciliation processes, and dead-letter queue (DLQ) strategies.
- Perform root-cause analysis on data discrepancies between PostgreSQL source and StarRocks target.
- Maintain comprehensive runbooks for common failure scenarios and recovery procedures.
Team Leadership and Delivery
- Lead Data Engineers: assign tasks, conduct code reviews, and ensure delivery against sprint goals.
- Mentor engineers on Flink internals, best practices, and performance considerations.
- Collaborate with data consumers (analysts, BI teams) to understand requirements and translate them into pipeline specifications.
- Drive technical decisions on tooling, frameworks, and deployment strategies (Flink on Kubernetes on GCP).
- Maintain technical documentation including architecture diagrams, data flow documentation, and operational guides.
Deployment and DevOps
- Manage Flink cluster deployment and configuration on Kubernetes (GKE) using Helm charts and Harness CI/CD pipelines.
- Build and maintain CI/CD pipelines using Harness for Flink job packaging, testing, and deployment to GKE.
- Manage Flink cluster configuration using Helm charts; maintain Helm values and chart templates for environment-specific configurations.
- Coordinate with infrastructure and DBA teams for PostgreSQL slot management and StarRocks table design.
Qualifications and Experience
Technical Skills (Must Have)
- 5+ years of experience in data engineering with a focus on ETL/ELT pipeline development.
- 2+ years of hands-on production experience with Apache Flink (Flink SQL and/or DataStream API).
- Strong proficiency in Java, Scala, or Python for Flink job development.
- Experience with Change Data Capture (CDC) patterns: Flink CDC, Debezium, or equivalent.
- Practical experience with PostgreSQL as a data source, including replication slots and WAL configuration.
- Demonstrated experience with columnar OLAP databases (StarRocks, Doris, ClickHouse, or similar).
- Solid understanding of distributed systems concepts: fault tolerance, exactly-once semantics, state management, and watermarking.
- Experience with Kafka or similar message brokers as part of streaming architectures.
Leadership and Soft Skills
- Proven ability to lead engineering teams, including task planning, code review, and mentoring.
- Strong analytical and problem-solving skills with the ability to debug complex distributed pipeline issues.
- Excellent written and verbal communication skills with the ability to document technical decisions clearly.
- Self-driven with the ability to work autonomously and manage priorities in a fast-paced environment.
Preferred Qualifications
- Hands-on experience with StarRocks Flink Connector and StarRocks primary-key table designs.
- Hands-on experience with Flink on Kubernetes (GKE) using Flink Kubernetes Operator or native K8s mode.
- Experience with Harness CI/CD and Helm chart management for data workloads.
- Experience with Flink Table API and Flink SQL for unified batch and stream processing.
- Knowledge of Apache Iceberg, Delta Lake, or Hudi for lakehouse architectures.
- Experience with orchestration tools such as Apache Airflow or Prefect for hybrid batch/streaming workflows.
- Familiarity with observability stacks: Prometheus, Grafana, and Flink metrics reporters.
- Prior experience in a technical lead or senior engineer role with delivery accountability.
- Understanding of data modelling best practices for analytical workloads in StarRocks.
Domain Knowledge (Nice to Have)
- Experience in data-intensive domains such as data mining, large-scale data processing, or analytical platform engineering.
- Understanding of end-to-end data process flows: from source system ingestion through transformation, aggregation, and consumption layers.
- Familiarity with data governance, lineage tracking, and metadata management.
- Exposure to real-time analytics use cases such as dashboards, operational reporting, or data exploration pipelines.
This is a hybrid role based in Bangalore, with an expectation to work from the office three days a week.
Skan AI is an equal opportunity employer committed to building a diverse, inclusive, and respectful workplace around the world. We do not discriminate based on race, color, religion or belief, sex (including pregnancy, sexual orientation, gender identity, or gender expression), national origin, ancestry, age, disability, medical condition, genetic information, marital or family status, military or veteran status, or any other characteristic protected by applicable laws in the locations where we operate.
We welcome people from all backgrounds and provide reasonable accommodations throughout the hiring process.
Skills Required
- 5+ years of experience in data engineering with focus on ETL/ELT pipeline development
- 2+ years of hands-on production experience with Apache Flink (Flink SQL and/or DataStream API)
- Strong proficiency in Java, Scala, or Python for Flink job development
- Experience with Change Data Capture (CDC) patterns: Flink CDC, Debezium, or equivalent
- Practical experience with PostgreSQL as a data source, including replication slots and WAL configuration
- Demonstrated experience with columnar OLAP databases (StarRocks, Doris, ClickHouse, or similar)
- Solid understanding of distributed systems concepts: fault tolerance, exactly-once semantics, state management, watermarking
- Experience with Kafka or similar message brokers as part of streaming architectures
- Proven ability to lead engineering teams: task planning, code review, mentoring
- Experience managing Flink cluster deployment and configuration on Kubernetes (GKE) using Helm charts
- Experience building and maintaining CI/CD pipelines using Harness for Flink job packaging, testing, and deployment
- Hands-on experience with StarRocks Flink Connector and StarRocks primary-key table designs
- Experience with Flink Table API and unified batch/stream processing
- Knowledge of Apache Iceberg, Delta Lake, or Hudi for lakehouse architectures
- Experience with orchestration tools such as Apache Airflow or Prefect
- Familiarity with observability stacks: Prometheus, Grafana, and Flink metrics reporters
- Prior experience in a technical lead or senior engineer role with delivery accountability
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Skan AI is an enterprise AI company that provides a process intelligence and automation platform. By observing how work actually happens, Skan creates a 'Context Graph of Work' and digital twins of operations. This allows Fortune 500 companies to understand the telemetry of their digital operations, enabling confident automation and lasting transformation through AI agents trained on operational reality.
