Building high-scale, low-latency streaming
data pipelines deployed on infrastructure we run ourselves (on-prem), not
managed cloud services. You will design and operate high-volume real-time data
systems end to end, with Apache Flink as the core stream-processing engine.
Requirements
• 7+ years
of experience in data engineering and software development
• Ability to
write high-quality code in Java/Scala, Python, or equivalent languages
• Deep,
hands-on production experience with Apache Flink — DataStream API and Table API
/ Flink SQL (core requirement)
• Demonstrated
experience with Flink state management: keyed state, state backends (e.g.,
RocksDB), large state sizes, and state TTL
• Hands-on
experience with checkpointing, savepoints, and fault tolerance — exactly-once
vs. at-least-once semantics, recovery, and savepoint-based job upgrades
• Strong
grasp of event-time processing: watermarking, windowing strategies, allowed
lateness, and late-data handling
• Experience
diagnosing and resolving backpressure — parallelism, operator chaining, and
network buffer tuning
• Experience
operating Flink on self-managed infrastructure (Kubernetes or YARN) —
application vs. session mode, high availability, and rolling upgrades
• Practical
experience with stream processing (Kafka Streams or equivalent) and messaging
systems for high-volume workloads, including exactly-once sinks and schema
registry usage
• Practical
experience with distributed query engines (e.g., Trino/Presto or similar)
• Practical
experience with ETL / data integration tools, commercial or open-source (e.g.,
Datastage, Informatica, Apache NiFi, or similar)
• Practical
experience with SQL-based transformation frameworks (e.g., dbt or others)
• Strong SQL
skills and understanding of data modeling and data warehousing for analytical
workloads
• Hands-on
experience with real-time / low-latency analytical stores (columnar or OLAP
engines, e.g., Apache Pinot/ClickHouse or similar)
• Practical
experience with big-data platforms and distributions (e.g., Cloudera, Hadoop
ecosystem, or similar)
• Practical
experience containerizing and operating data workloads (Docker; Kubernetes a
plus)
• Experience
with workflow orchestration tools (e.g., Airflow or similar)
• Familiarity
with data lake table formats (e.g., Apache Iceberg or similar), including
streaming ingestion, compaction, and small-file management
• Familiarity
with data governance / cataloging tools (e.g., DataHub or similar)
• Familiarity
with lakehouse management systems (e.g., Apache Amoro or similar)
• Familiarity
using AI tools for development and debugging (Claude, Cursor, Codex)
Skills Required
- 7+ years of experience in data engineering and software development
- Ability to write high-quality code in Java, Scala, Python, or equivalent languages
- Deep hands-on production experience with Apache Flink, including DataStream API and Table API or Flink SQL
- Experience with Flink keyed state, state backends such as RocksDB, large state sizes, and state TTL
- Experience with checkpointing, savepoints, fault tolerance, delivery semantics, recovery, and savepoint-based upgrades
- Strong understanding of event-time processing, watermarking, windowing, allowed lateness, and late-data handling
- Experience diagnosing and resolving Flink backpressure through parallelism, operator chaining, and network buffer tuning
- Experience operating Flink on self-managed Kubernetes or YARN infrastructure, including deployment modes, high availability, and rolling upgrades
- Experience with stream processing and messaging systems for high-volume workloads, including exactly-once sinks and schema registries
- Practical experience with distributed query engines such as Trino or Presto
- Practical experience with ETL or data integration tools such as DataStage, Informatica, or Apache NiFi
- Practical experience with SQL-based transformation frameworks such as dbt
- Strong SQL skills and knowledge of data modeling and data warehousing for analytical workloads
- Hands-on experience with real-time or low-latency analytical stores such as Apache Pinot or ClickHouse
- Practical experience with big-data platforms and distributions such as Cloudera or the Hadoop ecosystem
- Practical experience containerizing and operating data workloads with Docker; Kubernetes experience is a plus
- Experience with workflow orchestration tools such as Airflow
- Familiarity with data lake table formats such as Apache Iceberg, including streaming ingestion, compaction, and small-file management
- Familiarity with data governance or cataloging tools such as DataHub
- Familiarity with lakehouse management systems such as Apache Amoro
- Familiarity with AI development and debugging tools such as Claude, Cursor, or Codex
What We Do
Techsa is a Cairo-based software services and development company that builds technology solutions tailored to client needs. Its work combines custom and mobile-app development, IT consulting, and software innovation with IoT/data-management platforms and analytics, including AI/ML-driven analysis. It serves sectors such as water and wastewater, logistics, roads, and property maintenance, and also describes itself as a partner for recruiting, onboarding, and managing software teams.
